I'm sure you've noticed already, but you and your brother are two different people. You have your own heart, your own mind, and your own feelings. Do what YOUR heart says... not your brothers.

If meeting your father is what you truely want, then go for it. Your brother does not have to be a part of it if he wishes not to.