Sometimes cats just don't like each other. Usually given enough space and time they will come to being civil. Is the fighting noise and slapping or is damage being done?

It is also possible that the cats are responding to some signal that the one is giving that is not "right" to them. It could be illness...sometimes cats will drive an ill member out of the colony, or it could be some breach in your cats' particular social order.

How do the others respond to him? Is it possible for you to interven by showing he is a valued member while still giving attention to the others? Has he ever been aggressive toward the kitten or females? That could cause it.

You might try reintroducing the males as if they were strangers. Good luck. I've always had multple nuetered male sin the household and usually strife can be resolved.