I would definitely say something. I'd probably say...I understand you don't like cats, but can you stop making comments all the time? Or I'd say....well, the cat lives here so if you don't like it then don't come over. Who knows, I'd probably not be very nice about it after putting up with it so often.

He is definitely very immature and needs to grow up. Will your boyfriend talk to him about it, if you don't feel comfortable doing it? I can understand you not wanting to cause problems. In fact, I probably wouldn't have said anything when I was younger either. I've become a bit "hard" in my old age.