I don't think so If everyone is having a fun time why should it be a problem? My boyfriend's brother is only 3 years younger than him. They have a lot of mutual friends and hang out all the time. I see his brother almost as much as I see him, lol!
I don't know if it's different with two brothers, but I don't see what the problem is if everyone is happy and having fun Have you talked to your brother about it?