Well it sounds like you and your brother actually get along, and if you do, and he doesn't mind you hanging out with him and his friends, I don't see a problem with having fun with your brother and friends!