I would stop all contact with the brother
Oh boy, don't I wish.

But unfortunately, my brother lives with my dad. And other than that, my dad is alone. My mom died 10 years ago and his best friend/phone companion/etc, just passed away this past week. I can't desert my dad now.