I would just tell him, "Tough! She lives here and you don't! She can go where she wants, IT'S HER HOME."