If people don't respect the household rules, then they are not welcome. How difficult is it to leave the cats alone? He is trying to get under your skin, then pretending he doesn't know why you are angry. Tell them you would be happy to meet them elsewhere to visit, but they cannot come to your home unless they are respectful.