I too agree with everyone else here. He should have respected your request and left the cat alone. You were right to ask them to leave. You should do the letter thing and explain your position, and that you do love them very much but that such behavior will not be tolerated.

Hugs to you and your sweet babies.