Anna, it might take something as simple as saying "I hate it when you say things like that" when he does it. And maybe you have to say that more than once, or say something like "I really wish you wouldn't say things like that". He may think he's being funny and not realize that anyone is offended by it. Some people's insecurities show through in the strangest ways, and this might be one of those situation where he is trying to "fit" (awkward as it is).

I hope you can work it out, whether it be face to face, or through a letter, but I think I would prefer the face to face as to handing him a letter like that.