Oh SAS, I have done that very thing, and it is a terrible, terrible feeling. I very rarely commit such thoughts to email, purely because of how traceable digital messages are, but there was one day I sounded off about a co-worker who was getting on my very last nerve, and then sent it to her by mistake! I had to fly through to her desk and got there roughly the same time as the email. Some very quick thinking was required, but I did get out of it.
I think you are doing the best thing by just engaging with him for a while.
Bookmarks