Collars CAN work for small dogs - but you have to be very in control of your movements in every situation.....I would feel safer keeping the harness -BUT you must treat the harness as you would a collar on a large dog.
Your problem is one of two things - there may have been an instance when the dog was surprised or startled or even hurt - it only needs to a small hurt - when the harness was put on. This may have happened very early in the dog's life and it may have been some other thing , like a collar or a towel put over the head that caused it.

My gut feeling on this one is that you have a "who makes the choices in this family" problem.
The facts that the problem has grown from putting the harness on to taking it off, aggression was not apparent at first and the dog is happy and calm between the putting on and taking off all point to this being the reason.

For both reasons there are things you can do to improve the situation greatly - but you do have to be prepared to treat the dog as you would a much larger dog behaving the same way - imagine a German Shepard reacting to having a collar put on the way your dog is.......the consequences could be devastating. Your little dog thinks and behaves the same way as the German Shepard would and there are very safe, positive and effective ways of dealing with it.
I am not trying to put you in a stereotype in anyway but I do realise that it can be hard to realise your little lap dog is still a wolf in dog's clothing. My most troublesome cases always start with a phone call saying, "It may sound strange but my little (insert name of toy breed) has me terrified of him/her and I don't know what to do!"
Please post or email what you think the most likely cause of this behaviour is and I will try and give you some ideas.