I agree with not putting a collar on him, especially since he hates it so much.
I nearly lost my cat Clyde because of a collar. He really hated wearing it, he would drag himself along the ground trying to get the collar off. One day I hear the sound of books falling, so I went to see what was happening. Clyde had tried again to remove his collar, and had got it caught on something, and had nearly hung himself. And this was one of those safety collars too, it didn't break. The only reason I found him was because he was struggling so hard that he knocked over a pile of books.
Now Clyde is "naked" and loving it, he has a microchip in case he ever gets out.