Thanks everyone. I won't ever give up on him, unless god forbid he gets VERY mean.
Do you know what, he started to go downhill a few days after I let him out of his quarantine room. So I'll be getting that back in order & putting him back in there for a while see how it goes.
He is supposed to be off his ear mite meds in a few days but I'm still seeing them (he litterally had mountains of mites in his ears). I'm going to weigh him & hopefully he's either 5 lbs so I can give him revolution or see if my vet has something different that is less stressful to give him.

I have & still do pet him when he gets fed his wet food, I'd do the same with dry but I leave that out 24/7 so he's a nibbler on the dry so thats almost impossible to pet him.