Logan,

don't worry Butter knows he is much better off with you and would certainly come back.

Something similar happened to a friend of mine. She lives in a small apartment block (only 6 apartments) with a private garden around the building. One of her neighbours had a cat. The only thing is that the poor thing was not allowed in the house at night, no matter how bad the weather was. The woman said it would make her fur look better (no comment to this statement). The cat also associated being picked up with being thrown out so to this date she does not like if someone picks her up. One night my friend's brother came home and the cat sneaked in the house. This happened more and more often until the cat decided that she much preferred to live there. The neighbour got mad, she got the cat back but that lasted only a couple of hours and back she was where she wanted to be. From then on everytime the cat sees her ex-owner in the garden (when they are in she can go in and out the apartment as she pleases, when they are out she spends the day lazying on the sofa) she just hisses and runs away. There were threats of legal procedures; after a lot of arguing though my friend managed to keep the cat who has been happily living with them for the past 8 years.