I'll go with the bad experience theory. My mom and I catnapped a neighbor's young cat years ago because he was a sweet and gentle cat who was being terrorized by the neighbor "man of the house" (to use the term very loosely). The guy always wore a baseball cap. Here we are ten years later and "Sammy" is still frightened by anyone wearing a cap. It was a terrible experience for him.

We found a good home for him. He used to come when I called him and loved to be carried around snuggled in my arms. One evening a friend of my sister came over with a cat carrier and I called him... the rest is history. Since then he has lead the kind of life he deserved, being well cared for and well loved.