Cats absolutely know when someone is not "fond" of them.

Years ago, long before I became passionate about cats (I couldn't stand them at the time, we're talking 1980) my Dad asked me to cat sit for his snot cat Jessie, while he and my stepmonster went on their honeymoon. I agreed. The whole time all she did was was hide under my waterbed and hiss at me when I tried to be friendly to her.

So I can answer you honestly that YES, they do know when people don't like cats.

As you can see by my sig, I have changed my evil ways. I cherish the ground they walk on.