I can't believe that in my 40 plus years, this never dawned on me, but I think you've got something here.

My cats :

RB Cuddles (female) - tolerated me, loved my brother
RB Paul (male) - adored me and my sister
RB Kuhio (female) - worshipped the ground my husband walked on.
Halo (female) - seems to love us both
Cammie (female) - growls at the sight of me. My husband has nicknamed her "Velcro Cat" because she's always attached to him.

Looks like I'm going to have to get more males, if I want any attention.