Sadly, it is very possible. That doesn't mean you have to give up your cat, but you do have to change your habits. Wash your hands after petting him, don't touch your face after touching him. Vacuum regularly, and consider putting something like a blanket or towel that you can easily wash down on the couch or wherever he like to nap.

Speaking as an "allergic person" - I know that allergies can develop in anyone at point in their life. You may want to get tested to make sure it is the kitty you are reacting to, and oh, do clean the surfaces in your car and set a towel down where he like to ride in that as well.