I think part of the "forgetting" is after you have been away, you come back smelling a little different. Your clothes, even your hair, may smell a little foreign to your kitty. He should come around in time.