Neighborhood pets?

Hello all! I have a nieghbor who is really annoying me! But maybe I am being overly sensitive? My cats (3) are indoor/outdoor cats mostly indoor. I know cats are cats, they are going to roam around and check things out. My next door neighbor has two cats of his own and they are not all that friendly due to his daughter's overly aggressive behavior towards them (she is sweet, but just wants to love her poor kitties to death kwim?) Anyways so he has taken to my kitty Gizmo, and has referenced Giz as his cat several times. OOOHH! That just really irritates me! He allows Giz in his house, and encourages him to hang around his yard. When his cats would come to my yard, I would shoo them away, not to be mean but to send them home. I don't know why he gets to me sooo much? Do you have neighbors cats into your home? I guess I think of them like children, I wouldn't reference my neighbors children as mine, or let them in my home without their parents knowing. I am just looking for a point of reference on my feelings. Am I way out in left field, or is my neighbor crossing the line? (or maybe something in between ?)

TELL HIM HOW YOU FEEL!! Let him know you are not comfortable with YOUR cat being in HIS house, and that you aren't comfortable with him referring to your cat as his!! Maybe its just a misunderstanding and he doesn't realize he's getting to you, but you need to LET HIM KNOW!