Though I'm not in your shoes but because I am a kitty/cat lover. I would give the cat the attention he/she needs. I have two cats now who picked me to be the owner or should I say own me! Looks like this cat has picked you :-)
Maybe your neighbor isn't an animal person....meaning maybe she doesn't understand that animals need affection. Some people just view animals as JUST ANOTHER ANIMAL(she might be that type). I can tell you are not that type & so can that cat who has picked you! I know her child loves that cat & maybe you can work out something where the cat becomes yours ( if you want that) & the kid can come visit when he wants to.???? Hope that helps a little, but i doubt it does.