Take him to a no kill rescue, and give him a chance at a real life. Never mind what the neighbor says, just don't let her see you. She's not a Mommy to him.