Oh dear...{{{{{HUGS}}}}

I am afraid I can't offer much help, but as all the others already said, let your gut feeling decide!
You decide the limit...none of us can decide for you, as it's you who has to live with the cats...but whatever you'll do, be sure the Pet Talkers will support you!!!

As for your neighbor, I have witnessed something similar with my sister's neighbor, who was always a huge animal lover until the day their 1st grandchild arrived and their center of attention was limited only to that baby...I cannot understand it in a gazillion years...but it happens every day...