Our city just enacted a similar bylaw, limiting how many animals are permitted per household. I was very concerned since I have 12 cats and 3 dogs. However, I also learned later, that I was OK because my animals were "grandfathered" in---in other words, as long as I didn't add any new ones, the ones I have now can stay. Maybe the visits are to make sure you don't add more. I didn't check the bylaws Phred listed. Do you know when they went into effect?

All of mine were like yours, Gary. I fed them on the front porch until I could get them vetted and moved inside. If you feed them, the law says they are yours. Your neighbors MAY be legally right, but they are not right to punish you for the strays others have abandoned.