I have a drug house next to me and I could write a book. I did have a neighbor from down the road once ask me "you have two dogs and two cats in that house?" Yeah, so what's the big deal? My house is small but I have a huge fenced in yard for the dogs to play in. So I don't see the big deal.
As for your neighbors picking on you, I'd continue to be a responsible pet owner and let them see that. Some people jump the gun 'thinking' your pets will be running the neighborhood.