Maybe take the baby with you when you talk to the neighbors?