A lot can depend on police files - so get them to phone the police straight away with definte times, dates and a description of the man. They must demand a case or file number or identification code so that if this man comes back they have an immediate reference to the last time he was there.
The dog, children, or dog and children together should not be outside on their own. A supervising adult should always have a video camera at best, a tape recorder at least and a stills camera if no other option is available at hand at all times!!
You should tell everyone on the street that has dogs and/or children to do the same.
Every sighting, incident of harrassing or uninvited viewing should be reported using the incident number.
The more you have under the same number the more likely that the police will take you seriously.
I don't think this person is going to poison the dog but is more likely to go for a dangerous dog type court type battle.
Evidence is what is needed to prove him as the aggressor.