Did you explain that you were on the way back with a poop bag? I just find that with people like this its best to ignore them. If he disturbs you at night with his car call the police.