That is creepy, but I wouldn't worry to much. On occasion, I've been driving somewhere and noticed a co-worker's car outside or something. Later on, I'd be like "Hey, do you live in such and such area, I think I drove by your area." But I never hunted for their houses!

Hopefully, that was the frame of mind he had and just worded it totally wrong. I'd like to think that his intentions are good being that he brought the subject up. I do thinks it's odd him and his wife went for a Sunday drive looking for your house though.