Well I think it's off kilter! Not the looking at your house all by itself but the looking and then telling you. (so I agree with Cincy'sMom). It would be like looking in the medicine cabinet and then saying, oh me and my wife looked in your medicine cabinet" ok not exactly.. bad example! Well, it would bother me and if it were me, i'd find an opportunity to say something like, "so why were you in my neighborhood that day?" Then say no more, let the silence work it's charms- he will have to answer something. He has not flirted with you at all? If no, are you sure you haven't missed messages? He could be using "with my wife" as a protection to feel you out... I don't know, what does your gut tell you? Your initial reaction was to be on guard and there is certainly nothing wrong with that! Womens intuitions have been often proven to be on the right track! we can sense when things are just not on the up and up. Sometimes we have trouble paying attention to our intuitions- thats when we get into real trouble! So do what you feel is right.