It sounds like good progress, and he seems to enjoy being by your house. I think eventually he'll come around and you'll be able to stroke him, but it may take some time - so patience!

I think cats like it when you talk to them, I know Fister does.