I suspect you will have a *feel* for the way Toby adjusts to his new home when you are with him and see that he stays (or not) very close to the house and marks HIS territory under supervision. I can understand that desire to have the happy farm cat. The neighbor cat you talk about sounds like something out of a James Herriot story!! I sorta wish for that as well... but I am not sure it is possible unless the kitty is born and raised there and I think I would still want kitty in sight and locked in the house at night or when I was not home.
I really think your *gut* will help you know what is right and best for Toby - good that you are thinking ahead!!