Sorry I'm late to this, but I would definitely take him home too. As you say, it's up to fate now, but you might try talking to some of the people who've said they've seen him. See if one of them will coax him inside if/when they see him again, then hold him until you can come pick him up. They might do it, they might not, but it would be worth asking anyway. Or, maybe just give them your phone number and have them phone you when they see him and maybe you can make it over there before he leaves again.

Poor kitty sounds like he had a good home at one time, but then again, how good could it have been if they just abandoned him like that.

We've got fingers, paws and tails crossed that you find him soon.