I can really relate to this! When we go on holiday to some sunny or exotic clime, the idea is to have a great time and relax. But what do I do the first few nights? Lay in bed worrying about Fister of course. Is the cat pension OK for him? Is he very sad at being alone and missing us? Are they treating him properly? How long before we're back home and can snuggle up in bed with him again? (I used to count the days!).
He has always been a very nervous cat and very dependent on us, and he HATES the pensions. But there's no better solution, at least I can comfort myself by thinking that he's better off there than he would be if he was still a feral in his old back yard.
It's a real problem, I have to force myself not to think about him or ruin my holiday! But luckily I'm getting better at it nowadays.

And as Randi mentioned, the neighbors must consider us a bit odd. I sometimes have long conversations with him up in the window when I get home. But that doesn't matter, the neighbors just don't know how much happiness and love we're getting from our darling Fister. They're just missing out on the magic of having a cat.