Cats are really thrown for a loop when they are in a new environment, even one that we know is better for them. I think it's generally accepted that a cat will be far more happy at home with a once a day visit by someone to give them food while their human is on vacation than they would be in the most luxurious kennel with frequent attention. Although he definitely needed to be rescued, the outside was you cat's home and the place where he felt most comfortable. As others have said, time will change his attitude.

I went through what you are going through about a year ago; my rescue was an abandoned young tom cat and he was also more friendly towards me outside than he was for several months inside. I just kept sitting on the floor talking to him and focusing on him until he really, really wanted to be close to me. Now he loves to have his special time on my lap and today he licked my hand for the first time. I don't think he'd had much attention in his previous "home" and so had to make several adjustments.