You have nothing to feel badly about. You clearly did everything possible. This decision is as much Oliver's as yours. Maybe he just wanted to tell you that he appreciates the offer, but sadly must decline.
I'm sure you'll see that he gets a home, one where he can rule the roost. In the current situation, he's unhappy and so are your other kitties. That's no life for anybody.