If it's not working out for anyone in your house, then that's probably the best decision you can make, as heartbreaking as it is. You know your cats better than anyone and would know if the situation was irretrievable. It's very sad, but I know you'll work hard to find a more suitable home for Oliver and everyone will be happier in the long run.