Noah's Mommy, I know it is hard but I think you are doing what is best for you and all of your cats including Oliver.
At least you will be able to visit him at the shelter until he finds a new home.
Some cats really are meant to be only cats, my cat Samantha is one of those.
We tried to add another cat to our family once but decided it was better to have one very happy spoiled cat instead of two not very happy cats!

Wolflady and Trinity, you both described my cat Samantha perfectly!
Must be something about the name that makes them want to be the Queen of the castle!