It almost sounds as if he should be in a one-cat home where he can feel safe to explore at his own leisure.

Do what you think is best and don't feel guilty at all. You are a wonderful purrrson and whatever decision you make will be all right with me.