Ben-that doesn't mean that you can't take him to visit anyway - my sister frequently takes her cat to vist our parents, and her cat feels like it is her "vacation" home - none of the resident cats resent her and she doesn't feel threatened. Try it, and if he doesn't seem stressed, nor do the residents, there's no reason not to keep letting him viist.