Cats are more resilient than we give them credit for. I would bring him along. I myself would worry way too much about him being alone. I worry when I go out of town for a day and leave my husband and the kids alone with the cats! I'd be a mess... was there enough water? Was there enough food? What about the cleanliness of the litterbox? What if he ate a bug and got sick? What if he knocked something over and stepped on the broken glass? Yessiree... I'm a worry wart. If I can't be there with them, they better be here with me, or have someone checking in on them a few times a day.

All that said, I can see my husband saying "they'll be FINE" and rolling his eyes at me. Whatever you do, ulitmately its got to be something that works for you, and you are comfortable with.