You may find that as your baby gets bigger he will become more tolerant to the cats.

My daugher had asthma but was surrounded by animals and just grew out of it in the end. She certainly had more benefit from being around them than being kept isolated

Give your husband a sly kick from me for being an insensitive lump!