Yes, I would only keep him in his pen when you aren't home. Being a Lab, actually most any kind of dog, what he wants most in the whole wide world is to be with you!

Our family's dogs were always allowed to sleep in whatever room they chose, and Sheba (best smartest dog in the whole wide world ever) would "make her rounds" and quietly check on all sleepers several times a night!