Go with your instincts - tell him that as much as you enjoy his company there are times when you need "me" time