I would go with the honesty bit - you can't go wrong by being honest and letting him understand you prefer to have others along when you see him. If he takes it the wrong way then you'll know he wasn't for you... very simple