I agree with everyone else. Be honest, but polite about it. I've done it before. It hurt his feelings, but didn't stop being friends.