I don't really understand how it would be so awkward. I mean, if you don't know the person you could just say "I haven't had the chance to get to know you better but you seem like a nice person!" or something like that...

But if you don't want to participate that is of course your decision.