I think you did the right thing. I don't think you were too hard on him. It's not like you tore into him or made fun of him for forgetting his wallet. You just simply said you weren't comfortable with it, and that's more than most people will do anymore. I think you were honest and that's what matters the most. I honestly think if his intentions were good, he'd have emailed you and apologized. I'm sure he's embarassed, but a gentleman who'd made a mistake would certainly have written to apologize and ask for a redo.