Sounds like you have figured out part of the problem yourself! Puppies will remember abuse like that for a long time My Maggie is terrified of large, bearded men that smell like cigarettes, and we have had her for almost 2 years now with NO abuse tolerated. Just make sure when you socialize the puppy, that other people are aware of the prior situation and introduce themselves slowly to keep from freaking him out.

BTW, absolutely ADORABLE puppy