Glacier's advice sounds right on the mark! Where did the pup come from? Was she with another family first? It certainly sounds like he is afraid of males..do you have a beard by the way...? I know it sounds strange but I have heard stories/myths? of dogs not liking men with beards because they are afraid of them. (The beard makes the crucial "face reading" features more blurred and they are not sure of your intentions) It's probably just a myth but...you never know!

Enjoy your new puppy and good luck!!