When I've used that phrase to refer to men?

Easily.....American Legion meetings, Union meetings, work, ..........

and as Karen can attest, I'm not alone in the use of that phrase to refer to a behavior, regardless of gender.

As to the "expert" comment, I'll leave it alone, as we're not even in the same ballpark in regards to what constitutes an expert.