Well, since you said the "average" guy, not a model type guy I had to vote no. I do not want to see a chubby guy, or a super skinny, boney guy without his shirt on, thank you.
There is an older man (I would estimate him to be in his 60's maybe) that I see jogging on occasion and he looks very good without his shirt.
So for me, it totally depends on the shape of the bod we are talking about.