A couple of ideas - hope they help.

If you want to play using your hand, get an oven mitt. Trust me, I know from experience that they work.

If it would be better to avoid this kind of play, try a feather stick or string or other toy that doesn't involve getting your hand near him.