I think rather than "swatting", you should probably try to give a firm "No" or other appropriate command (I know some people don't like just "no", but rather "No Nosy" or something like that). Then, redirect his attention in a postive way with a toy or something, and then praise, praise, praise! I am a real believer in positive reinforcement.

Good luck.
Logan