My first cat, RB Amber, was an outdoor kitty I rescued and brought in. I took him out on his harness daily - he insisted! He knew that the only time he was allowed out was with the harness on. Point: I don't think you need to worry about taking your cat out on the harness.

You say he can escape from the harness. THAT is not good, you should find a harness he can't escape. Try the Rope N Go harness:
http://www.ubraidit.com/supplies-thumbs.php?cat=81

For darting, keep a water squirt bottle near the door. You will soon learn to pick it up before you open the door to allow the dogs out, and then you can deflect him (with some quick action, lol) You should also squirt him ANY time he goes near the door, even if it is not open.

Is the yard fenced in? You can make it more secure, so it is more difficult for the cat to get out of the fenced area.

As for other folks letting the cat out, that is a huge concern. Can you put the cat in a bedroom when folks are over?