Thanks for the suggestions. They're gone now.

I just kept spraying the holes a couple of times a day, and that was enough to kill them or drive them off. They kept trying to get back in the holes they had started, but would fly off. I saw a couple of them floundering around on the porch, so they must have tried to bore out the holes some more, but the wood was too toxic and it killed them. I only saw one determined one yesterday, but it would check out the hole and then fly off.

Pests!