I would call an exterminator. If you can get a recomendation that is the best way to find one.

He will be able to get rid of them and may be able to let you know if you need to be worried about anything else.

I don't think it is cheap though.

I had one come out a few years ago to take care of yellow jackets in the attic and it was about 110. However, I did not have to worry about them anymore.