Really sorry to hear about the trouble you're having. But I have to agree with a few of the others. I think it would be the biggest mistake of your life to up and quit a job you enjoy and are looking to take over when your friend retires. You said that all your friends are there, so you seem to have a lot of moral support. And Cyber-sibes is absolutely right. You will always meet "that" same person no matter where you go. I don't know how old your aunt is, but have you thought of what it could be like if she wasn't there for you???
Travelling for shows is nothing compared to a move like you're thinking of. Maybe wait and sleep on it, have that supper with your boss, and remember, wherever you go, you'll always have some people that will annoy you.