Your new vet can request her file from the old vet. You won't have to give an explanation (unless you want to, of course) to the old vet.
At least that's how it worked when I changed vets years ago. But I also had an excuse that the new vet was right around the corner from my house.

But yes, I would definitely change vets. Good luck!