You are doing the right thing, letting them get used to you. They've had some harder times, so it's not unexpected that they'd be wary at first. Just keep "being with" them, maybe figure out what kind of toys or treats they like and bribe them nearer you every time. They're so pretty, and look so soft, I know in time you'll be able to pet them.