After the first refusal I wouldn't have given him a chance. Hard right leg (or even stick on the croup as well) with opening left rein. I guess he didn't feel balanced enough with that short approach around the first fence.

I just have a question, do you ride jumpers or hunters or both? I'm guessing jumpers from the courses and the colourful jumps.