Try going back a stage or two. If he will only let you go two feet away then go one. If he will only stay for 1 minute return in 30 seconds. Also always vary it. Move 6 inches away, come back. Move twelve inches away come back. 10 inches, 5.....etc. Also vary the time. 20 seconds, 10 seconds, 30 seconds, 5 seconds...etc. You don't want to give him the opportunity to break his down stay. Always try and return before he breaks it and always throw a mini party of praise when you release him.
If one of my dogs broke a stay in their training I would say something like "uh-uh" and walk back and gently put them back in position. Then I would re-give the stay signal (but not the verbal command) and just take one step back and count to about 5 slowly, then I'd step back and give them their praise party and release.
Again, if you are finding you are consistantly having problems, then just step back a stage.
My dogs have very solid stays, in fact if one of them ever broke a stay in competition I would probably faint from the shock of it, LOL

Good luck!