If you're 5'4" and he's 14 hands you aren't too big for him at all. Ponies are incredibly strong for their size. Go for it, if you are comfy riding him you'll feel more secure.