It's perfectly OK if he's not interested in food every day. Are you feeding him live food or pellets? If you're feeding him pellets, I like RepCal pellets best. For live food, I recommend minnows(mosquitofish), or big guppies, but I really recommend against feeding goldfish too often, as they can carry some pretty nasty diseases.

You don't sound horrible, you sound like you're trying to do the best for your new buddy!