I think you're experiencing a case of the "New Mommy Worries". Don't worry - I think all new moms have been there to some degree, and it will pass with no lasting side effects.

Relax!!!