This happened to me all the time when I wasn't wearing earrings. I finally just went out and bought sleepers so I could wear earrings all the time.

With some people they never close up but with others it doesn't take very long. If it's really sore try using a salt water soak on it and something like tea tree soap or a clear soap like Pears.