The 3 apartments I lived in, we generally waited 10 minutes and then removed other people's stuff to keep the line moving.

I didn't see that anyone complained to you or anything, so just keep going!