My family has kept bees, it is amazing to see when they "swarm." That's how you can get a wall full or a great ball of bees in a short period of time. The new queen is born and needs to establish her own hive, so she leaves, trailed by hundreds or thousands of workers. Wherever she lands, they congregate. Bees in this state usually won't harm you, I've had them fly into me and bounce off.

Do examine where you saw them going in and out and seal it off completely.