I didn't "let" him do anything... by the time I saw him, he'd already been scoping out the whole scene.

That must have been what happened.. He thought "Now I've seen everything!" and departed for wherever they go.