I wouldn't think it would hurt him, from the bit you described it doesn't sound like he ate too much of it. Tommy ate part of a dish rag once and it passed through ok .

I'd imagine it will pass through pretty easily, just keep an eye on him Tommy loves to put anything in his mouth too, and I have to watch him closely when we're out and about.