I am sure it doesn't harm him. I have known another kitty like that -he was absolutely ok.
BTW Filou left his mom when he was 14 weeks old which was not too early. However he sucked his younger (but bigger) adoptive brother Tigris' fur for at least half a year before he decided to be an adult