Yes you should be worried if you haven't seen mom cat for days. Something could've happened to her. The kittens will die if not kept warm and fed every couple hours. They need to go where someone can feed and care for them.