I read that when a bunny makes noises that it means they aren't comfortable, but I'm not sure..

I don't think it's cruel to hold a bunny! I think it's a good thing to help you bond. And if he doesn't seem bothered by it, it's in no way harming him.