I don't think you can say she has never scared him, he is making it clear to you that that is not true. It's a very bad habit to start off pet ownership by denying or ignoring the messages your pet is sending you.

Toddlers are often noisy and have chaotic, random movements. If your niece doesn't live with you she carries strange scents with her too. It's very common for cats to be frightened of toddlers, especially at first. Although you may supervise her very well, many people don't, and kittens end up with broken necks, legs, tails, and internal injuries from toddlers fairly often. So their instincts are natural and correct for the most part. These are all reasons that animal behaviorists say people with preschool children should not have kittens.

It is harmful to punish cats for sending out warning signals or defending themselves when they are frightened.

You may need to distract him with interactive fishing pole type toys when she is around. He needs to have high places where he can go to feel safe and observe her behavior, such as tall cat trees, bookcases, tall dressers, and the like. And sometimes you may need to have a room ready with his accessories where he can stay while she is around.

The book 'Think Like a Cat' has a chapter on cats & children, as well as much information on how to interpret the behavior of your cat and effective ways to manage it.