By all means, separate your cat from the toddler when she is around. Cats are very independent animals and don't always react well to the sudden movements of small children, even if it isn't directed towards the cat. I always tell all visitors at my house to just lightly pet my cat and move on. She is unpredictable. It sounds like this cat has given you the message that he is not happy, so heed the warning! Better safe than sorry. Good luck and don't get mad at the cat.