We are saying the statement was not racist. "I hate people who have Spanish accents" would be racist (sort of) but "I'm not fond of Spanish accents" is not.