I totally understand what you are going through. My dad threatened to hold me back in the 6th grade because i was never good at math. Math to me is like a really hard foreign language! just dont get it, and still dont get it. He would always yell at me about how stupid i was, didnt matter how much i explained to him i didnt get it.
He compared me to my other friends who were doing so well and his come back to me always was "well if it's so hard for you then why does *so and so*-friends name inserted- get all A's or B's or whatever!
He just never understood that i just didnt get it.
Im really sorry your dad said that to you, he should have never said that to you. I totally understand.
Of course i was the complete opposite of you though, since he thought i was stupid i acted stupid (so i was called alot worse things as the years went on) just because he hated the fact that he had a daughter that was dumb and since he hurt me, i hurt him.
Best of luck to your math and if you ever need to talk about your dad just pm me anytime!