I would be honest. "My personal feelings toward horse racing won't make me very good company at these races." I had that happen to me a while back too. Our boss wanted to have a company picnic at a race track. I told them not to expect me if that was the venue.