I would just tell him that you're not interested in hanging out one on one with him. I personally don't think you owe him an explanation of any kind. Just tell him that, in case that what he's interested in, you want to be up front and let him know that it's not in the cards.

Especially if you work with him, I'd be careful saying that you're too busy, or not on the market . . . because if he ever sees you dating someone (maybe even a fellow employee at some point), that could lead to an uncomfortable work environment.