I'd give him an ear mite treatment anyway, unless you are absolutely positive that he doesn't have mites. They'll look like white specks in the ear crud in the q-tip.

Some cats just have dirty ears. I had to clean Havoc's ears twice a week for quite some time until I switched ear cleaners and it finally cleared up.