You just keep doing what your doing! You are doing a great job!!! When you clean do everything slowly and talk to him in a quiet voice the whole time. As long as he is in the cage he will probably use the litter box as a bed. They seem to do that for some reason!
He will eventually look forward to your visits. He is young and it willl be easier than if it was a full adult but he already has the feral instincts so he might always be a little skittish but I promise he will come around. Eventually you may have to force the hands on but for now the slower the better.
Great job! I'm so glad you are helping this little guy. I know you will be posting pictures of a lap baby someday in the future!! I can't wait.
Keep us posted!!