Perhaps he just needs a friend. Be honest and tell him you just want to be friends but I think it is mean to just shut out someone because they are different.