It may be just his personality, just a lot of time and care may make him realize you are no threat to him, but as he is a rescue, we don't know - he may have been hit by human hands before, which is why he's skittish and defensive.