He doesn't want to go past the house but he does want to go to the backyard of that house? Did I get that right?

How old is he? Is he neutered? There maybe an unspayed female at that house and he smells her in the backyard. If he's not neutered I'd get him neutered as soon as possible.