It sounds like he is marking his territory. Use an enzyme cleaner like 'Outright' to clean up. Or plain vinegar. If you can catch him at it, take him outside IMMEDIATELY and tell him no in a stern voice.

Eventually, he'll get it. By the way, females mark too.

Mary