Maybe you should leash train your pet and stay outside with him when he has his outside time. Are there laws in your area about stray cats? It sounds like he isn't neutered. Is your cat neutered? I hope everything works out!