If you want to just chase him away, have a plant sprayer water-filled ready. If he appears spray it at him, or better still a hosepipe. It's rare for males to be unneutered these days.
But if you'd rather find out where he belongs and return him, contact cats protection - they may be able to leave a cat trap at your gate for you. Then they will take it away if you catch him. Or they may know of him or have other suggestions
He may have strayed follwoing a female's scent, but again it's unusual for females not to be neutered too.
Good luck either way, not nice to have your own animal family disrupted