Yes and unless he is castrated the meat will be tainted with billy too...
However I know lots of people keep their boys in the same shed as the girls (behind sturdy metal hurdles though), and the boys go out at night time. It seems to work and haven't heard of any issues re taints in milk.
Even if he doesn't get the female kids into kid, he will still constantly bother them, I have a couple of wether kids and even they jump (their sisters mainly...)..., so both my boys will go as soon as reasonable size.
I am planning to keep a couple of boys intact next year, but take them off the nanny and bottle feed in a separate pen, even away from the female kids form the start.