I kept black welsh mountain for years and never castrated anything. All the sheep were always run together, though one year when I was short of grass and they hadn't come on as well I split the males from the females for extra feeding.
Never had any problems with any fighting or mating as the senior ram simply kept the younger boys away from the sheep, not particularly by fighting just making sure he was standing between them. My fields aren't that large either, but everything always worked well
I did have problems getting anything up to size in the early years till I discovered I had a mineral deficiency in the soil, after that if anything was not thriving I got them Selenium and Cobalt, and also treated them homeopathically to help absorption. Also later on I kept seaweed meal on offer and that seemed a good help.
My first thought would be have the soil checked for mineral content, and take it from there, plus if you want to really speed them on feed concentrates at judicious times, and, of course make sure they are getting plenty of grass, and they are not having a worm problem
All the best
Sue
Dark Brown Eggs
PS I have eaten entire ram lambs at most stages and times of the year. There is definitely a stronger flavour to a well grown ram, but I personally don't find it "tainted" though I wouldn't slaughter a young ram or ram lamb in the breeding season.