Ram lambs that are fed up with concentrates from a young age do not have a rumen that works as efficiently as those that have been fed a predominantly roughage diet. The rumen just does not develop to be an efficient converter of roughage, and so when put out on grass alone these animals rapidly lose condition.
I've had it happen with a couple of rams I've bought in the past, and the one I have now who is running with ewes. He's only had 7 ewes to deal with and has been fed the same as them throughout the winter but he looks really poor, whereas they all look fine.
I find the thing that puts condition back on better than anything and long term is spring grass. Those first few months of growth are far more nutritious than the grass will be at any other time of year. It's high in energy and feed value and it's ad lib. I would feed him enough now to keep going on, as you have been. But don't overdo the concentrates. You want his rumen to work for itself as it should. Maybe give him some natural yoghurt, or rumen stimulant. I'm sure you'll have wormed him and done all the obvious things and as soon as the grass comes through, you should see a big improvement.