DM, you may have answered your own question !
"They are grazing as there is a lot of grass in their pen." vs dried hard feed !
As long as they are lively, moving around, drinking ok, then the chances are that what you are offering simply doesn't match the great natural diet they are finding.
No I would not give them milk, either
they are ok and just eating nature in which case milk would be a waste of time or
they are not well and milk will not help - if they are lethargic, not keen on moving round etc. - then come back and tell us, or tell us what they are and are not doing.
Otherwise relax and enjoy - you pigs are just doing what comes naturally.
Finally a thought - if you are feeeding in a trough, did they do this with their mother. Many weaners find food on the ground, and are not used to a trough.