You are doing nothing wrong my boys and girls live outside until they are nine months old and my killing weight ranges from 35-47K which in my pig keeping book that I have is fine a little fat for flavour but good quality meat. I did start to think the same as you, my pigs have wheat, barley and yogurt the dairy I have learned is important, all the little fat weaners you see when buying are usually given the milk from the cows on the farm. You are doing everything right, what breed are they?.