Yep, that's the problem. You can't just add a dwelling to a plot of land, you need planning permission for the dwelling, which means more money (to buy a plot with a dwelling or pp for one), or taking a long approach.
So one way to start is to get bare land nearby to somewhere you can afford to live, and build up the holding without living on it, to the point where you can demonstrate the need for a dwelling and the viability of the holding to sustain itself.
And you can start to build up a holding with livestock (you need livestock to demonstrate a need to live on site, really) by renting odd bits of land, you don't necessarily have to start with buying land. Although many landowners will be wary of renting land to inexperienced folk, so maybe before any of that, get yourself experienced with livestock.
One Planet Development in Wales is a bit different to the approach the other countries take, might make the process more suitable for you.
Have a read up of all the Chapter7 stuff on the This Land is Ours website too.
linky