That's exactly what we did, bought a building plot with an additional 8 acres. Great, if somewhat slower way to start, we had very young children at the time and decided to wait with the actual smallholding bit until the house was built. Took two years - house is exactly what we wanted and how we wanted it, we slowly started to build sheds/animal housing and developed the garden/polytunnel.
Now got goats, sheep, pigs and poultry...
Go for it!
Typing this with a bottle lamb snoozing on my lap...