If you're farming the land and required to be present (i.e. livestock) then you can seek permission to build, subject to an agricultural tie. It isn't a trivial task but if you really are farming then the permission is not something that the authority can decline.
Friends of ours bought some agri land from Stanley Kubrick who was quite certain that they'd fail. They didn't and when they built the house he was livid. But it was hard and involved living on site in crappy caravans until they'd proved the point that they could make it viable. Bloody good goats cheese, too.