I'm a bit out of my depth because I live in Scotland where the whole sale is handled by a solicitor, so they would be dealing direct with the seller's solicitor and would learn all the details from source. The situation is different in England - I assume that is where you are
I think that if the status quo carries on unchallenged from one owner to the next then it may simply become fact, with no-one to gainsay the neighbour, and you have effectively lost 65% of the land you have bought. Can you make it a condition of purchase that either the land is sold to the farmer and you renegotiate the purchase price, or that the current agreement ceases with the purchase? You could then renegotiate an agreement, a legal agreement made up by a lawyer who understands land purchases if you so wished, or use the land yourself, or offer it for tenancy to someone else on your terms. If you do the latter, then you can make sure that everything is set down legally, with a time limit and details of who looks after the fencing, who limes the land, who owns the crops, how often their livestock is wormed, that the land must be kept in good heart and so on. If this cannot be sorted before you buy, then don't buy - you would be landing yourself with a big stick to beat you.
I agree with Sally that you need to know and to understand the situation. At the moment you seem rather fuzzy about it, but land ownership and court cases over just who owns what, have been the bread and butter for lawyers since long before King Alfred was burning cakes. Land ownership squabbles have also made families penniless - sort it out before you buy, or choose another property if it's not quickly resolved.