Personally I'd talk to the farmer directly and get his view. If he believes he has grazing rights, and you can't live with that, don't buy the property. If you have to get your solicitor involved to get him evicted, you've already alienated a neighbour before you move in.
If the vendor has misrepresented the situation then s/he, and potentially the estate agent too, are at fault; you may be able to reclaim any costs you've incurred thus far.
I do hope the farmer's a nice person and it all gets resolved amicably