I'd have a chat with the vendor, then depending on what he says, go and see farmer, I'm sure he'd prefer you to be upfront about it, maybe he's concerned someone may build a house there, or start bring in rough motors and leave them about as an eysore, I could understand him wanting to prevent that.
Be happy to sign some restrictions like 'access only for essential animal welfare and land maintenance' (or something like that).
Put yourself in his place, he may have simple reasons.