I would wonder why he needed to go on - he might have a perfectly legitimate reason or simply be asserting his right of entry as owner. If neither of you set up a written agreement then you're paying for use of the land in return for money but haven't specified what other use can be reasonably made - he could decide to fell a tree for firewood or given access to a contractor and be within his rights as far as I can see unless you've agreed otherwise.
I had 9 years continuous use of land and at the outset the landowner promised to refence, dig out a secondary access to water so I could subdivide, erect a handling area/hay storage area of some kind, roofed or not.. As soon as I was in that disappeared obviously - one year he thought I was undergrazing (I was but for a reason) and he decided it looked a mess and went in, cut the long grass, baled it weeds and all and left them to rot - I wasn't amused but not a thing I could do. I later got 6 weeks notice to move out by 1 January just after taking redundancy, because he'd agreed to sell the land and knew I didn't have th cash to buy it myself - he'd told other tenants of his land I was going but not me! Gutted but I didn't want to fight over it and it was the last day before I could get a lorry even close due to icy single track hill lanes, had to walk up and down the road with each pony, load it and head back up for another - the lorryowner could barely help as was awaiting a hip replacement tho he kept the loaded ones entertained til I got back. I cried over it and miss having the space, it has made every winter since very hard, but I didn't have a written agreement and with or without the established tenancy (and I was in pre-2003) it was by then so badly fenced there were security issues arising, and it wouldn't have been beyond someone to make my life very difficult, not least as there was a large capital sum involved..
If you've not long been on the land, I would suggest very strongly that you go and ask without accusing, take it very gently, assuming you want to get an extension on your lease anyway.. Most farmers do things for some reason, whether we know or agree is another matter, but everyone deserves the right to explain. Personally I think if you padlocked him off his own land, the average farmer would just take boltcutters to it and/or drive over the gate in said tractor, and you'd not have the lease very long..