Hello and welcome from me in Derbyshire!!
I can sympathise re your planning problems. We are in green belt land here, plus an area of outstanding beauty, which makes anything we want to do exceptionally hard with regard to buildings of any kind. But funny enough the same planners allowed an agricultrual field not far away to be turned into a Tesco ....aaah.....that was a BROWN field site though. Well, to me it was a field, same as ours. (not that I want a Tesco of course)
The problem is, that horses are domestic animals, and thats why you need change of use etc. I have horses, and find it hard to get stables for them on my land. I do have lots of hen sheds having a lot of ex battery hens. And yes, fruit and veg growing in a field are wrong it seems - you are extending the curtailage of your garden. Someone I know did exactly what you did re planting fruit and veg. He put a post and rail fence up separating the field from his new orchard, and then got in trouble as the planners did not like either the orchard, nor the fence as it was not in keeping.
There is no pleasing some folk is there? Good luck with your planning issues.