I agree, creating a garden with lawn, flowerbeds etc might be pushing your luck, but I think growing a few plants would be okay. Planting trees is definitely okay on agricultural land, as is creating a meadow like a wildflower meadow, for example. It's seen as boosting biodiversity.
It really comes down to individual local authorities and how nit-picky they are, and nosy neighbours. After all, if no one puts in a complaint, I doubt the LA will look into it.