No idea what definition of permanent pasture is, but shouldn't be an issue with conversion for horticultural use. It is not a "change of use" for planning purposes. That said, if land is adjacent your residential property, the planning officer will be wary of anything that seems like an extension to your "garden".
I bought 3 ha of pasture land ever-since used for grazing/hay (albeit reseeded many many years ago) to convert to horticultural use. I submitted a "Do I need planning permission" request: the response said it involved a "change of use" which would, in itself, require approval, but that advice was incorrect and I responded accordingly. (I never received an acknowledgment!). When it came to the actual planning application for support facilities there was not the slightest reference to a "change of use" by the Planning Officer, which is as I expected.