Most native hedgerow woody plants are fine, don't think I've read of any native trees/shrubs that are bad. Hawthorn, blackthorn, bramble, ash, willow, sycamore, beech, etc., we have elm hedges here, and Rowan too. Rosebay willow herb is great too.
Woody (or is it deadly?) nightshade sometimes grows as a vine through hedges, think it's toxic, but don't know how much they'd need before it is harmful. Same with ragwort, sometimes grows through hedges, but they need quite a bit before it hurts them, something like 5-10%body weight or more (so avoid, but don't panic if you accidentally pick up a few leaves with whatever branches you collected.
An important one to avoid is cherry - it's fine fresh, or dry, but wilted has toxins. I imagine it would count as fresh if you cut it and goats ate it within half an hour (?) but I don't know how long it takes for the toxins to develop, and subsequently leave the leaves and would hate to think of you sneaking branches over the fence for them to find and enjoy later and get sick.
I'm cutting and drying bunches of branches for wintertime forage for my goats, you might like to think about this as they love green stuff and variety come January.