If you give goats "free" access to any tree - they will destroy within a short period of time. There is nothing wrong with keeping goats in a non-grazing setting, i.e. a yard, as long as you are prepared (and able) to provide greens - preferably freshly cut branches from hedges and trees for most of the summer and into autumn. By all means grow willow for your goats (ours particularly love the hybrid willows you get for growing as firewood/fast-growing hedges etc) - we do that, and it works well - OUT OF REACH FOR THE GOATS. You do need to go out cutting most days - and you should be "legal", as in only cutting hedges with the owner's consent. (and not besides heavily used main roads, where pollution may well be off the scale...)
Goats also love lots of fruit and veg - carrots, cabbages, kale (all three good for winter as replacements for branches, though my goats even love to de-bark willow branches without leaves...), bananas, apples, courgettes... the list is quite long. I would not include citrus fruit however (or anything else with high acidity).
"Readigrass" (sold for horses) is also a good "treat" for goats in winter, especially milkers running through.