Yes fence it off - google it if you can so you know what type it is - and remember some of the leaves do come off in the wind etc and will blow into a field, so if it's the poisonous type that may be a problem - azaellias (think I spelt that wrong) rhodis and the like, shiny leaved plants in other words, are all potentially lethal. They are all the more lethal when the leaves are brown and dying, the poisonous content (if you will) gets worse.
We lost a new sheep a few years back with just a few small nibbles of rhodi. Tim Tyne, the sheep book man, says in his book he also lost a new expensive tup to rhodi as soon as he got it home. Easy done.