Yes - goats (and sheep, lambs, dogs etc etc) eating layers pellets would be your biggest problem. There must be something in these that makes them irresistable to any farm animal (maybe I should try them too!!!!), but not recommended.
Also if any laying hens have access to the goatshed - expect eggs in the most unsuitable places, I visited a farm where one of the hens was laying her eggs in the hayrack, and they ALL fell through the mesh... unless you got there very quickly! Just makes a mess and probably encourages vermin too.