Plums, what sort of fencing do you have? Really needs to be 4ft tall to contain goats, although I used to have sheep netting, and two strands of plain wire, connected to next doors electric supply, at my old house, and had no problems. Thats not high enough to hold this gang of goats in - its plain sailing to get over sheep netting for them.
Is there plenty of grass on your side of the fence? Ours is very slow growing, with the lack of rain. If shortage of grass, the goats will try and leave home. If thats the case, leaving hay out may stop them going off.
I can sympathise, as can most goat owners, its part and parcel of goat owning, trying to stop them escaping!!!