I know what its like to have neighbours like that, and also what goats are like, so you have my sympathy. But, I think I am right in saying that you are responsible for fencing your stock in, using adequate fencing (whatever that is, with goats, I am not sure there is one!). To be honest, I doubt you can make the neighbour fence his land against your goats.
Your problem will arise if the neighbours start to take you to court or something, claiming the goats have wrecked their garden, eaten some expensive shrub etc.
I fully understand that fencing goats in is a nightmare, but I think maybe I would try and make sure the fence is secure, because these people sound like trouble, and you do not want any further hassle from them. If they have got the police involved, they obviously have an issue with the goats.