It is a valid concern as certain classes of wormers have been shown to have a very serious impact on earth worms.
Can you cite your authority ?
There have been several studies, and all that I have read previously indicated some effect but none were 'very serious'.
Generally it seems to be thought that in standard concentrations there is some impact to worms in burrowing ability and cocoon production.
From memory pour-ons have more impact than injectables, and I suspect your pigs were injected.