The worms in the compost heap are different to the ones in the open garden, so if it's brandlings you have (darkish and vaguely stripey) keep them in the compost bin or wormery, not the garden. Garden worms, of which there are lots of varieties, tend to be bigger and they are the ones that make long holes which help aerate the soil and down which they drag decaying vegetation. If you add lots of organic matter to your soil you will find that your naturally arriving worm numbers will increase.
We used to have allotments in Edinburgh where the New Zealand flatworm had been accidentally introduced. It was a terrible pest and eventually it got to the point where seeing any normal worms was an extreme rarity. The NZ flatworms would hunt the bigger worms, chasing them down their holes, wrapping their bodies around them and sucking out their juice. Some research was underway at the Uni and they found that certain beetles predated the flatworms, so there was some hope that eventually a status quo would be reached. When we moved here we didn't bring anything from the allotments with us in case any flatworms hitched a ride, then we discovered that they are already in our nearest village
I am hoping they don't make it up the hill, as we have a wonderful population of compost and earthworms here. Without them, if you dig manure or compost into the soil then look again the following year, it is exactly the same and has not broken down or been incorporated at all - weird !!