That's strange as well , for those left behind bees often go back to where they came from within two days of being there if the timing is right . .
Think of the old saying :-
A swarm in May is worth a load of hay
( a weeks work for six or so men )
A swarm in June is worth a silver spoon
( Most families had a silver spoon, it was quite valuable especially for checking fungi boiled in water for toxicity ..it turns black if it's toxic ) .
A swarm in July is not worth a fly.
The honey season is well underway, the weather is getting hot and flowers are not producing a lot of nectar or pollen that a massive nest would need . The feral nest will also have swarmed several times , each time the size of the swarm issuing isreducing by around half
Towards the end of July a swarm may well consist of a fist full of bees a mated &/or a couple of virgin queens .
There is not enough of them to make a viable nest that will over winter as there is not enough time for the swarm to build up or to get enough stores before the end of the plant flowering season ,. which tends to start in the UK around mid August .
Sometimes the queen evidently leaves the newly issued swarm of later issues to go with some drones from the swarm on a mating flight where she will mate with up to a half dozen or so drones ( hopefully not from her own line) up to seven miles from where she leaves the clustered swarm waiting for her return.
If the queen bee is eaten or killed by something whilst on a mating flightor during the flight from the old nest to the new one , the whole swarm stays put and dies unless they can be quickly re homed by a keeper on some fresh eggs and sealed brood .
When a small cluster of bees remain behind it is usually because there is an un mated queen in them & they cluster round her to try & ensure their survival . If there is any evidence of a nice white comb being built, that's a for sure sign there is /was a virgin queen who may have returned and laid a few eggs in the wild comb .
Thus they won't try to go back to where they came from , if like you say the larger party of the swarm have been snaffled by Mr Numpty there's not much you can do for them .
If you know of another keeper who needs bees to build up a hive so it has a chance of over wintering they may be interested in getting them one evening after dark and using newspaper in a super to unite them .
But in most cases it's not a viable proposition for the bees will need treating for varroa and could also be bringing in disease , financially it's not usually worth it as well .