By clearing away the fruits that have fallen and are no good to you, you are forcing the wasps to get onto the better fruits that are still on the tree.
By the time the wasps are eating fruit, the nest is no longer producing young, so you are not increasing wasp numbers by letting the adults eat fruit. These adults are the workers who have reared this year's hatch, and will die once the sugary food sources give out. So it's better to leave them some old fruit you can't use, or they will find fresher fruit to eat.
(The young wasps hatched during the year were fed insects by the worker wasps when they were larvae, then they pupate, hatch and mate. The males die and the fertilised queens go into hibernation. While feeding the larvae, the worker wasps were fed by eating the sugary waste the larvae produce from processing the insects' chitin. Once the sexual brood has pupated, the old queen stops laying, so the workers' food source is not available any more and the worker wasps have to revert to eating nectar and any other sugar they can find.)
So your non-pesticide options for keeping wasps off your fruit are :
- let the wasps have the windfalls and damaged fruits, and pick the good ones for yourself daily or twice daily.
- trap the wasps with sugary / jammy water in a jam jar, suspended from the branches, with holes in the lid just big enough for the wasps to crawl in. Empty and refill every few days. They are adult wasps who will die anyway once the food sources give up, so I s'pose it's not really that bad to trap and drown them :/