We have had lots of experience with wasps, in our first home we had a wasps nest on a tree near a path, which was removed by the council, the next year they were in our wooden garage about 15 foot from previous tree, we got rid of that one, and the next year we noticed nest on neighbour house in barge boards/soffits. So I would say that they come back to a similar area every year.
With regards to gentle wasps. In the new house I cut back a HUGE pyrocanthea (incorrect spelling) and discovered 2 wasps ( and they looked very much like wasps rather than bees) in the dead leaves, as I raked the leaves back there was the paper wasps nest. I had to smoke them out as the kids kept cycling too close to nest.
We recently had a massive influx of wasps not too far from the hives, they landed on a willow tree which was covered with blackfly larvea and proceded to mingle about on the leaves. They have gone now, but we were worried as last year we lost a hive ( before we moved) to wasps. Not sure where the wasps have gone, but the hives seem to be OK.
We also have mortar bees, which have caused the chimney to crumble off (bricks resting on thin air) so whilst they are OK in small numbers over a protracted period of time they can cause quite a bit of damage. ( We have replaced lime mortar with a lime cement mix). But the mortar bees are vey obvious in their 'buzzing' of the brickwork.