We usually remove old swallows nests as long as nothing else has moved in for the winter. This is because if there are red mites or other parasites in there they will wait until spring then infest the new hatchlings. Mostly our swallows prefer to start again each year, the only exception being when we had a really dry spring so there was no mud to build with so they re-used old ones and added straw.
For wasps nests, they are usually suspended from the top, unless the sides are touching something as well in which case they will adhere to that. We have scraped them off with a paint scraper, very carefully because of course they are so delicate. I have wondered if spraying them with hair spray would help or just make things worse. You have to be very careful to lift them with flat hands, not poke through with your fingertips. It's probably also a good idea to check there's no-one overwintering inside

(especially before putting it in the loo)
They are so beautifully constructed that it would be wonderful to prepare one in cross section - must be possible because I've seen them done like that. Do send us a piccy of yours on display once you've done it