I would think that swallows are unlikely to abandon nesting at the place they know. They are though likely to abandon a particular nest. There could be any reason for this, including that bees, wrens or something similar have taken over the old nest.
They could well be choosing another nest site and are starting to build.
We have three pairs here now and one pair have very excitedly decided on a new nest in a lean-to. Frustrating for us as this was the year we intended to demolish it
We usually knock down any nests in the poultry houses during the winter, because swallows and hens share red mite as a pest, so if the mites overwinter in a swallow nest, then any nestlings are likely to be overwhelmed by the blood suckers. We also knock down nests which have been predated, because predators have long memories and would just come back the following year - making the swallows choose a different site and rebuild from scratch seems a good move. Also predated nests have usually been half destroyed anyway.
[member=3211]Anke[/member] don't give up yet! Our swallows arrived late last year, well into May, but still managed two broods each.