We've not had swallows dive bombing us - ours are very laid back and don't seem to mind us working away quite close to them. Our cat was a different case. From the moment they arrived until they left, the swallows and the cat had a running battle. They would repeatedly dive bomb her, coming just a few inches above her head. Unfortunately she did catch a couple over the years but mostly they were more acrobatic than she was. Cats are awful - she used to taunt the swallows by deliberately sitting on their path to and from the nest, washing her paws nonchalantly but in reality poised to burst into action and catch one.
The swallows number one enemy now is a pair of sparrow hawks, feeding their own nestlings. We have seen them take a couple of young swallows from the air, and they have just had their second newly fledged blackbird - they are so vulnerable for the first 2 or 3 days after fledging.