Bumble bees nest in the ground. The queen will have hibernated through the winter in that hole, then started laying eggs. As they hatch, she forages and feeds the new bees herself. When those are ready to work, they will crawl out of the hole, which is what you are seeing, and the queen will stay underground laying more eggs. They never have huge colonies though, like honey bees, just small families.
Please please don't try to kill them, and you won't be able to chase them away without killing them. Bumble bees are struggling, as they have for the past few wet summers. I don't think your pullets will try to catch them - the buzzing should put them off. Bumble bees rarely sting unless their life is endangered.
How about putting something over the hole, big enough to prevent you failing to notice the nest and stepping in it? I would suggest a wooden box with wire mesh over the bottom, placed top down over the hole. This would be four vertical walls, an open bottom, and wire mesh over the top. The bees could still get in and out, and they would set off flying from a foot or so's height, so they would be less likely to crawl before they flew off. They prefer their hole entrance to be a bit disguised anyway.
Bumble bees tend not to last for many months, then they all die except a new queen or two, which fly off to find their own hibernating nest. They don't come back to the same nest the following year.
I hope you don't mind going to a little bit of trouble to support these wonderful creatures