I'm sorry to hear you had a distressing experience. We all do, and like you all try to make sure it is only the one time.
I don't think you can make the gap too small for such a small bird - I have it at about 1/2 centimetre for full-size birds. The jaws aren't sharp, so it can't cut the head off. It works by the jaws pushing the vertebrae askew, snapping the spinal cord.
Make sure you have it just a little way down the neck from the wattles and aren't catching the wattles in it.
You can do no harm to double-check the bird is dead by pulling either the body or the head before releasing the jaws. Pull at an angle, then you are doing the same thing as the broomstick method. You may get more blood pooling into where the jaws have been - I hang them up by their feet for a while to let the blood collect in the neck before I gut them.
A vet once told me to check for an eyelid reflex - if you pass your finger across the eye, close but not touching, and the eyelid flickers, it's not dead. Conversely if the eyelid does not flicker, it's dead.
Hope that helps and you feel more confident about the next ones.