We do it on day 3 (they get their iron jabs then, so we just get it all over with in one go

)
Echo what Robert says about not letting them out - we keep then in til the wounds have healed before letting them out, usually by day 7
If you think it's bad with big pigs - you should try finding a Kunekune testicle at that stage
We have a friend who wanted us to show him how to castrate them himself, once he realised it was like trying to find a grain of sand in a purse he understood why the vet does them at 6 weeks old normally
If you decide to have another go, hold the piglets so their bums are pointing downwards - gravity should help them 'drop'

The welfare regs state that if it's done after day 7 of life it should be carried out by a vet.
HTH
Karen