If they are older than 7 days the castration has to be done surgically (as opposed to ring them as lambs would be, and actually pygmies are probably too small for ringing anyway)and by a vet. Not a difficult thing to do, best to speak to vet about it.
However removing horns at this stage will not be an option, yes in theory it may be possible but it is a dangerous procedure, causes loads of blood loss and a real danger of getting infected/slow to heal etc etc. Vet would be unlikely to agree to it.
The majority of pygmies keep their horns anyway...