Ballingirl is right, and I also find this true with my horses. Once they are castrated, they are not as stressed and tend to put their energies into eating!!
My friend had a castrated Anglo Nubian male. I have never seen a bigger goat - he was as big as our pony. Although at the opposite end of the scale, I have never seen anything as small as our castrated male pygmy. He is nearly 6 months, and is tiny, even for a pygmy, yet his parents and siblings are normal sized.