I have pygmy x and the males, whether intact or castrated are always rather aggressive. Mine are fathered by a big sturdy pygmy - not registered and probably overheight. They actually grow well, and by about 8 months are ready for killing. So I would think putting yours in the freezer is the best option.
Does he smell, as in, like an intact male? If he does, just thinking that may taint the meat? If not, then cannot see a problem.
Oh, and goat meat is very nice - nothing like lamb, as people tend to think it will be