It really depends if your meat is for home consumption or you plan to sell. We now cut up our goat carcasses ourselves and therefore decide on the size of joints etc ourselves too. It doesn't matter a jot if they are on the smaller size, as most GG wether kids are. But I have found the BT (8 months) kid last year was 50kgs liveweight, 26kgs deadweight, and we got about 13 kgs of meat (all boned except the hind legs), plus a good amount of fat we rendered for soap.
So if you want only meat for yourselves you should be able to continue too use a dairy male (maybe you can find one that is a bit bigger but not too much) to increase the size of your stock slowly. But if your business plan includes goat meat sales, then I think large dairy goats crossed with a Boer may be better. AN x Boer would be ideal I agree.