Good you have a butcher and customers but research your area is there a slaughterhouse near that will take goats or do you have to travel 100 miles to one, also what are their charges -adds to cost.
Boer is the meat breed and dairy x boer preferred dam but also some people rear dairy or dairy x boer male kids for meat from dairy herds from a week or so old but obviously not such a good carcass as pure boer but lower cost than setting up breeding herd from scratch. That could be an option to start with milk replacer creep, hay browsing, grazing, to try it out before committing to breeding herd.
As others have said goats need access to shed all year round and hay /straw as if raining not eating. Higher vaccination / wormer costs as have different dosages to sheep often double dose for some. Electric fencing is a must. Most goats usually have twins quite common to have have triplets / quads -especially AN. They will de bark trees!
I read an article -boer goat breeder selling carcass straight from slaughterhouse for £6 per kg. (south England) I know a breeder who markets meat direct to public averages £10 per kilo.