It is absolutely impossible to give any specific advice without actually seeing your set-up, discussing your management plans (time available, what finances you can put in etc).... and what is it you really want to get from our goats? For fibre goats you will need to develop relationships with other producers to make sure your fleeces get you some return. Wrt housing and fencing you are probably looking at similar set-ups, just take horns into consideration for fibre goats more so than for dairy goats.
Dairy goats in particular need a good daily (365 days per year) routine wrt milking, feeding, breeding & kidding... so unless you can provide that maybe dairy goats are not the answer. Also - what to do with male kids? Any breeding enterprise will need to tackle that problem BEFORE you get any animals for breeding. Even with fibre goats, once castrated boys get to the end of their quality-fleece lifespan they are normally eaten to bring in some profit.