At the end of the day, they are your goats and its up to you, who has them, and at what age, so don't feel bad.
Personally, most of mine are removed from their mother after birth and I bottle feed them. I then know, especially with multiple births, how much milk they are getting, and the smaller ones do benefit from bottle feeding, as sometimes they can get pushed to the back of the queue by siblings if left with mum!!
My exception to the rule is one of my smaller diary goats, Molly, who is an exceptional mum and always does her kids well, and adores feeding them.