If you bought him as a kid there is no way that he can be proven, and you would not have any come-back from the seller, as obviously s/he cannot sell the boy with any guarantees. If you buy in a kid than that's one of the risks you take, for probably a lower price than if you bought a proven older guy.
The only possibility would have been if you bought the result of a polled to polled mating, as he could then possibly be a hermaphrodite (look like a boy but be infertile). However you would know about that from studying the pedigree.
However I would not despair just yet and do what Jaykay says, after all if he is mating adult girls he may simply just be too small and not get to the right place. Did you observe him mating the girls - did he do the final thrust motion (sorry it is not meant to be pornographic...), but if he didn't than most likely he is not doing it properly.
But for this year it may be best to get in another boy and let the wee man grow on a bit and then get his sperm tested next autumn. (if he was vaccinated with anything, possibly Schmallenberg he may have been temporarily infertile, same thing could happen with toxoplasmosis infection. ). Do you haven any kittens around the goathouse/hay storage?
I personally would not let a male kid run with females all the time, as these wee boys are VERY keen to get going - he could be working himself out/exhaust himself quite quickly.