I have some sympathy with that view Sally!
Which is why altho I dont understand why some people dont wear hi viz, I wouldnt be in favour of legislation mandating it as that would make vulnerable road users (ie non motorised ones) effectively responsible for stopping speeding drivers from hitting them, rather than drivers being responsible for driving in a safe and slow enough manner to be able to see people/children/animals and to stop if something runs across the road, let alone another road users a lot slower moving in the same direction.
On the hat front, there might be more of a case for legislation altho it is already compulsory for children (I think under 14) to wear one on the roads. Wonder if the adult exemption was a wish not to turn into a nanny state or whether it was more to do with the fact the Queen rides hatless :-))
NB the Highway code already recommends hi viz in gloomy conditions and lights in dark, so altho it's not prescriptive in the way a 'You Must' clause is, it can already be used as an argument for contributory negligence in the event of an accident. But the vast majority of serious accidents involving horseriders are not because a car didnt see the horse and rider, it's because they were racing and didnt have the patience to wait a minute or two for the rider to reach a safe passing place, added to a total ignorance about the likely reactions of a horse to having a car revving its engine and driving up to within an inch of its hocks.