I'm no expert in this either but wildflowers won't be able to cope with "garden" conditions and a typical ryegrass sward - they will be out competed by the ryegrass. I think you're spot on that you woudl be better to remove the turf if you are going to sow seeds. Another alternative would be to cut the area very close and remove the clippings for a season to starve the soil so that the wildflowers, that are better suited to harsh conditions, can outgrow the ryegrass. You can buy wildflower plants that you can insert into the sward - they woudl have a better chance of establishing than seed.
The folk across the track from us oversowed a ryegrass field with a horse mix and you wouldn't know the difference.
We have a wildflower area but it's in the wrong place. It has gowans (big daisies), violets, yellow rattle, loads of land cress - it looks great for about two weeks then looks awful. The hens love it though and their scratching has undoubtedly improved it by scattering the seed.