I think you'll probably need both sand and compost... I once worked on a farm where the veg garden was pure clay. They brought in a trailer load of sand most years, plus added loads of muck; that improved drainage and fertility. Of course, on a farm scale a trailer of sand was nothing - we got several of them each year for the tracks alone...
I would maybe suggest you try for raised beds, if that is feasible. Then you have a smaller area to deal with, i.e. you need less sand, and can concentrate on growing those things in there that need very good drainage.