You absolutely can just use it as fabric.
I'm in the middle of making a casual top from some yardage I wove. Since I am not loom-shaped (!), I have used an ordinary, tailored sewing pattern, with princess seams, set in sleeves and a zip neck, like a fleece.
Handwoven tends to be much more loosely woven than many commercial fabrics, so to stabilise it, and avoid it fraying badly before I'd had chance to zig-zag the edges, I ironed on lightweight knitted interfacing on the back of all the pieces. The fact that it's knitted interfacing is critical, as this leaves the fabric soft and flexible still.
I've just got the zip and the hem to do, so I'll take a pic when I'm done.
Currently warping to weave some yardage to make bags.