We have a fridge that we keep just for milk, to store it til we have enough for a cheese and it's set pretty cold. We had to stop using the milk for a couple of weeks as both our girls got an infection and needed antibiotic. From the day we stopped keeping the milk we used up what what was in the fridge and it was still ok to drink a full week later. Slightly "goaty" maybe but not by any means "off".
We only pasteurise ours when we make it in to cheese, we prefer to drink raw milk.
Dave