Is he vaccinated against clostridial diseases? I had a lamb a little like this, he was 4 weeks old though and although still on a bottle he wouldn't eat anything. Turns out he had a whopping great abscess brewing on his top lip which once came up was lanced drained and treated with Synulox. He was ok after but still not right and very lethargic, we were passing the vet one day and popped in and luckily saw the farm vet who gave me an anti-inflammatory jab, he's been right as rain since.
If he's been out could he have got a chill? Have you checked his mouth for abscesses or cuts. Will he eat hay? I would be inclined to give him 3 days of pen&strep and see if that perks him up, bring him in and keep him warm and if not better in a couple of days I'd give the vet a ring.