Same happened here last year, and the lambs were wild, so they were left with the flock. Tried the bottle on them with no success, and tbh at that age it is almost impossible to get them to take it anyway.
If there is grass out there and they have nibbled on beet then I think they will survive, but probably be on the small side. I would still feed my ewes anyway, so my lambs knew about feeding from the trough and did tuck in.
Not sure about removing them from the flock, at 5 weeks they are big enough to not be taken by a fox, and they will just learn very quickly to be in the middle of the group at night time.
Both of mine are still there, although one of them in particular is quite small. Mine are Shetlands, and these two will not be used for breeding, but go into our freezer in the autumn.