Sounds like if you've been having to pick him up so he stands, it's a bit more than "unsteady on his feet". Sounds like something not right inside, may well be that without enough colostrum and possibly lacking roughage such as hay or straw to eat while his rumen was developing, his guts haven't developed to use grass properly. If he's improved on the regime you've described, you might be able to keep him going at that, but scouring after worming twice with ivermectin doesn't sound good. If your vet has checked him over, including faecal sample for any nasties, you may find that he will be a "poor doer" and if not able to get up himself and scouring it might be kinder to put him to sleep.