You might well be right about the stroking, certainly rabbits will do this with their babies if you touch them. I suppose we leave a scent on them that the mum recognises as foreign. Worth leaving him be for a while and see if it stops.
Incidentally, I saw a video of someone stroking a goat kid and the woman was saying 'see, the doe is fine with it' but it was clear from the video that in fact the doe was pretty stressed.
So stroking the kid quite a bit might be stressing his mum anyway.