I think the fact that he's not letting you try to move the tag around indicates that it's hurting or annoying him. Is it crusty round about the tag hole? is it red? Before you try to move it again, soak with warm water to release any crusty bits, clean the tag too, then push the tag back through. The hole may have been stretched in which case it will just fall out or push through again, and you would need to replace the tag. I know many folk don't like using blue spray, but for a potentially infected ear, or one with a raw hole, I think it's justified. To hold him still, reverse him into a corner, hold him crushed into the corner by one of your legs at his neck and chest preventing him from pushing forwards, and the other in front of his back leg. Tags are hateful things.