Can you put the trailer in the field and feed him in there? Every day I mean, just to make loading boring and routine and not necessarily always to do with traveling?
Also any sessions are best done on the basis that all day is completely fine for them to load, and that standing still is ok, but any movement backwards makes for a circle round - they get bored of the circling! And if they know you have all day they tend to load in 5 mins :-))
A Dually type head collar can be useful so any forward movement is rewarded by release of pressure.
Sometimes horsesdont like leaving the exciting pony party - our big chap was like that, ok at loading to go out, but didn't want it to end. So maybe when you box him up for the next few times, just box up and go round the block, or for a longer drive not stopping then drive back home. That way he will stop anticipating where the end point of his journeys is. Or take him to an event but instead of going home, once youve loaded him take him in a circle and back to the event. So he got to come back to the nice exciting place! And then load and home.
It may be worth if problems persist borrowing or buying a CCTV camera so you can see whether there is actually some issue that is making the horse reluctant, but if it's one way only I would think it is more not wanting the party to end!