I'm not a big felt expert but I know something about horses and how they move. When they are strapped to a pack saddle the thing will move a bit, worse when full, and with all that rubbing (and your pony's sweat) you'll probably felt the outer layer even more (good), but also the layers will tend to slide over each other (bad). Even if you have stitched them up, some of the stitches will come undone with movement.
Would you be able to maybe make a big, thick felt pad and stuff tight it into a leather cover that you could then rivet onto the wooden frame? That shouldn't move. Or you could try googling harness pictures for ideas?
Also I would put a numnah or saddle cloth under the pack, just in case. Saddle sores on the back/withers are quite painful