Many people start off with Dexters (including me!) because of their size. But as WBF suggests, they are (how can I put it?) not the most placid of animals.
I once had one that I bought in as an adult, that tried to kill me. I didn't keep her long!
But those I've raised from calves have been placid enough, as they're well handled, but they sillt have quite strong characters.
I do think though that you are expecting to do too much with them. As you rightly say, they are a dual purpose animal. But the favoured ones are the beef types, and having been selected for their meat characteristics, their milking qualities will have deteriorated. Until recently there was a milking herd of Dexters, and the milk yield you quoted was probably from this herd. But their conformation as meat animals would be nothing like that which you would need to aim at nowadays for your proposed top breeding herd.
You might find that Red Polls would be better for your requirements, not only because they have a calmer temperament, but also, bearing in mind that very few animals become top prize winners, then those that don't make it will still have a more marketable carcase.