Brimwood - I'm not actually saying that all Dexters are difficult to handle - just warning you that they are less reliably loveable than most other native breeds.
But as a smallholder I personally think there is a lot to be said for following your heart. So if you've always wanted a Dexter, then I'd say
get one. 
You're only here once, so at least try one and make your own mind up. I would suggest maybe getting a young heifer that's halter trained and at least you'd have an idea about how quiet she is. I must admit there is a certain charm about small cattle, and even when they're arguing with you they're still a lot easier to handle than a big continental. You could always have your little Dexter and keep a more commercial breed as well which you could cross her with to get a marketable calf.
In spite of what I said about Dexters, I still have one left that I put to our Hereford bull each year. She produces a good sized calf that always fetches a decent price in the fatstock market.