Well, you can talk to the old king of Solitude in Sovngarde, and you learn that his and Ulfric's Duel wasn't exactly fair. Ulfric didn't just ground him with the thu'um.
Quote from the High King
"When Ulfric Stormcloak, with savage Shout, sent me here, my sole regret was fair Elisif, left forlorn and weeping.""I faced him fearlessly - my fate inescapable, yet my honor is unstained - can Ulfric say the same?"
edit: I personally believe that, eventually, Skyrim will be realigned to the empire in sole dully hinted at manner. You will hear some details, but the whole truth and all the facts for the reunion will be unknown.
It wasn't fair? You could say the same to any other fight where one person bothered to train to death, and where the other didn't.
Ulfric isn't a Dragonborn, so he had to learn how to Shout the hard way. And the hard way involves to train several years to learn even the simplest of the shouts. If the ex-High King (which was also a Nord, and thus capable of learning how to Shout) didn't bothered to learn or practise, it's his fault. But they both had equal capability from the start.
So yes, it was a pretty fair fight to me.