Just about everyone you talk to in the game says Torygg was about as good a High King as Elisif would be High Queen.
Which is to say, just an Imperial mouth piece painted in Nord genetics.
This is the reason I really don't understand some Nords. Everyone knows High King was defeated in Nordic duel. If you dig up on the quest with Elisif in Solitude, you will eventually find out that if he refused the challenge, there would be a new council for a new High King. So he accepted the duel.
Ulfric defeated him, and now he is a murderer. He used Thu'um, but anyone can learn Thu'um if they study it for years, and since Ulfric isn't Dovahkiin, he surely took his time to master it.
So basicly, all the Nords who support the Empire should pack their bags and get out of Skyrim and lick boots to Imperials. I kinda see them here as they're being hypocritical and twofaced. "We want the best for Skyrim, we are proud Nords, but we will forsake and turn a blind eye to our traditions, let the Elves to take our Talos worshipping, and we will blindly follow a nearly bold man who is not even a Nord, just because he thought Ancient Nord Duel was a murder, even tho we know he just wanted an excuse to take Skyrim under his control" - It is just dumb, and I really doubt Nords would accept this if it was real life. I just think this was a best scenario for Bethesda to make two sides; Imperials and Stormcloaks.