Oh sure, I can respect him for dying bravely. The thing people neglect is that he could have declined Ulfric's challenge. That would have made him look weak, and he was Nord enough that he couldn't accept that.
In Sovngarde he says his only regret is leaving Elisif behind, not anything about Skyrim- which reminded me of what Ulfric says that he was more concerned about entertaining his queen than being a true high king.
Considering by all accounts Skyrim was in pretty good shape overall and worship of Talos was not banned in practice, I don't see what the problem with his one regret was. He had nothing to regret about being a ruler. I personally think he made a wrong choice by allowing himself to fight Ulfric, but considering Ulfric cleverly used a psychological weakpoint of Nords, I understand it.
You need to remember when Ulfric talks about stuff like that, being part of the Empire and not rebelling to him mean the person is a bad leader no matter what. He's a very biased source. The biggest complaints with any particulars anyone has of Torygg is that he gave pro-Empire speeches, which is a pretty small complaint.
Being more concerned about all of Skyrim than a few griping Dunmer is more excusable.
While he might not be able to do a lot about the squalid and horrible living conditions of the Dunmer, he could certainly do something about the abuse they receive from Nords in the city. He could also allow Argonians to live in the city and have decent living and working conditions (at least by Dunmer standards), rather than let them be used as little more than slave labor. I believe the Dunmer currently aren't allowed to live anywhere but the Grey District (correct me if I am wrong). He could change that at least.
These are things that wouldn't take much a time, merely a speech or two and an edict or two and that's it. Perhaps some people being punished by the law. No significant amount of wealth or forces spent, however.
So yeah, I consider Ulfric a pretty poor ruler because of this.
It doesn't help that he started a Civil War that has torn the country in two and cost countless lives without even TRYING to talk.