Witnesses of citizens from his very own city heard him praising the empire at an obsessive rate and started doubting his leadership.
Direct quote needed on this, please. I've talked to most of the people in Solitude and I can't recall one of them saying this directly.
People in Skyrim were peacfully worshipping Talos until Ulfric made his move. Because of HIS actions the Thalmor were allowed to enter Skyrim.
Talos worship was banned as part of the White-Gold Concordat, and it was banned throughout the Empire--everyone (more or less) was free to worship Talos until that came down. It had nothing to do with Ulfric or Torygg.
As for the Thalmor's presence in Skyrim...not sure if there's any cited source for when or why they arrived. My guess would be that Thalmor "embassies" started popping up after the Concordat was signed which, again, would have had nothing to do with Ulfric. They've been meddling with the Empire long before the war started in Skyrim.
What he did weakened both Skyrim and the Empire. Where two fight, the third one wins ( aka Dominion ).
This at least we can agree on.

It's the reason why my character's stayed neutral and will probably stick with forcing a truce via "Season Unending." (Otherwise I'd be leaning a bit Stormcloak-ish. My next character will probably side with them.)