If you side with the Imperials, Jarl Laila Law-Giver (who has the best last name in the game) gets kicked out from Riften, replaced by the incredibly corrupt Maven Black-Briar
If you side with the Stormcloaks, Jarl Igmund will be kicked out from Markarth, and Thonar Silver-Blood, who are practically running the town (and has been exploiting the Forsworn for quite some time), takes the throne. Although the Thalmor got kicked out as a side bonus
So, each side already has negative sides
However, seeing that the... "Empire" is already nothing but Cyrodiil, High Rock and Skyrim, I'd say the Empire is taking a bad turn, and the best thing to do is to secede from the Empire. True, the "Empire" needs Skyrim warriors now more than ever, but seeing the bad moves taken by the Empire in the past, I truly doubt that even if Ulfric have never rebelled to begin with, the "Empire" will have the balls to fight the Dominion
The Empire didn't even try save Valenwood. It was the first province to be swayed/captured by the Thalmor, and at that point, the "Aldmeri Dominion" was still just that: Aldmeri Dominion, not Aldmeri and Bosmeri Dominion. The Empire still had High Rock (and Orsinium), Hammerfell, Skyrim, Cyrodiil, Elsweyr, Black Marsh, and what's left of Morrowind. Fighting off the Dominion should still be relatively easy
Yet, the Empire let things take turn into the worse: first Valenwood, then Elsweyr, and Black Marsh, one after another. The fact that the Aldmeri Dominion couldn't win the Great War alone should be miraculous enough.
After that they ban their own God, the God that they have worshiped for more than an era, virtually changing the "draw" situation into "losing", and Hammerfell seceded
The Empire is nothing but a remnant of the past. Seeing as Thalmor agents can operate freely within the Empire, there is nothing much that the Empire can do against them. The Empire signed a deal while the enemy had virtually nothing left in Cyrodiil, that alone had left a bad impression to me