Elisif will remain Jarl, and her entire court will remain in place.
Solitude guards and the Legion soldiers in and around Castle Dour will be replaced by Stormcloak soldiers.
The area of the city between the main gates and the main entry to Castle Dour takes some damage. I don't think it's permanent, except perhaps for the collapse of the "gazebo" type thing over the well in the marketplace. If I'm even remembering that right.
If you had any unfinished quests involving Captain Aldis (bringing him a copy of The Mirror, talking to him about Angeline's daughter) you will not be able to complete them as he will disappear from the game. I actually got a "Quest Failed" on one of these when I forgot to finish it before Galmar sent me to take part in the battle, so it's possible that at least they will not linger in your journal forever if you decide to blow them off and go ahead with the battle anyway. But that may have been something that the Unofficial Patch fixed, as other users have reported that the quest is still active in their journals and they might even have a quest marker where Aldis used to be.
I don't know firsthand of any other quests that can be borked by doing the battle, personally I've never known a resident of the city to die because of it and I think most of the questgivers there are essential until you finish their quests so it seems unlikely. I did have one issue where I was unable to become Thane of Solitude after completing the CW for the Stormcloaks but I suspect that was due to a bug where the dialogue option leading to the Thane quest disappears forever if you don't follow up on it the first time it appears.