What exactly do you have to do to restore it to its former glory?

All the main quests.
5 "Small Jobs" (of any type or combination) for each city (Markarth, Solitude, Whiterun, Windhelm) except Riften. The "Small Jobs" are the Burglary, Shill, Bedlam, Fishing, etc from Vex and/or Delvin.
1 "Special Job" in each city except Riften. Delvin will give those to you after you've done 5 "Small Jobs" in each city.

I may be wrong but there is like 20 or so small jobs you need to do plus the special tasks

You have to do five small jobs in each of the four other cities besides Riften, as noted. However the first 20 jobs you are given are not guaranteed to add up to 5 each per city, you may get more than you need for some cities before you even get one or two for someplace else. And you do not have a way to keep track (in-game) of how many you've done and where unless you want to scroll all the way back through your journal and add them up.

So if you want to get through it as quickly as possible, keep a scorecard of your own so you know when you've done 5 jobs per city, and then if you keep ending up with more jobs for the same place you can either go back and turn them in unfinished (no repercussions except, I suppose, a lot of extra 'tude from Vex and Delvin) and hope to get a new one that you actually need... or you can just reload the previous save and ask for work all over again and see if you get something different.