Lets start with Azura. Originally the Dunmer were Chimer, but were "robbed" when hte Tribunal used the tools of Kagrenac on the Heart to gain their powers. Most of us know this and as a result, Azura turned them into Dunmer. From this point, it seems, when your entire race has been cursed by a Daedra Prince, you have already hit hard.
Then, you lose your independence when your Tribunal hands over your lands to the EMpire in exchange for a little autonomy. Dunmer house traditions are heavily disrupted and their way of life interrupted so that the guys on top can stay in power.
So the gods you worship have cursed you, and your living gods have handed you over to an Empire of men.
However, it seems this was not enough as Ald-Ruhn is destroyed is destroyed in the Great Oblivion Crisis. (Be it known, it seems, Summerset Isles seems lore wise, like it was the worst place hit during the crisis, but youd think Mehrunes Dagon would grant a reprieve to the one people who worship Daedric Princes). So now, ontop of worshipping people who seem to have little value for you as a race, they have no qualms of killing you either.
Then to make matters worse, we have the events of the Infernal City, where the Ministry of Truth crashes, destroying Vivec the city, causes the Red Mountain to erupt and destroys the Dunmer Homeland. Now they have been devalued by their gods, attacked by their gods, had their land handed over AAAAAAND have lost a good portion of their homeland.
FInally, they are driven to Solsteim when the Argonians invade Morrowind. They are also forced to flee into other provinces and scatter.
Does it seem like the Dunmer are slowly going the way of the Dwemer? Or do you think they will finally hit a reprieve and bounce back from their last few thousand years of bad luck?