Wasn't this spot ebony ore mine spot for imperials?I didn't understand can someone explain the story please?

to keep it short: imperials abandoned raven rock due to no ebony ore (not really sure), solstheim became territory of morrowind and House Redoran took over Raven Rock.

From what I have heard, Raven Rock was abandonded and then picked up by the Dunmer.

Hmm hope someone in DLC explains it.

Well, I think the ore in the mine dried up, then shortly after the Red Mountain erupted and demolished most of the buildings in the city. That's why if you look at the trees on the southern half of the island, most of them look like they've been blown back by an explosion.