So I went past the gate to Morrowind and I noticed that the land between the mountains and the Inner Sea is barely wide enough to hold a small village, and Vvardenfell seemed very small, too. So I tried to see how the map dimensions relate to each other.

http://skyrim.nexusmods.com/imageshare/image.php?id=47535

I estimated the length of a mile based on the exterior cells. In Morrowind one cell is supposed to be 385 feet wide (128 yards, 117 meters), in Oblivion and Skyrim it's half (193 feet, 64 yards, 59 meters).

As you can see, a mile in Morrowind is "shorter", which makes sense because Vvardenfell is much smaller than Cyrodiil or Skyrim, but the game maps were about the same size.
This http://skyrim.nexusmods.com/imageshare/image.php?id=47537 shows Vvardenfell scaled down to the size it would have in Skyrim.

So I say it's a good thing Vvardenfell was destroyed, because there isn't enough room to rebuild everything.

Skyrim feels huge when you're walking through a forest, but when I stand on a mountain south of Riften and can see Winterhold, it just seems small. Sometimes the game feels much to dense, locations that seem quite distant on the map are right next to each other when you visit them. It a bit like Morrowind where you found an ancestor tomb behind every other hill. Especially with horses and the improved running speed, I think it would still feel dense enough if the map was 50% bigger, without additional points of interest.