Its mentioned time and time again, but it seems that very few of you know it. Winterhold fell to the sea in the Great collapse, I assume the area around the college was once winterhold. Since it fell to the sea (Many decades ago) most of it has been buried or carried away.
No, people understand that perfectly well. However, if you go down the cliffs, there is no evidence that a city was ever there. No ruins, no stones, nothing. It just doesn't exist in the game like it should. There isn't even any evidence of the walls that would have once surrounded the place.
Which makes the whole "falling into the sea" seem like little more than an excuse to simply not build a big city. Or really a city at all, because what's left hardly qualifies as even a village.