My question is where is the rest of it, I swam deep into that sea and can't find any ruins or remains of it. Logic would say you'd least find some proof of it. Hell why wasn't a quest made out of trying to find some great Winterhold Heirloom out of there. It's just something that bothers me, only thing that could find that possibly connected, was that Temple nearby, but nowhere else.

It's odd and to be honest pulls me out of game.

I think the most likely explanation is time constraints.
Winterhold is supposed to be the capital of an entire hold, yet youll find no-one else living there except in those few shacks
(and the College).
A likely explanation is that they just ran out of time and could not give Winterhold any attention in that area.

At least we have (somewhat) of a lore explanation for it.
I would have hate to have seen another Sutch happen.

Well, if you want to apply logic to a fantasy game, look at the remaining houses: all built of woods save for the lower walls, made of stacked rocks kept together by lime. Not to mention that a few years have passed already. These are no dwemer constructions built from advanced materials nor the megalitic nordic ruins which still dot the landscape. It's not that strange that no remains are found.

I just see it as its so old its disolved into the sea never to be seen again.

If it was a tsunami the wreckage could (would?) have been sent out to sea.

Also the town is made mostly of wood and the destruction was almost 100 years ago. May have just decomposed. If you were looking for underwater houses I think it is realistic they are not there. Underwater piles of rubble that are covered up with sea life and blending in is more likely.

This is what I figured: the wood has floated away, plus if the currents were violent enough to wash away that much of the town then it would also be strong enough to carry away all the debris. Either this or "magic".

Edited for punctuation.