Nords aren't Scandinavians. Tamriel isn't Earth. I don't see why real-life height of vikings should affect TES lore.
...True.
On the other hand, what is not true, is that vikings were the shortest Europeans at that time. They were considered comparatively tall, and though they were shorter than modern Scandinavians, they weren't
very much shorter.
The average
medieval Dane, Swede etc. though, was very short compared to modern Scandinavians (and about the same height as other medieval Europeans). The reason is that their life standard/life quality/whatever you call it in English was lower in the 12th - 14th century than during the previous centuries, mostly due to overpopulation and poverty.
(I'm Swedish btw... very short and with dark hair, so in Tamriel I'd probably be mistaken for a Breton.

)