Not so sure about the health damage-thing though. I live in Norway (Just think about Skyrim...) and I bicycle to my school every day (even when it's layers of snow outside, like now) and It's not really that cold.
I bike a lot in the winter, too. But I'm quite careful about what I wear. Its thin, but I've got a wind resistant outer layer and insulating inner layer, with some thought given to how well it handles sweat. And if its actually COLD (below 0f, which is, um maybe -10c) I still can't stay out for more than an hour or so.
Obviously a heroic adventurer from Skyrim would be made of sterner stuff, so sure, no health loss. But reducing the recovery rates for stamina (and magicka, and maybe even health) to 0% would be entirely reasonable, and would well simulate how a cold environment grinds you down. Sure, the cold wouldn't kill you, but if you get in a fight, you'll be in real trouble afterwords, until you can get somewhere warm or svck down some potions.
Bonus effect to cold tanking your stamina recovery; Nordic Ale becomes useful as a post-combat pick-me-up. Reducing your stamina recovery by 30% (or whatever) is not a concern when its already 0%...