In Morrowind you could wear a robe with your armor.This made your character look really cool and you didnt had to just carry armor without wearing it.In Oblivion they abandoned this although i am not sure why.In Skyrim they did the same thing.The problem is that there is absolutely no reason why your character should only wear armor or a robe.This is frustrting since the game has some very nice robes that are quite pointless to wear

It doesn't bother me. I think wearing one clothing at a time is a good thing.

I can actually understand why you can't wear some robes (college robebs) over your armor in Skyrim (although you could probably wear monk robes over some armor) but something which I miss is the abillity to wea clothes under your armor, after all who would want to have cold steel against their body.