Why doesn't it make sense? Read Hallgard's Tale, and check out one of the cooler Daedric Princes, Boethia.
I guess the reason is two fold. One part tradition and one part common sense. Traditionally (Since the first D&D tabletop games), rogue / assassin characters have always been limited to light armour because common sense dictates that even just lifting your arm to scratch your nose would make a hell of a noise when you're decked out in metal plates from head to toe.
Some people can discard common sense in favour of gameplay. Others can't. Both sides have an equal right and opportunity in Skyrim to play how they want.
Edit: Just to clarify, those bound by tradition considers leather the only viable "light armour" for stealth. Elven and Glass armour would also be impractical.