But Divayth Fyr did.
Anyway, as a balancing factor it makes sense to me that a mage would be gimped if he chose to be armored. Otherwise what's the point of choosing to go the traditional, unarmored route? Once you reach a certain level of competency, you can make up for your shortfalls by way of enchanting zero cost spells for armor-wearers, or using Shield spells for unarmored mages. (Sorry, can't remember what they renamed Shield in Skyrim... Ward and stone skin? Something like that)
the reason mages are traditionally unarmored is because they spend their lives in libraries pouring over books to advance their knowledge and and further understanding of magic. this doesnt leave them time to go out and train to use armor effectively and it only hinders them in spellcasting because of the restriction of movement. just like they never had time to go train in the effective use of martial weapons. they forgo this for the powers that magic affords them. they dont need any of that because a fireball will always be more destructive than a sword and their defensive spells make up for their inability to wear armor.
traditionally of course. in skyrim for example its perfectly fine for them to whack stuff with a sword in one hand while shooting fireballs out of the other running around in full dragonbone plate. it doesnt make any sense in the traditional sense but there you go.