Not really. They weight nearly nothing, they're quieter while sneaking, and they're required for the Mage Armor perk.
^^This.
I don't like the argument that armor is always trumping robes. You do wind up with perks saved (by just snagging mage armor in alteration tree). People always say "just get more perks (to get actual armor)" but if that involves grinding non-character playstyle skills, then its a poor solution and a bore.
Armor should give a penalty to spell casting.
The way it is now, it's much superior to having no armor.
You waste less perks on armor to have no weight and that it doesn't slow you down than on Mage's Armor perk in order to make unarmed style as good as armored.
On top of that, mages need to cast their armor constantly while armor users simply put armor on and don't have to worry about anything ever again.