I use it quite often because I enchant normal clothing for smithing and alchemy. Both fortifies can be placed on hands, rings, and amulets, while smithing can be placed on armor and alchemy can be placed on head pieces. Typically for RP purposes, I use a circlet and the hammerfell garb, a pair of gloves, and a silver ring/amulet for smithing, gold ring/amulet for alchemy.
Also if you don't want to wear armor and want to create a custom mage's outfit, with a high enchanting skill and perks, coupled with a fortify enchanting potion (snowberries, spriggan sap, hagraven claws, and blue butterfly wings), then using regular clothing in the way to go.
Unarmored is tricky to play in this game as there is no Skill or Perk Tree for it. However, as you say, there is no protection, so it makes for a challenging build.
One thing you can do is Enchant those clothes with spells that can help your lack of protection.
As some have said, being unarmored with the alteration perk helps. However I used to have a destruction/restoration mage and I would not wear chest piece armor, but I would wear boots, gauntlets, and a mask. So you can wear regular clothes, but wear the little things on the outside and you'll receive
some armor benefits. This is why quite a bit of the magic in this game in my opinion is overpowered, because the best magical benefits are given through clothes that have no armor.
There's a fine balance, but once you get the hang of it, wearing regular clothes is awesome. As a mage, you can run around a lot faster so you really don't have to worry about armor.
EDIT: If you're still having trouble running around with no armor on, get a tank companion who wears heavy armor and rushes into battle with a two-handed weapon. Lydia is useful for this. Just be careful when using AoE spells.