Thanks for the answers.
I understand I am free to do as I please (like wear armor if I want) but that would not feel like a legit 'pure mage' to me.
I'll give some tips.
Altmer makes a really good Offensive style mage type, while Bretons, Orcs, Argonians make more defensive style playwise. (Funny, right? Defensive magic casting orc!? It's actually pretty damn good!)
If you want to play as a pure mage, You'll be wanting to use destruction constantly in fights. This will mean lugging around potions until you can create decent -cost gear. You can't really rush in with swords as a mage at those lower parts of the game because Destruction can't receive any bonuses other than from potions. All you need to make a sword effective at low skill in later game are the Enchantments for one/two-handed and potions/smithing.
If you plan to go the no robe route, then Alteration + Lord Stone will be your bread and butter. for defensive capabilities.
I'd also suggest picking between either Wards for their extra Armour bonus, or just plain Shields. Wards with Ward Absorb are great against other spell casters, Shields though would definitely make life a lot easier and they stack with the Mage armor perks. How you perk in the block tree is up to you.
Conjuration is great, but it doesn't necessarily have to be perked into to receive the benefits of a Dead Thrall. You could get two, I guess. But one is all I ever find I need.
Illusion, this is the only tree you absolutely have to grab dual-casting on to stay effective at those later levels. And it's a really good tree. Fury, Calm, Fear, Invisibility. The utility of this skill is just amazing. Defensive, Offensive, whatever.
Restoration. This is all personal Choice, I spec pretty heavily into this tree myself though. All perks except apprentice and higher, and no dual casting.
Destruction. You can go whatever route you want here, You can go for one or two specialties, or go for all three. I went for all three without the last perk into Flames, I don't really need them running from me and making them harder to kill. Though for crowds, I imagine it would be great crowd control. Dual casting is only worth taking if you plan to get impact as well. And then it just gets ridiculous once you have no cost cast gear =/ Personally If I take it, I don't wear much no-cost cast gear to keep the game challenging.
Alteration. No-armour mages best friend tree. I personally take just about every perk in this tree but master and dual cast, though right now I am debating on taking the Dual-cast perk so I can have longer lasting armor. With a shield, Ebony Flesh, and Lord stone it's possible to hit the armor cap with no issues. I Think you can with wards as well, though I'm not entirely sure.
Have fun! It can be a major challenge on those higher difficulties, but it can be loads of fun aswell =]