My problem is that every attempt I have made at a mage feels so lack luster. His damage is ok sure but he has terrible defense even with Oakflesh.
Personally I don't bother with Oakflesh. The amount of protection it gives is paltry, and you can only increasing by wasting three valuable perk points that will become obsolete once you get the master level spell (dragonhide). I'd spend those three points elsewhere and just concentrate on not getting hit. My destruction mage is pure offense.
For defense, I dodge and keep my distance. Fast creatures like sabrecats and bears have to be put down very quickly or I die. One strategy for them is sprint/jump for higher ground and then turn and rain destruction down on them from above. The PC can climb slopes that will give trouble to animals and other NPCs.
Conjuration seems like a nice distraction tool but my summons die quickly.
Until you get some higher level summons this will be a problem. My nigheblade uses conjuration as a distraction while he sneaks in or runs away, but my destruction mage does not use conjuration much.
I think the biggest issue overall is that my magic drains insanely fast in anything above a skirmish. I have points in destruction and the first couple of perks for Novice and Apprentice perks, additionally I put everything into magicka when I level and am wearing fancy robes.
I took the casting cost reduction perks for my first mage build because I did not want to enchant because it felt "cheap" to get "free casting." A no-enhcant mage may work okay if you are playing a conjuration or illusion mage, but not so well for a destruction mage. You can do it for sure (the Resident Pianist points out how in his post above) but I don't like carrying around 100 pounds of potions and staves. When my first destruction mage hit 70 destruction skill and started picking up the expert level spells, the magicka needed to cast them was insane. I had already perked Adept cost reduction but I could only cast my Adept spells once, maybe twice before I had to switch to lower level spells. That was not terribly fun. So, I changed my attitude about enchant.
So I rolled up a new destruction mage and am perking enchant but not putting any points into spell cost reduction. That will free up perk points for enchant and alchemy. Alchemy is useful for a destruction mage because you can make potions that increase your destruction spell damage.
My plan is to get about 95% cost reduction to Destruction and Restoration and be a Destruction/Restoration mage with a Destruction spell in one hand and a ward in the other. I know, I know, wards are really underpowered in Skyrim, but I think they could be fun if you reduced the casting cost on them enough. Plus, I don't want to use dual casting impact most of the time. On certain opponents (like fast snowy sabercats) I may use dual cast impact stagger, but it is not fun to use it all the time. I don't play on Master (wards are pretty much useless on Master because they are broken, causing stagger by all sorts of opponents).
If all else fails, I can switch to Destruction/Illusion or Destruction/Conjuration just by enchanting some new gear. That is the beauty of using enchant for cost reduction instead of taking those perks. You have a lot more flexibility to change the character around.
You also have to pay attention to what you are fighting and how well each element works, some spells work better against certain opponents. The novice destruction spells will get you through the game until you get your enchant high enough to start using the higher level destruction spells. If you cast your novice spells in bursts, they will do a lot more damage and drain less of your magicka. The flames spell works this way anyway. I don't know if the frost or shock spells stack the damage the way the fire spells do, but with the first level frostbite spell, I can cast it in bursts to cause some serious slowing on opponents. Works even on Dragons and prevents them from breathing on me when they land. That's how I killed Dragons up until I was level 20 and started being able to reduce my casting costs through enchant.