Sounds like a nice build and it seems to have the potential to be very powerful too. From your post it seems like you'll be focusing on Destruction and Restoration a lot, am I right? Going Spellsword (or Spellmace in this case

) means you miss out on the awesomeness that is the Impact perk from the Destruction tree, and also means you have no blocking abilities. The biggest problem I see is that with neither Impact nor Shield Bashes giving you the power to really stagger your opponents, you may struggle in crowd situations and your enemies could quite easily rush you. I wouldn't use Robes for this build, since robes are questionable even on a pure mage... instead, try this:
Use Light Armor and fortify it with Alteration spells during combat (e.g Oakflesh, Stoneflesh). This gives you the strongest possible armor with minimal effort (even stronger than Mage Armor, since if you're wearing a full set of any armor, even if its as weak as Fur, you get a hidden +100 to your Armor Rating). Light Armor keeps you fast and nimble, allowing you to constantly keep that 10-20m of space in which Destruction spells shine between you and your opponents. It also means you have the choice of using the Alteration spell in tougher fights, or not wasting time on petty opponents and just hammering them while just in your Light Armor.
Back this up with the Respite perk from the Restoration tree keeping your Stamina high, and also providing plenty of juice for Mace Power attacks if your opponents get too close. Also make good use of your shouts - use Unrelenting Force to keep opponents coming at you one at a time, and Whirlwind Sprint to dodge away from tougher enemies and give them another bout of roasting / zapping / freezing followed by a couple of sturdy whacks

maybe even consider an Amulet of Talos to give you a Fus or Wuld every 10 seconds or so!
This is what I would do with a Spellsword character, since it's a dynamic build that requires you to actually think and adapt somewhat during combat (while not to the "oh [censored] one touch from that Giant's club and I'm hitting aetherius" extent a pure mage might need to). The problem I find with Heavy Armor-using mages is that combat just becomes autopilot, since all you have to do is hold down a trigger while just tanking hits... might as well tape that button down.