There just don't seem to be enough perks for the build I want, so I was considering not using armor - or at least armor perks. Would this be doable at high levels on expert?

I find this playstyle awesome but I usually end giving up because of NPC killmoving my character, right at the time I was going to pause to potion or cast a healing spell... many many times... one shot'd by marauder archers... or eaten alive by dragons.

Ok, so maybe I will put in armor, and take out conjuration. I find it cheap anyway. What do you guys think of this for a build? http://skyrimcalculator.com/#152529

I just made a new spellsword with Master difficulty. I seemed to do a lot better with robes instead of light armor because of more magicka = more spells casted.
Even the conjuration spells can cost a lot of mana in the beginning. Flame astronach + bound sword left hand + bound sword right hand.

I′d keep both. Light armor for close combat and robes for ranged battles. Don′t give up conjuration.
Mage robes and headgear weight only 1 each anyways.

I know, I don't want to give up on conjuration, but I just don't have enough perks for it. Since I am going heavy on enchanting, I will just enchant light armor with magic regen and all that.

I ended up going with a Dark Elf. I am loving it so far.