hmm...
You'll probably want to pick up dual-casting since you can't off-hand recast mid fight, so you'll want as much duration as possible.
Alteration will never get you the same armor rating either. With 3/3 mage armor it maxes out at only 300 armor.
Alteration also tends to take more perk slots than an armor perk tree... however the magic resistance you pick up is second to none.
It will definitely give your character a weakness... but that can be fun to play... so I say go for it.
I went unarmored alteration on my 1hand spell sword and love it.... but then... I'm a spell sword so have a nice fat mana pool to cast healing spells while I fight....
You may consider doing it as a 1hand instead of 2.... but if you really wanna go for it, do it.