You do not need armor if you don't get hit.

With conjuration and destruction, it is possible to avoid getting hit altogether in many situations. And there is restoration magic, in case you get hit.
If you invest in alteration, you can eventually get as much damage protection as the best armor. But then again, you probably won't need it by then because you will probably end up paralyzing everyone before they can hit you.
Mage armor perks in alteration is widely considered to be not a very good deal. If you plan on getting hit, perhaps wearing real armor is better.