As a pure mage I have worn mages robes for 31 levels, nothing more. Yes it is possible. Yes there are enemies that would one hit me, draugr death overlords, spriggens were nasty, could not mess with frost trolls or the big cats.
Basically it forces you to
1. Not get hit
2. Use the environment to put space between you and enemies
Its not easy obviously, but it is 100% doable.