Play it on expert. Still easy but some enemies will one hit kill you
Working on better armor now!
I play a stealth archer, conjurer, illusionist on expert. Level 52 without dying right now, and I'm not bragging or lying it really is that easy.
The one thing I can say though, is that if melee was easier or less risky it would kind of kill the point of it. The combination of skills is designed to kill things from safe range, you use quite a few abilities at early levels, you have to take things slowly and carefully. If rushing in with a big sword worked equally well it'd be disappointing.
Part of the success of the build is due to being able to take enemies one at a time, and with illusion you have complete control over a single enemy.
Not an excuse for the lack of challenge or some of the overpowered abilities/perks entirely, but using the best tools for the job of killing things without dying, you probably should have better success than someone using a less cautious play style.
Once you can just spam summons at things though, and you can calm any enemy but dragons, yeah it's actually less involved than a melee build. I still use just archery for a lot of enemies due to that, and Dremora Lords for killing stuff I don't feel like bothering with myself.