I never intended to play a mage my first playthrough, but I am glad I have been.

I'm not even running Conjuration. Just Destruction, Alteration, Alchemy, Enchanting and some Illusion.

I am on Adept difficulty (I think thats the one just below the hardest difficulty).

At first I was worried I wouldnt be powerful enough, but like all games the more you play and learn the better you get.

I dont 1 shot everything, but I dont WANT to. When the combat gets too easy, the fun goes away.

Adept is two levels below maximum difficulty. There is Expert and Master above you. I'm guessing you're below level 25. Destruction is beast 'til then. After that w/out Conjuration things get extremely hairy.

/edit I will say though that, if you mix and use most of you magic trees playing as a mage, it is very viable and still pretty easy on master. You just have to quicksave a lot and move quickly.