Honestly, using weakness to magic, then magic boost potion are &--#62; than anything else for burst damage (Aside from backstab). I said it once, but my suggestion wasn't warmly accepted, yet I still stand by what I said. I think you could go all the way with it, if you're willing to go for alchemy and making potions a lot, it could be very interesting. Enchanting is common, it works great with it, but only alchemy + enchanting with pure Destruction, that is something fun. Conjuration and Illusion are handy, but only limiting yourself in Ench/Alch/Destro is challenging and fun. Do post here when you try it out if you liked it, and you just gave me idea for a new toon

I think I am going to load up a new game right now and try it out. I was planning on this for a build: http://skyrimcalculator.com/201232
I would only summon in extreme need. I would roleplay a wizard who is hesitant to play with Daedra. Instead, he would use nature to aid his magical ability - much like Gandalf in The Lord Of The Rings books.
EDIT: Basically, I would just use normal destro, but in boss fights and such, I would keep potions I made that increase destruction damage.
EDIT 2: Yeah, this seems fun to me. Obviously a Breton or Altmer is ideal, but I will also be playing as the Dragonborn, so to me, nothing else besides a Nord fits that. Yu can look more like Gandalf with a Nord than you can with a Breton anyway.