Both

My destruction / conjuration mage uses no weapons, not even conjured weapons nor staves (well exept when required by quests). Works out quite well.

Why not go with both? If you're going to be a pure caster - and by that I assume you're wearing cloth (no armor) and using purely spells as offense, Conjuration mixed with Destruction works. Summon an atronach or raise an undead to tank for you while you blast away with your element of choice.