A staff. Seriously.
But if your mage feels like he needs to carry a back-up weapon, I would find one that has some personal meaning to you. My first pure mage carried a dagger for that purpose, until one day he managed to kill a draughr deathlord at relatively low level, and the deathlord dropped an ebony greatsword, so my mage took it, smithed it to the best of his limited smithing ability and enchanted it with some powerful enchantments. It made a great backup weapon because it had a story behind it.
And anytime another mage pulled a weapon on him, he would draw his enchanted ebony greatsword and show them what's what.
The greatsword, or any 2H weapon, is great close quarters backup choice. Just because of the ability to effectively utilize the hotkeys better than a 1H weapon. You can equip two different spells, set the LH spell to (1) and the 2H weapon to slot (2). Then it is a very smooth transition and keeps the RH selected spell when the hotkey is activated.
However, a ranged attack from a bow is probrbly the safest option aside form the absolute wisest option of...RUN!
Then again, a potion carrying Alchemist should have access to an unlimited arsenal of 1000 reasons to never be in a Magicka shortage to start with.
EDIT : A note on the use of staves...
IF the Staff of Magnus actually worked properly and did in fact "absorb" magicka. It would be THE single best staff in the game, for a mage.