The Draughr's can be quite tough later on too, if you go head to head with them....The Draughr Wights, and Draughr Deathlord's and so on. All boils down to your armour and such. At your point into the game it will be something like Hide armour, so simple Draughrs will give you grief.
However, don't underestimate the use of traps that are originally meant for YOU

Play tactically, even when playing as a mage, there are tactics that can be utilized, same for a Swordsman.