I was worried about morokai, having heard the horror stories and he did hurt quite a bit. But the dremora lords (I had to resummon him once) did a pretty good number on him.
Toughest 'normal' fight was probably draugr deathlords on my warrior. 'Normal' fight being defined as one where I didn't use berserk.
Oh, and that guy called the skinner, he was a proper bastid until I discovered wolves could howl and cause fear.
Edit: No, I forgot about the dwarven sphere boss in part of the college arc, he was
ridiculous. One shotting me and the antorach left and right. took ages to wear him down, casting from the shows and reloacting every time as he want to look for me. Probably should've been a higher level though, or had a companion, or used pots XoD