I'm levelling up way too quickly - I killed about 6/7 bandits in a dungeon and levelled up 2 levels! Been playing 2 hours and I'm level 6 - and that's after about 3 dungeons and the slaying of a few wolves.
I'm playing the game on 'adept' difficulty and realise that I can increase the difficulty slider but surely it's not going to make it as hard as Oblivion/Morrowind was? It took me hours and hours to progress to level 1 in Morrowind.
One of the major complaints about those games was all the necessary grinding you had to do in order to progress and not be 1 or 2 shot by mean, scary mobs. So leveling IS a bit easier now in that you should level as much as you need by actually doing quests and exploring rather than going on a nice crab walk of shame to get that extra bit of xp.
Also, yeah, you should probably increase the difficulty so that the game is challenging. It's a little fun 'zerking into a cave and blasting the first 3 guys you see in a couple of seconds, but it gets old fast. The only way I'd enjoy mobs dying that fast is if there were lots and lots of them to stay in constant battle with.
Increasing the difficulty to a level you're more comfortable with should slow things down to a better experience for you.