Are you a mage? I have managed to create a Mage, Warrior and Rogue and I've noticed the Mage levels the slowest of the 3 (even with the 20% increase from the stone). Seems magic doesn't level near as fast as physical weaponry, at least that's how it feels anyways. I've also noticed enemies that wield magic are often the hardest hitting.
I've seen an absence of magical enemies too. I mean, at lvl 6 I encountered One undead nord (Duergr, dwarger, dregra... I can't spell) who could cast frost magic, One spriggan (But I think she was just throwing bees at me, al la The Wicker Man), one bandit mage who looked terrifying... and that is it.
Mind you, I am quite happy about that.
As to: Mage skills increasing slowly, I don't find this surprising. You need to keep casting spells to boost your magic skills, but unless your in combat, which spells make sense to cast? Conjuration makes some sense. Keeping a summoned helper around isn't a bad idea in case of an ambush, but most protection spells are such that it makes little sense to keep them on all the time, you'll just be at low Mana and with the short durations it would have likely expired by the time you would need it. Then you've got destruction spells, that require real enemies to boost your skills.
Compare that to sneak... you can sneak around a town if your bored and kick your sneak skill up 10 points, just by being in proximity to a group of people who bear you no ill will. They'll think it's odd that there is some guy lurking around the bushes randomly and staring intently at them... but thats an RP thing, not really a gameplay thing.