1. Race - Different choices give different races and abilities
2. Health/Stamina/Magicka selection - kind of obvious
3. Skill Level/Perk Selection- Your skill goes up, you become better at something, perks give you big bonuses
4. Equipment you're wearing/wielding - How high a weapon is smithed, enchants on it and your gear can make a huge difference in defining your character.
5. Potions being used - Using poisons can boost damage and debuff your enemies, as well as boost your skills and H/M/S.
Okay so when I first started playing, I started out on Adept just like everyone else. I'm big on customizing my character and making them my own, so I was really charged at the idea of being able to place two seperate enchantments of my choosing on my character. I was shocked at how quickly the skill levels up, any efforts to actively level enchanting result in it surprassing any skill in speed it levels.
You get that enchanting all of the way up (easy to do) and with smithing, you're hitting like a truck, got enchantments on everything, and with potions, you can quickly make a mint as well as not have to worry about buying a potion ever again.
So after not too long, not only did I go from Adept, to Adept but turning it up to Expert for Dragons, then Expert all of the time, then Expert but Master for Dragons, now I'm Expert or Master at all times. I don't do Master at all times because of the difficulty system, playing a destruction focused caster on Master is akin to ramming ones head into a wall.

So then my first thing was to not bother with alchemy any more, so thats out. Then I don't bother actively leveling enchanting, but it still goes up with little to no effort. And then I stop actively leveling smithing. Essentially, I find myself using less and less skills actively, using items that exist in game that look cool and building characters around them (Helm of Yngol for example).
The issue is not just the difficulty, its that the enemies don't use those same skills and perks you use to be competitive, so the only choice you have is to crank up the difficulty and turn them into a wall of health, or to use a smaller and smaller group of skills in order to play the game.
Some people are fine with this, but if someone complains about Super Mario Bros, someone saying "Well just don't get any Fire Flowers, stop complaining." is kind of ridiculous, but for some reason, this is a perfectly acceptable response. "Gimp Yourself".
I shouldn't have to play with one hand tied behind my back at all times, because someone else wants to be able to handle everything with easy. If they want to be all-powerful and not have a challenge, they should have to turn the difficulty down, not have me not use certain skills, or not perk them, and have to crank it up at all times in order to maintain challenge.
There's nothing wrong with self-limiting to a degree, but I shouldn't have to gimp myself completely and only use two-thirds of the game system in order to keep it interesting.