Guess I've been playing too much WoW/Rift, and really like my character to be the best it can be (go Air Force, ooops). Min/Maxing still does it for me.
Don't forget, there are 240 perks in the game, that will impact your character. If you want to be stronger in, let's say mage area, you need to put your perks there. So, just running around and not doing a solid build will definitely impact your character's strength (meaning what he'll/she'll be, i.e., mage, warrior, etc.)
You'll be a jack of all trades but a master of none.
Also different races have different strengths and weaknesses. Nords, for example, have bonuses in strength making them ideal for heavy armour and melee weapons but not so good for wielding magic. The Altmer, however, have bonuses in magic wielding but not in melee weapons or heavy armour.