So you want everyone to be proficient at everything at every moment? A life time warrior being able to raise destruction as efficiently as a life time mage? Perhaps I am being a bit to rash with this conclusion.
Let me rephrase: What do you hate about a class based system?
Thats the thing :
If you chose to
use Destruction as a Warrior, you have begun to create a different class. That is not to say that the Warrior shouldn't at least have the option to use it, and indeed he will be much less effective at it than a Mage who spends perks in the skill.
These 'class' and 'character' skill arguments only attempt to justify the removal of legitimate options for a character.
Examples :
Prison is burning down, a Warrior shouldn't be able to pick the lock?
A horde of Draugr rest in a pool of oil, the Warrior shouldn't be able to cast a simple flame spell?
There are most certainly classes in Skyrim, there are no TITLES associated with them. Every single perk you spend, every skill/spell you use culminates in the development of a 'class' or more appropriately : archetype or sub-archetype.
Options, they are a good thing.