Oblivion had more of a specialization system than Skyrim does. Neglect certain stats? Kind of like people now must neglect certain perks? Specialization is a person's choice. If all perks were obtainable not everyone would get all of them. People would still specialize in something if they wished to. Those who want everything would play enough to get everything. More of a specialization system in Skyrim would be only allowing you to level a few skills up to level 100 while the other could only be leveled up to 25 or 50, but not to 100. If you reach to top level in Skyrim that would mean all your skills are level 100. Where's the specialization then? Not everyone gets to the top level just like not everyone would get all the perks. It's all a matter of choice. Why not at least have the option for people who want to?
Let me just get this clear: You think that a system were what you chose as your characters skills doesn't matter because it doesn't give them any advantages over your non-class skills in terms of usefulness(I don't count increased skill increase as a useful/noteworthy ability) have more specialization than a system where you get a limited amount of perks to spend and most of a skills power lies with the perks? Sorry, I don't see that one.
The specialization lies in the perks, who in one way have replaced classes. Even if you have a skill at 100 it is going to be somewhat weak without perks(unless you exploit enchanting etc).
Also: Specialization is not a chaice in that manner. It is a part of the design, and it has been as long as one could chose to play different archetypes in gaming(even the ealiest tabletop games like D&D). It has always been an important part of RPG design, and it always will be.
If anything the best option to do with the perks to make characters more unique would be to add more chosable perks to the current perk-trees.
If the player must go out of his way to get anything but a generic character the character system is horrible. Saying that they should include it because some may want to use it is not a viable stance on this because the game is designed around having a limitesd amount of perks. It would be the same as adding the option of turning in to a dragon. Some might like it, but it doesn't fit. IIRC Todd Howard mentioned in an interview before the game was released that they added the perks to give the characters more specialization, and they do.
In terms of game design Limitations = Specialization = Reason to chose x character-type/playstyle. If you have no limitations you could jst as well have used a set hero, like John 117 Master Chief.