Propably due to the reception Oblivion had, enemies kept scaling with you, and leveling poorly could mean enemies could become too powerful to defeat. Allowing players to overlevel enemies prevents that possibility in Skyrim.
Looks like Bethesda just can't win the level scaling battle

Because having all the enemies, most or even half of them scale to the player is a terrible idea in an open world game, especially if you make it so that with the scaling they also upgrade their gear.
The best way to do it is have most enemies have varied caps throughout the game world, for example Wolves would be capped at levels 3, 5, 8 and 10, but you would only encounter level 8 and 10 Wolves in the Winterhold and Windhelm area while all other areas are 3 and 5.
Then some enemies should always be exactly equal to the player's level, such as maybe some of the higher tier Bandits and Spriggins.
Finally some enemies should always be scaled above the player, always, to give constant challenge as players may find enemies that are equal to their level potentially easy, especially with the questionable AI. Enemies like Dragon Priests, Alduin, Dragons should all be scaled to be at least 3-5 levels higher than the player.
To go along with that, especially in the case of enemies that are always equal to the player, only some should be as well equipped. Just because the Bandit Marauder is level 50 right along side you doesn't mean he has to be decked out in full Daedric gear, said Bandit could be wearing Elven or even Scale based on the region they are in or simply by random determination.