I think the term is an appropriate reaction to the cutting/consolidation of features that have no clear reason to be cut or consolidated, other than to appeal to the widespread gaming community who may not "have" the patience/intellegence to master these features, in the eyes of the Exes.
Example: FO3 to FO:NV- cutting of Big Guns and putting in Survival isn't "dumbing down." Heavy Weapons was a catch-all for various weapons that were not rifle or pistol-like in nature, despite having skills for various types of weapon damage. And Survival added a new feature, basically cooking, to the game that wasn't there beforehand, which enchanced the roleplay aspects of the game.
However, in Morrowind there was at least 5 melee weapon types. In Oblivion, this was cut down to two, removing one branch (Spears) and consoldating four skills into two (Blade and Blunt, previously Long blades, Short blades, Blunt, and Axes.) Furthermore, Skyrim switchess this to One-handed and two-handed weaponry. There is no clear reason for this consolidation, and there can be no explanation as to why someone who is profiecient with a dagger is equally proficient with an Axe or a longsword- unlike with our heavy weapons example above, these skills were previously clearly defined and made sense in terms of damage type, and the "new" skilltypes do not bring much to the game. In this case, the term "Dumbing Down" is appropriate- we can only reason that the exes made the decision to "simplfy" skills to appeal to the masses, rather than the reason of "this skill doesn't make sense, logically speaking. Why are we using it?"