But is that change based only on weight? What about the extra damage gained from leveling the skill? Magic should have the same type of scaling without having to spend extra mana for extra damage.
Why should it be the same?
One-Handed and Archery rely on skill level and damage perks as main sources of damage. That's because the base damage of weapons is low.
Destruction relies more on higher-level spells than skill level and perks. That's because the base damage of spells is high.
Imagine this: Halve all damage bonuses from One-Handed skill tree, but instead double base damage of all weapons. Effect - you're doing same damage, but the skill "plays" differently, as you need to focus more on finding new weapons than simply increasing the skill with your old ones.
And this is exactly how Destruction was designed - since spells don't have upgrades like weapons do, there's really no problem in swapping them with new ones - so there's less focus on skill level, and more focus on getting new stronger spells.
Different? Yes. Worse? No. Just different.