Actually this would cost you alot more perks to do effectively.
Say you use a sword and an axe, You now need all the axe perks as well as all the sword perks for equal effectiveness. Otherwise you would just want both swords or both axes and save yourself 3 perks.
That's right, you would save 3 perks but the thing is that specializing in dual wielding is quite a cheap fighting style (in terms of perks). Basically, everything you need is nicely packed in one skill: one-handed. Imagine the large amounts of perks you save by doing that; all 2-handed and block perks for sure, then it all comes down to personal preferences.
In my opinion a dedicated dual wielder doesn't need more than one-handed, enchanting, smithing and light armor. You may add some alchemy for the poisons but that's it.