Calm and Fear seem completely pointless. Oooh, I don't want him to hit me, so I'll spend a bunch of magicka to make him run away! I can achieve the same result without spending any magicka at all by having my char be the one that runs away. (And believe me, playing as a cloth mage, I spend a lot of time running away from enemies.) Plus, while I am running away my magicka is regenerating. Or I can cast a summon and run and hide behind it.
Until you get quiet cast, frenzy just lets the enemy know you are around. Sure, they might start fighting each other, but if they are close by, they are just as likely to take out their frenzy on you. Without invisibility, you now have a serious problem on your hands.
Another demerit is that if you miss your target, you are screwed. At least with destruction I have area effect spells. And with conjuration I don't have to aim at all.
I am finding illusion to be mostly worthless at low levels and once I get to the master level, makes them game rather boring. (Walk into a dungeon, cast Mayhem, clean up the left overs. Yawn.)
If the entire argument for illusion being so powerful rests on expert and master level spells, then I just don't see that it is all that great.