Do you feel as though you're being stretched too thin mana-wise using all four schools? I've been playing a wizard but I haven't really bothered with illusion.
I've never found much of a problem but as I said I don't play pure mage. I intentionally designed a character that can do everything well enough to be useful on master difficulty but nothing so well it "breaks the game" (I limit myself with fortify smithing/enchanting/alchemy to prevent excessive damage, also part of why all my enchants are for magicka cost reduction). That way I just run around and use whatever seems like fun at the time. I haven't actually hit level 81 yet on a character but I have used all the skills perked this way on a character above level 50 (I'm a chronic restarter).
Here is the level 81 build: http://skyrimcalculator.com/#24308
Obviously when leveling you would perk in to what you are using at the time. You may also want to shuffle some stuff (I personally only go for silent casting in illusion but I don't like enemies running away or fighting each other). Basically those are just the "bare bones" perks for each tree in my opinion to make them useable on master. Yes I use conjuration, no I don't perk it. None of the perks really help my summons be better beating posts and/or distractions. Storm Atronachs run off and hide too much and none of it effects the Dremora at all. Having 2 summons "breaks the game."