DISCLAIMER: I don't mean to give offense to any modders' hard work, so I hope that this is taken as purely constructive opinion and nothing more.
Now, on to the list!
First, I want to start with things to avoid. There are tons of spells out there which I avoid using in my games, usually because they have one of the following problems...
Overpowered
This is probably the most obvious thing. Sometimes effects that seem innocuous at first can end up being abused in ways that the designer didn't intend. This is unavoidable. Sometimes overpowered spells are fun and that is exactly what some people are looking for. This list focuses on spells that Bethesda could have included in the vanilla game without breaking balance. I think most of the time, the reason some spells seem a bit overpowered is because the designer might be forgetting about some of the underlying realities of the magic system in Skyrim: magic damage bypasses armor, magnitude can fairly easily be increased with potions and enchantments, spells scaled from perks, etc.
Functionally Duplicating Other Spells
There are a lot of spells out there which do this. A good example would be an alteration spell which turns a hostile NPC into a ghost/ethereal, so that it's attacks do no damage to you for a short period of time. This is the functional equivalent to the "Calm" spell from the illusion school since they both accomplish the same thing. These kinds of spells don't really add anything new to the game. But worse, they tend to give their parent school of magic access to effects which that school normally doesn't have. This is bad because it breaks the balance of the magic system in that it makes a particular school of magic more versatile than it is meant to be.
Belongs to Wrong School of Magic
Adding a spell effect to a school of magic in which it doesn't belong throws off the balance of the magic system. For example, there are tons of custom spells out there which add direct damage effects to schools other than destruction (a "shadow damage" spell added to conjuration for example). This throws off the balance of the magic system because it allows the player to focus their skills and perks on fewer schools of magic while still having a diverse set of options available to them. Only destruction (and debatably restoration when used on undead) should be doing any form of direct damage. Alteration spells shouldn't be modifying NPC AI behavior, and illusion spells shouldn't be enhancing stats or warping time and space. Of course, some spells have a hard time fitting neatly into any particular school, in which case best judgement should be used. This is one of the most aggravating design problems for me, because I see so many neat spells out there that I would enjoy using if they belonged to a more appropriate school of magic. If I play as a mage that is focused on Conjuration, Illusion, and Restoration, I shouldn't have access to direct damage type spells.
Duplicates Skills/Shouts
Many spells simply duplicate the functionality of other skills and turn them into spells. Things that you normally can only do with alchemy, enchanting, smithing, or with shouts for example. A spell that takes the mechanic from the Unrelenting Force shout and turns it into a "force push" spell would be a good example. This doesn't really add a new mechanic/ability to the game, but gives the mage access to what makes other skills unique and special. If you create a destruction spell that poisons the enemy, you've taken away the need for the player to focus on alchemy and given them many of the benefits of alchemy. Some NPCs in Skyrim are naturally weak to poison. This should be a time for the alchemist to really shine, not the destruction mage. Shouts become less special if you get the same functionality out of a spell. A good spell doesn't degrade the usefulness of other skills. I don't like using spells that fortify or poison things for this reason. Another good example are spells that open locks, bypassing the need to put points into lock picking (yes, I know that earlier TES games had spells that do this, but Bethesda seems to have had the good sense to take those out). A big part of Skyrim is making the difficult decision about what perks you are going to choose. If the mage becomes so incredibly versatile through spells that he can do everything any other kind of character can do, then perk and character development choices become boring, no-brainer decisions every time.
(I realize that Bethesda has already duplicated functionality of many Spells and turned them into Shouts. Examples like the elemental shouts, Ice Form duplicating Paralyze, etc. I don't think that was a good choice on Bethesda's part, but I still don't like to use spells that make this problem worse.)
Conjuring Anything
The ability to conjure other creatures from the game degrades the usefulness of the necromancy tree. Why bother looking for a Draugr corpse to reanimate when you can simply summon a new one? Besides, the creatures that you can already conjure provide about as diverse a set of different tactical options as the player is likely to need anyway. Are there very many tactical reasons why you'd want to summon a Draugr instead of a Frost Atronach or a Daedra? Most of these spells just clutter up your menus without giving you additional options that are all that meaningful. Limiting conjuration to a few spirits, Daedra, and Atronochs makes sense from a lore perspective as well. You are essentially warping in a creature from a different dimension/plane, so it makes sense to limit conjuration to creatures that only exist in those other dimensions/planes. Conjuring a giant kind of breaks the lore of what conjuration is actually doing.
"Prism" Spells
These are spells that simply increase the effectiveness of your other spells by raising their magnitude. A spell that temporarily increases the power of your fire spells by 30%, for example. I leave these out because they essentially duplicate alchemy and enchanting. If you don't feel that your lightning spells are doing enough damage, there are mods to generally boost the effectiveness of the magic system in general. They are also prime candidates for being overpowered because they can be added on top of alchemy and enchanting.
Mystery Spells
I only rarely run into these, but I do see them every once in a while. These are spells which are difficult to figure out what they actually do. I've seen spells that create visual effects, but their practical effect isn't explained in the spell description, nor is it obvious to the player what the spell is actually doing, other than creating some visual particle effect.
And here are some things to consider including with your spell…
Synergy
Spells that combine well with other spells are really fun from the player perspective. Especially if you can make cool combos without resorting to a "prism" spell. It's always a good idea to have a spell or two in mind that will combo with your spell particularly well. http://skyrim.nexusmods.com/downloads/file.php?id=16225 has several spells that combo in a cool way, like the "Gate" spell, which teleports a target to directly in front of you, and the "Astral Chain" spell, which teleports the enemy back to the previous position, doing damage based on the distance teleported.
Useful Lifecycle
If your spell is easily attainable for lower level characters, it's nice if that spell is still useful later in the game. If the spell just doesn't scale well and ends up being useless at higher levels, consider making a more powerful version higher up the skill tree if appropriate.
Dual Casting
The player always expects that they will get a "better" version of the spell if it is dual cast. Unless it really doesn't make sense, make sure the dual-caster can get extra power out of the spell. There are so many cool spells out there which don't support dual casting where it would make sense for them to do so, which is a shame.
NPCs
It's an awesome feeling to cast some really cool spells on your enemies. But it's arguably even better when your enemies cast those spells on you! Some spells, particularly Illusion, would be too difficult to get the NPCs using correctly. But it is especially awesome when enemy mages make use of utility spells in a tactical way during combat.
Leveled Lists and Merchants
Make sure to get your spells out there in the leveled items list so that an adventuring mage can discover it while exploring the world. Otherwise, they should be able to purchase it from a merchant from somewhere.
Quest Attainment
Particularly powerful spells (Master Level) just beg to be given the quest treatment. It's just so much more satisfying to not only obtain a really powerful spell and add it to your bag of tricks, but to feel like you really earned the spell. This can be a lot of extra work, but it's usually well worth it, even if the quest is dead simple. Of course not everyone wants to do a quest to obtain really powerful spells, but that's what the console is for.
Compatibility with Magic Scaling Mods
One of the most common complaints of the vanilla Skyrim magic system (it doesn't bother me so much), is that the effectiveness of spells do not scale with the players skill level over the related school of magic. Mods which overhaul the magic system so that spells scale with the growth of skill levels are very popular. If possible, it's probably a good idea to think about whether or not your spell will still work well when used with a mod that does this. Obviously, taking the time to test with every single one of those mods is out of the question, but simply making some basic design choices with this general class of mods will be appreciated by your fans.
I want to give a shout to some spells that I think are examples of good design . This isn't an inclusive list or anything, they are just some random spells that I've been using lately in my current play-through, and which meet most of the standards described above. Unfortunately, most of the big spell pack mods out there which add anywhere from a dozen to hundreds of spells only ever contain a small handful of spells that I want to use. I'd think about combining them together into a single pack of the real gems, but I wouldn't want to upset mod authors who put a lot of effort into their work. Anyway, here are some cool spells...
Spell: Reflecting Ward
Mod: http://skyrim.nexusmods.com/downloads/file.php?id=8916 by BaronVonChateau
School: Restoration
Level: Adept
Description: As long as this ward is up, all melee damage done to you is also reflected to your enemy.
The reason I think that this is the best spell included in this mod, is because it seems so obvious that it should have been included in the vanilla game. This has got to be one of the best spells for Paladin archetypes decked out in heavy armor. It's just so satisfying to be in a slugfest with another melee NPC and constantly taunt "Why are you hitting yourself? Why are you hitting yourself?"
Spell: Befuddle
Mod: http://skyrim.nexusmods.com/downloads/file.php?id=16225 by Enai Siaion
School: Illusion
Level: Novice
Description: For 10 seconds, the target is staggered each time it casts a spell.
This is a great "opener" spell against enemy mages, especially destruction mages. I'd really like to see this spell dual-cast capable for a longer lasting version.
Spell: Seduce
Mod: http://skyrim.nexusmods.com/downloads/file.php?id=16225 by Enai Siaion
School: Illusion
Level: Apprentice
Description: Make a humanoid of the opposite gender an ally for 30 seconds or until hit 3 times.
Powerful, useful, and well-balanced. It takes a bit longer to cast than other spells, so you can't abuse it, and you have to keep your eye out for opportunities to use it. But it's great in tight spaces where Fury isn't likely to help you all the much.
Spell: Blood Pact
Mod: http://skyrim.nexusmods.com/downloads/file.php?id=16225 by Enai Siaion
School: Conjuration
Level: Apprentice
Description: Toggle - When active, zombie health is increased by 75 and duration is increased by 50%. Caster health -25.
Even though this kind of feels like a "prism" spell, it really makes sense in the context in which its used. It's also one of the very few Conjuration school spells that are good additions to the game. Conjuration is probably the most difficult school to add new and useful spells to, so this one really stands out.
Spell: Master of Disguise
Mod: http://skyrim.nexusmods.com/downloads/file.php?id=5283 by Concept83
School: Illusion
Level: Expert
Description: With your expert skills in the school of Illusion, you will now be able to target the corpse of a fallen fow, and steal its faction. For the spell to work as intended, you must not be seen from an enemies around you. Will not work against animals or dwarven creatures.
I'd like to see this spell support dual casting and perhaps make it available at the Adept level, but the implementation is fantastic. The spell really works and it has a lot of flavor. Bonus points if the author can change the appearance of the player to mimic the dead corpse.
Spells: Blink/Shadowstep/Teleport/Omnipresence
Mod: http://skyrim.nexusmods.com/downloads/file.php?id=8535 by Sjogga90
School: Alteration
Level: Various
These are fairly straightforward teleport spells. There are a lot of these out there by many different authors. The reason I like this pack is because it puts these spells into what I feel is the correct school (Alteration), where I think most other mods put their teleport spells into Illusion, which I don't think fits in that school. These spells are well balanced and they scale to more powerful versions neatly.
