I Haven't the slightest idea on why this happens, I don't even get a error message when it happens either, it just closes.

I've seen this reported by others as well. Those that admitted to the use of mods specifically mentioned the Midas Magic mod. Are you using any mods that add spells or alter spell behavior in any way?

I do not and I have never been able to reproduce this kind of behavior.

http://skyrim.nexusmods.com/mods/13166

I use Mighty magick skyrim. I can turn off the system for conjuration but that doesn't stop it from happening.