Wow, I'm now reinstalling the tool kit, modifying the main menu really didn't go well at all lol.
I modified the shell_redo.fla.forbuild file slightly, just to make sure I had the right file, and published it so it spat out the swf version then built the mod to test it. The mod built without errors but when I copied it into my mods directory for Rage64 it loaded up with the rage tool kit version of the main menu (where idStudio and Development Menu are options). After deleting the mod and restarting Rage64 it still had the Rage Tool Kit menu version there and same again when I tried running it with no mods in the directory at all.
Luckily I had previously built a mod with just the stock files in their original condition, when I ran Rage64 with that mod it must have recopied the correct menu back over because the normal menu was back (the one with the mods option) and everything functioned properly even after removing the mod.
I'm not entirely sure what caused it, but I think when I edited the flash file my copy of Flash may have saved/published it in an incompatible version and when the tool kit tried to build it, it just fell back to the tool kits own menu, so I'd be very careful if you are trying to modify the menu as well.
I figured I'd reinstall the whole tool kit just to make sure all my tool kit files are back to stock then try again, I think I'll pay a bit more attention to what's happening next time
On the plus side, now I kind of know why you wind up with a mod a few hundred MB in size even before you edit any of the files.