Uninstalling scripts vs leaving them in but replacing with b : V - Skyrim

» Sun Jun 24, 2012 7:55 am

Don't know about Dawngard as I haven't tried it, but what Amethyst Deceiver is showing is very easy to reproduce. (But remember, Bethesda can, and does, code things in to their application that we cannot. So it's not really a fair comparision anyway.)

Just take a mod you know well (so you will know if it has scripts running) and uninstall/delete it from your data directory. Enable Papyrus logging and restart the game. You should see the "This save relies on data that is no longer available" (not sure of the exact message) and chose to continue. Exit the game and go look at your log file. You'll see the errors Amethyst Deceiver posted.

What I'm saying is, if someone plays through Dawngard, making saves and all, then decided to uninstall it, this issue will come up, based off what I've read.

I'm not doubting that there are errors, im just shocked that the system works in such a... stupid way. It seems like the only way to ensure that this doesnt happen is to start a new game every time you disable a mod.

» Sun Jun 24, 2012 5:42 pm

I'm not doubting that there are errors, im just shocked that the system works in such a... stupid way.

Okay, yeah. I completely agree with you there... it really annoys me personally, because:'
1) I have to start a new game and I really don't enjoy that
2) I get complaints from people using my mod because of problems that I cannot fix...

That's why I now post on my mod to stop the qeust before uninstalling (and the next version will have a "Shutdown" option that does it for them.) That's why I really appreciate what Amethyst Deceiver is doing.

[Rant]
Personally, I think Bethesda SHOULD HAVE KNOWN about this issue. And if some how they didn't, then they were being very sloppy. They are the ones who built this system and any decent tester would have uninstalled a mod and seen the results. (Did they really not think players would uninstall a mod??) So to me, it's just plain wrong that save files are so easily corrupted.

As Thomas Kaira also posted, it's a serious problem. It corrupts save games. What more needs to be said?
[/Rant]

I actually started writing a post to sort of "document" the save game problems but I haven't posted it because I don't think most people are ready for the truth :wink:

» Sun Jun 24, 2012 2:47 pm

Let me add my 2 cents here based on experiences I've had with editing scripts already in the save:

1. Erasing a script does NOT erase the data being kept on it in the save. This means any properties or variables in the old saved version of the script, the game will still attempt to load those every time the script gets loaded (and fail, resulting in an error dump). I actually avoid this now by moving Properties I'm not using anymore to a "Deprecated properties" area down at the very bottom of my script, so that the game will still load them and not dump an error into the log. They won't do anything because I'm not referencing them in the script at all, so they don't do any harm remaining there.

2. If the script was registered to update, even if the "cleaner" script is empty, it will still continue attempting updates. This is why RegisterForSingleUpdate() is preferred, since if the update fails, it won't re-register. You'll also get a truckload of errors about being unable to compare NONEs to types and not being able to call functions on NONE objects when the script makes that inevitable final update. In the end, you'll probably end up with a debug log several megabytes in size by blanking out a script that regularly updates.

Once a script is in your save, it is in there, as-is, PERMANENTLY. Absolutely nothing come hell or high water can change the data stored about that script once it has been saved, and if ANY of that data goes missing, the game will vehemently complain.

I do not think "blanking out" a script as a form of cleanup will have the intended effect. In fact, I think it will actually make things worse, as the log will mushroom big time with complaints about missing variables and properties.

that only happens if the esp still contains the properties in the VMAD. as ive shown in the papyrus dump, i was able to sever ties with the script successfully. i did mention that i killed all active threads by calling Unregister in any scripts that requested one (and i have never used a regular update registry only a single)

most scripts probably don't need to be blanked out, more specifically, they just need to be ended cleanly. since the majority of my scripts are single fire one-liners, blanking them out had no ill effects. more complicated scripts will indeed need more care to end safely (especially if the save is in the middle of a while loop).

while loops, update loops and lengthy registers are the ones to really wath out for. most script functions end within seconds and have little chance of being caught in-process by a save game. if you know your own scripts very well (as you should) you can end them cleanly without leaving error residue.

my main concern right now is, what happens when a user accumulates tens of thousands of scripts from having installed/uninstalled many hundreds of times. just because there is no error in the log doesnt mean the save game isnst still interacting with the script (i am about 99.9% sure that it is, which is the real concern with this method of uninstalling)

» Sun Jun 24, 2012 3:54 am

It really bothers me that Papyrus doesn't garbage collect at all. That's more than an oversight, that's just flat-out lazy. Or rushed, take your pick. You don't even need to wait for Dawnguard, just take a look at your debug logs when you load a save made on an older version of Skyrim, and you'll see a whole bunch of errors relating to vanilla scripts either not being found or missing properties, or whatever.

But whatever the case, it is an astonishing flaw in the system, and I believe if you were to show that to a serious programmer, he would be quite shocked as to how poorly the scripting engine handles garbage-collection (or rather, doesn't).

At this point, I don't care whether or not it was intentional. It is insane that Papyrus doesn't clean up after itself. Unprofessional, actually.

that only happens if the esp still contains the properties in the VMAD. as ive shown in the papyrus dump, i was able to sever ties with the script successfully. i did mention that i killed all active threads by calling Unregister in any scripts that requested one (and i have never used a regular update registry only a single)

most scripts probably don't need to be blanked out, more specifically, they just need to be ended cleanly. since the majority of my scripts are single fire one-liners, blanking them out had no ill effects. more complicated scripts will indeed need more care to end safely (especially if the save is in the middle of a while loop).

while loops, update loops and lengthy registers are the ones to really wath out for. most script functions end within seconds and have little chance of being caught in-process by a save game. if you know your own scripts very well (as you should) you can end them cleanly.

Believe me, when your scripts begin getting several hundred lines long, it gets REALLY tough to get everything when ending the scripts. What I do now in my uninstalls is I reset the script variables to default values and performs any needed UnregisterForUpdates so that you can get back into things with minimal interference if you choose to uninstall and reinstall.

» Sun Jun 24, 2012 12:46 pm

It really bothers me that Papyrus doesn't garbage collect at all. That's more than an oversight, that's just flat-out lazy. Or rushed, take your pick. You don't even need to wait for Dawnguard, just take a look at your debug logs when you load a save made on an older version of Skyrim, and you'll see a whole bunch of errors relating to vanilla scripts either not being found or missing properties, or whatever.

But whatever the case, it is an astonishing flaw in the system, and I believe if you were to show that to a serious programmer, he would be quite shocked as to how poorly the scripting engine handles garbage-collection (or rather, doesn't).

At this point, I don't care whether or not it was intentional. It is insane that Papyrus doesn't clean up after itself. Unprofessional, actually.

Well said.

As a professional programmer (C#/Windows/ASP.NET), I completely agree and probably part of why this bothers me so much. (The other part being having to deal with the mess in my save and my mod.)

Okay, a side note that is related:

I have had reports (but not tested it myself) from one of my beta testers that showed changing the Load Order of the mods as causing problems too. And given that the form id includes the mod load order, it makes sense. So in my new Skyrim build I'm creating for my new new game, I will NOT be changing any load orders. (Again, not a very scalable implmentation of "user mods" on a game that (imo) Relies Heavily on the modding community for it's lasting popularity!.

» Sun Jun 24, 2012 5:47 am

One thing that struck me today: I had a user on one of my mods complaining of save bloat and execution lag when using my mod (script lag persisted after uninstalling) that... magically disappeared when he started a new game.

What does that tell you?

It tells me this: there was so much script junk left in his save from load order alterations/mod updates/mod uninstalls that it was starting to clog up the VM and make it choke. How does that strike you?

(Okay, I'll stop now as I'm building up a head of steam here)

» Sun Jun 24, 2012 7:24 am

well, for those complicated scripts that need to have extra care in stopping, a few things i can think of off the top of my head that should help retrofitting scripts into uninstall mode:

events

OnCellAttach (or any load call) - add a gotostate leading to a blank state so that the script only has a chance to ever run this once.

OnUpdate (or any lengthy time based GameUpdate), always outside of a state, and always has unregisterforupdate inside it.

while loops are trickier, you may have to either create a condition that is 100% passable as false (something like While (Game.GetPlayer.IsDisabled()) - which will immediately knock it out of its loop as soon as the saved thread is finished or you may need to leave the code intact and look for other ways to kill the active thread

as sollar mentioned above calling stop on all your quests will kill a huge chunk of your scripts as well as clear aliases and persisntece (for non specific refs)

this is a huge PIA, and i have to agree the system seems very poorly designed. but unfortunately we have to make the best of it as we can

» Sun Jun 24, 2012 4:53 am

Wait a sec. Scripts attached to aliases aren't deleted with the alias when the esp those alias are defined in is unchecked?

» Sun Jun 24, 2012 2:12 pm

unfortunately not.

i had a script error on an alias that wasnt even filled, on an esp that wasnt loaded.

this is a huge problem

» Sun Jun 24, 2012 10:47 am

Wait a sec. Scripts attached to aliases aren't deleted with the alias when the esp those alias are defined in is unchecked?

Nope. The first time a script is loaded, it gets hardwired into your save and is there for good. Like I said, Papyrus doesn't ever clean up after itself.

» Sun Jun 24, 2012 7:03 pm

Then all those mods that will add aliases to the player character just to use the new and fancy OnPlayerLoadGame...

» Sun Jun 24, 2012 6:09 pm

back with the BSA test:

simply unchecking the esp/esm and leaving the bsa instact left 400+ errors on my papyrus logs (accumulated from 135 scripts missing). as i originally expected, unchecking the esp unloads the bsa entirely (as good as deleted as far as the game is concerned)

using a dummy esp with the bsa intact (modified scripts for uninstall):

i got a clean error log with no errors same as the previous test. was able to make a clean save, but cannot unload the esp or else the bsa disappears

otherwise, there is no difference with BSAs vs loose (MAJOR problem for steamworkshop users, since there is no way to upload an uninstall file on the same page)

IMO BSA is causing more problems to compound this issue. if you use a BSA, you should pack your scripts loose and safely unload the esp after the uninstall process is done, so that the user can leave them visible to their save game (SW users are SOL, which is yet another reason why I will NEVER use steam workshop)

» Sun Jun 24, 2012 10:05 am

OK. Another two horrible doubts. If a mod defines a new creature and the mod is uninstalled making the creature cease to exist, the scripts attached to that creature persist? and what about scripts attached to all the NPCs that are dead and then deleted by the game?

» Sun Jun 24, 2012 7:00 pm

if the script is finished running, its finished running. if there are persistent refs in the script (objectReference properties) those last forever as long as those refs exists in the game (dead, disabled or alive) - although I have a feeling an empty placeholder variable remains persistent in memory in its place. the script OBJECT however is constantly being referenced every time the game loads, even if none of its functions are being used at all, once that script disappears from the scripts folder, the save game will dump a handful of errors complaining about it. it will not however, continue to run the script throughout the active loaded game, if it is already ended properly.

» Sun Jun 24, 2012 2:11 pm

Okay, I just did a test on the "blank scripts" idea and I was indeed able to shut the game up about errors pertaining to my scripts (my debug log now has zero entries for my own scripts on load). Thanks for that, Amethyst. :smile:

I guess I'll be distributing them in future updates, then. If only to keep other people's error logs clean.

» Sun Jun 24, 2012 3:13 pm

you can also safely delete the esp after a clean save is made, so long as the scripts stay in the scripts folder for the entire lifetime of the save game (i swear, papyrus is like a grumpy little kid, you have to leave him with a fake toy to shut him up in his crib)

» Sun Jun 24, 2012 12:23 pm

with this new information, i think the biggest lesson ive learned is how i will plan to approach script writing in the future. i have to assume that users are ignorant to uninstall procedures, and will have to try my best to write scripts that do not linger on (at least as much as possible.) - at the very least call an "unnecessary" condition-guarded unregister inside the onupdate just to be safe.

» Sun Jun 24, 2012 7:58 am

with this new information, i think the biggest lesson ive learned is how i will plan to approach script writing in the future. i have to assume that users are ignorant to uninstall procedures, and will have to try my best to write scripts that do not linger on (at least as much as possible.) - at the very least call an "unnecessary" unregister inside the onupdate just to be safe.

Okay, I've been thinking if there is a way to have the scripts stop registering for updates in the OnUpdate() event. What if there was a global flag that was passed in to the script that if it returns to the default (0) value, it stops. If uninstalling the mod will kill that variable, it would work?

Edit: A "global" variable sticks around in the save, right? I'm thinking the value should return to the default if the mod is gone. But I haven't tested. If not a global variable, then what about a hard coded "true" being set in the property window. That way, when the mod is gone, that value won't be True anymore.

The thing is, does this really solve the problem? The script is still running and still in the save file. If the script properly ends, doesn't it clean up properly? I haven't checked with the SaveGameScanner myself, but from what Thomas Kaira is saying, sounds like it doesn't anyway. Is that your experience also? If it's still in the save file regardless and properly cleaning up doesn't matter, it's worse than I thought so getting the errors to stop is about the best that can be done.

» Sun Jun 24, 2012 5:22 am

as far as i know a script can't run on its own unless it is already threaded,and fed into itself by a loop or another active script. if you kill all threads, the script should be as good as dead.

without an esp installed, the script would continuously throw errors if its associated properties all return a value of none, and since i did not get any of those errors after killing all scripts i concluded that the scripts is dead after the uninstall and only "checked on" during the init process on every game load

» Sun Jun 24, 2012 1:38 pm

i just tried this out and it works:

i loaded one of my older saves that have a bit of "Script missing complaints". i no longer have access to whatever mods those belonged to (since i cant remember some of them, they are from months ago). so what i did was opened the CK with noting loaded, clicked on a object in the CK window (i chose the NPC "player") and add new script.

then i just left it as scriptname whatever-is-the-script-being-complained-about and deleted the extends actor part. left the rest of it blank and compiled it

initially it complained about the lost variables from the original script that was embedded in the save, and after the 2nd save load the log returned no more errors (other than the usual skyrim ones).

so if this whole dummy script thing is necessary to clean up save games, this method is an easy way to get rid of those errors without needing the original source code of a mod long uninstalled.

» Sun Jun 24, 2012 4:21 am

then i just left it as scriptname whatever-is-the-script-being-complained-about and deleted the extends actor part. left the rest of it blank and compiled it

Do you have to keep that new "whatever-is-the-script-being-complained-about" in your scripts folder now? Or can that be cleaned up in any way?

» Sun Jun 24, 2012 4:33 pm

I've only scanned this thread, and I'm on my phone so I can't contribute all that much, but I haven't seen this link to the wiki in this thread and just wanted to add it to the discussion:

http://www.creationkit.com/Save_File_Notes_(Papyrus)

Cipscis

» Sun Jun 24, 2012 3:12 am

Do you have to keep that new "whatever-is-the-script-being-complained-about" in your scripts folder now? Or can that be cleaned up in any way?

no, its there forever as long as i use this save game. at least it's a totally blank script file. it doesnt really do anything but shut the error log up every time the save game loads. without it, each script missing generates a few lines of errors. (at least the embedded properties from the original are now totally gone from the save game)

on another note:
i forgot to mention though, that if your mod changes any vanilla scripts, you may need to run an uninstaller quest depending on how extensive those changes are. then after the user makes their final save, they would have to delete the pex files to restore the orignal code from the bsa

tips for changing vanilla code: NEVER delete existing properties and try to avoid adding new ones if you can

» Sun Jun 24, 2012 1:05 pm

Removing

If you remove a function that was in the middle of running when a save was made, the old function will be loaded from the save and allowed to finish. A warning will be printed to the script log, and further calls to the removed function (usually because some other changed or removed function is calling it) will fail.

The above is wrong, as illustrated:

This:

event onLoad()    on = true    while on == true        chooser = RandomInt(1,3)            rndWaitTimer = RandomFloat(10.0, 30.0)        wait(rndWaitTimer)            if chooser == 1                self.PlayAnimation("PlayAnim01")                mySFX.play(self)                wait(0.5)                placeAtMe(FallingDustExplosion01)                wait(3)                self.PlayAnimation("PlayAnim02")            elseif chooser == 2                self.PlayAnimation("PlayAnim02")                            mySFX.play(self)            elseif chooser == 3                self.PlayAnimation("PlayAnim03")                        mySFX.play(self)            endif    endWhile        endEventevent onUnLoad()    on = falseendEvent

Was changed to this in the USKP:

event OnCellAttach()    on = true    while on == true        if( !Is3DLoaded() )            on = False            Return        EndIf                chooser = RandomInt(1,3)            rndWaitTimer = RandomFloat(10.0, 30.0)        wait(rndWaitTimer)            if chooser == 1                self.PlayAnimation("PlayAnim01")                mySFX.play(self)                wait(0.5)                placeAtMe(FallingDustExplosion01)                wait(3)                self.PlayAnimation("PlayAnim02")            elseif chooser == 2                self.PlayAnimation("PlayAnim02")                            mySFX.play(self)            elseif chooser == 3                self.PlayAnimation("PlayAnim03")                        mySFX.play(self)            endif    endWhile        endEventevent OnCellDetach()    on = falseendEvent

I still see this in every game load:

[06/21/2012 - 02:05:59PM] warning: Function fxDustDropRandomSCRIPT..OnLoad in stack frame 0 in stack 221513 doesn't exist in the in-game resource files - using version from save

Which still throws these all the time:

[06/21/2012 - 02:06:25PM] error: Object reference has no 3Dstack:    [ (0001E68C)].Sound.Play() - "" Line ?    [ (000EBAA9)].fxDustDropRandomSCRIPT.OnLoad() - "" Line ?

The USKP version of the script was designed to fix this stupidity with the OnLoad() block, but Papyrus has decided to disobey the change outright. This should NOT be allowed to happen.

On new games obviously the change takes effect and these scripts remain silent when you're no longer near the objects. It should be noted that in every case where the game has complained in the logs about the object having no 3D, I'm nowhere near one of these things, which means the code for those scripts is executing continuously in the background - and my level 52 Argonian has encountered dozens of these objects which implies dozens of copies of this script are eating away at active script cycles for no good reason.

» Sun Jun 24, 2012 12:55 pm

This is insane... apparently, gamesas designed this system under the assumption that a specific set of scripts would be around at the start of the game and never change throughout play. A reasonable system would notice that a script has been removed, and then remove all references and resources associated with that script since it's obviously no longer present. Instead, THIS happens. Craziness... and now modders have to compensate by creating 'uninstall' mods that strip out all this cruft themselves (assuming that's even possible in all cases).

They should have stuck with encapsulating scripts in the mods themselves. Why did they think it was a good idea to have scripts stored outside of the mods they are implemented in?