[Scripting] Savegame Bloating : V

» Tue Jun 19, 2012 1:52 am

Ok, I'm in a bit of a bind here. I recently released my first Skyrim mod - my Crime Overhaul - and users have run into a rather devastating bug, one that I've been able to replicate (savegame bloating), but I have no idea why it is occurring.

The mod only uses two scripts.

A simple script that simply checks file versions:

Scriptname RenCrimeVersionQuestScript extends Quest  {This is the script to determine which version of RCO is running.}Float VersionFloat PrevVersionMessage Property RenUpdateMessage AutoEvent OnInit()	RegisterForUpdate(5)EndEventEvent OnUpdate()	Version = 0.91	if (Version > PrevVersion)		RenUpdateMessage.Show(PrevVersion, Version)	endif		PrevVersion = VersionEndEvent

And the bigger script, likely where the savegame bloating is:

Scriptname RenCrimeSMCGQuestScript extends Quest  import GameObjectReference Property RenCrimeGuardRing  Auto  Cell Property prevplayercell AutoArmor Property RenGuardRing AutoActor[] myGuardActorArrayActor[] myCivilianActorArrayint playercrimegoldint prevplayercrimegoldFaction Property currentcrimefaction AutoFaction Property nullcrimefaction AutoRace Property actorrace AutoRace Property horserace AutoRace Property dograce AutoRace Property ArgonianRace AutoRace Property ElderRace AutoRace Property BretonRace AutoRace Property DarkElfRace AutoRace Property HighElfRace AutoRace Property ImperialRace AutoRace Property KhajiitRace AutoRace Property NordRace AutoRace Property OrcRace AutoRace Property RedguardRace AutoReferenceAlias Property RenGuardFollowRef  Autobool guardfollowingplayerEvent OnInit()	RegisterForUpdate(0.1)		myGuardActorArray = new Actor[128]	myCivilianActorArray = new Actor[128]	endEventEvent OnStoryCrimeGold(ObjectReference akVictim, ObjectReference akCriminal, Form akFaction, int aiGoldAmount, int aiCrime)	Actor VictimNPC = akVictim as Actor	Actor CriminalNPC = akCriminal as Actor	;Debug.Messagebox(CriminalNPC + " got " + aiGoldAmount + " crime gold for victimizing " + VictimNPC)	endEventEvent OnUpdate()		RegisterForUpdate(0.1)		if Game.GetPlayer().GetParentCell() != prevplayercell || prevplayercell == none		prevplayercell = Game.GetPlayer().GetParentCell()		int x = 0				while x < myGuardActorArray.Length			myGuardActorArray[x] = none			x = x + 1					endWhile				x = 0				while x < myCivilianActorArray.Length			myCivilianActorArray[x] = none			x = x + 1		endWhile					endif		playercrimegold = currentcrimefaction.GetCrimeGold()		if playercrimegold > prevplayercrimegold				int golddifference = playercrimegold - prevplayercrimegold				; player committed a crime		;Debug.Notification("Player committed a crime and added " + golddifference + " to their bounty.")				int x = 0				while x < myGuardActorArray.Length			if Game.GetPlayer().IsDetectedBy(myGuardActorArray[x]) == false				myGuardActorArray[x].StopCombat()			endif		endWhile				x = 0				while x < myCivilianActorArray.Length			if Game.GetPlayer().IsDetectedBy(myCivilianActorArray[x]) == false				myCivilianActorArray[x].StopCombat()			endif		endWhile			endif		prevplayercrimegold = currentcrimefaction.GetCrimeGold()		Actor randomActor = Game.FindRandomActorFromRef(Game.GetPlayer(), 8000)		if randomActor.IsGuard() == 1				currentcrimefaction = randomActor.GetCrimeFaction()				int x = 0		int add = 1				while x < myGuardActorArray.Length && add == 1			if myGuardActorArray[x] == randomActor				add = 0			endif			x = x + 1		endWhile				if add == 1			int arraylength = 0						x = 0						while x < myGuardActorArray.Length				if myGuardActorArray[x] == none					arraylength = x					x = myGuardActorArray.Length				endif				x = x + 1			endWhile						myGuardActorArray[arraylength] = randomActor		endif			else		int x = 0		int add = 1				while x < myCivilianActorArray.Length && add == 1			if myCivilianActorArray[x] == randomActor				add = 0			endif			x = x + 1		endWhile				if add == 1			int arraylength = 0						x = 0						while x < myCivilianActorArray.Length				if myCivilianActorArray[x] == none					arraylength = x					x = myGuardActorArray.Length				endif				x = x + 1			endWhile						myCivilianActorArray[arraylength] = randomActor		endif		endif		int x = 0			bool keepcrimereporton = false			while x < myCivilianActorArray.Length		Race myrace = myCivilianActorArray[x].GetRace()				if myrace == ArgonianRace || myrace == ElderRace || myrace == BretonRace || myrace == DarkElfRace || myrace == HighElfRace || \				myrace == ImperialRace || myrace == KhajiitRace || myrace == NordRace || myrace == OrcRace || myrace == RedguardRace			if myCivilianActorArray[x].IsDead() == false && myCivilianActorArray[x].IsPlayerTeammate() == false && Game.GetPlayer().IsDetectedBy(myCivilianActorArray[x]) == true				keepcrimereporton = true				x = myCivilianActorArray.Length			endif		endif				x = x + 1	endWhile		x = 0			while x < myGuardActorArray.Length		if Game.GetPlayer().IsDetectedBy(myGuardActorArray[x]) == true || Game.GetPlayer().GetDistance(myGuardActorArray[x]) <= 400			keepcrimereporton = true			x = myGuardActorArray.Length		endif		x = x + 1	endWhile		SetPlayerReportCrime(keepcrimereporton)		if keepcrimereporton == true		;Debug.Notification("Turned on crime reporting on player...")		else		;Debug.Notification("Turned off crime reporting on player...")	endif		if Game.GetPlayer().IsSneaking() == true || Game.GetPlayer().IsWeaponDrawn() == true				x = 0				while x < myGuardActorArray.Length			if Game.GetPlayer().IsDetectedBy(myGuardActorArray[x]) == true && guardfollowingplayer == false				RenGuardFollowRef.ForceRefTo(myGuardActorArray[x])				guardfollowingplayer = true				x = myGuardActorArray.Length			endif			x = x + 1		endWhile	else		RenGuardFollowRef.Clear()		guardfollowingplayer = false	endif	endEvent

I cannot figure out what is causing the savegame bloating - anyone who could provide me with some insight would have my utmost gratitude!

» Mon Jun 18, 2012 6:13 pm

The simple answer is http://www.creationkit.com/Save_File_Notes_(Papyrus), also relevant is http://www.creationkit.com/Persistence_(Papyrus).

Edit: You probably should make use of states too avoid possible threading issues.
And honestly, at this point am I rather confused how big mods, with big/many scripts should be possible long term.

» Mon Jun 18, 2012 4:29 pm

The simple answer is http://www.creationkit.com/Save_File_Notes_(Papyrus), also relevant is http://www.creationkit.com/Persistence_(Papyrus).

Edit: You probably should make use of states too avoid possible threading issues.
And honestly, at this point am I rather confused how big mods, with big/many scripts should be possible long term.

Right... but it's two (relatively) simple scripts with no heavy variable use. They're quest scripts, not object scripts.

And we're talking serious bloating - megabytes upon megabytes.

There really isn't any state that is necessary except the default, but it couldn't hurt to try, I suppose.

» Mon Jun 18, 2012 7:30 pm

You only need to call RegisterForUpdate() once.

» Mon Jun 18, 2012 6:18 pm

It looks like you've forgotten to increment your index in your While loops starting on lines 92 and 100, or am I missing something there? Oh, an on that note, the "+=" operator is now available in Papyrus.

Also, you don't need to call http://www.creationkit.com/RegisterForUpdate_-_Form inside your http://www.creationkit.com/OnUpdate_-_Form block, unless for some reason you want to stop http://www.creationkit.com/RegisterForSingleUpdate_-_Form from working and ensure that the update increment is always the same.

Just to test if persistence is the problem (it'll certainly be a contributor, I'm just not sure how much it'll be contributing) try making your arrays much smaller and see if it decreases the bloat.

Cipscis

» Mon Jun 18, 2012 9:41 pm

It looks like you've forgotten to increment your index in your While loops starting on lines 92 and 100, or am I missing something there? Oh, an on that note, the "+=" operator is now available in Papyrus.

Also, you don't need to call http://www.creationkit.com/RegisterForUpdate_-_Form inside your http://www.creationkit.com/OnUpdate_-_Form block, unless for some reason you want to stop http://www.creationkit.com/RegisterForSingleUpdate_-_Form from working and ensure that the update increment is always the same.

Just to test if persistence is the problem (it'll certainly be a contributor, I'm just not sure how much it'll be contributing) try making your arrays much smaller and see if it decreases the bloat.

Cipscis

Thank you, Cipscis. I've fixed those while loop issues. And the reason I called the RegisterForUpdate in the OnUpdate block was in 0.9 I had it at 0.5 instead of 0.1, and speeding it up a bit should increase the speed at which Guards start to follow the player (as well as a call to EvaluatePackage).

Unforunately, it still hasn't lowered the amount of savegame bloating.

» Mon Jun 18, 2012 11:50 pm

Registering for updates when you're already registered shouldn't be a problem, but I just wanted to make sure you were aware of potential issues.

It's probably worth noting that, for such a short update time, it might be possible that multiple instances of your OnUpdate event will be running simultaneously. Take a look here for details - http://www.creationkit.com/Differences_from_Previous_Scripting#One_Script_Running_Multiple_Instances_of_the_Same_Event

I'm not sure whether or not it would be an issue, but it might be worth testing. You should be able to get around it by using states - put your current code in an auto state and create an empty state. At the start of your OnUpdate event send the script into the empty state, then return it to the state containing your code at the end of the event.

Cipscis

» Mon Jun 18, 2012 4:51 pm

You mean like this?

Spoiler

Scriptname RenCrimeSMCGQuestScript extends Quest  import GameObjectReference Property RenCrimeGuardRing  Auto  Cell Property prevplayercell AutoArmor Property RenGuardRing AutoActor[] myGuardActorArrayActor[] myCivilianActorArrayint playercrimegoldint prevplayercrimegoldFaction Property currentcrimefaction AutoFaction Property nullcrimefaction AutoRace Property actorrace AutoRace Property horserace AutoRace Property dograce AutoRace Property ArgonianRace AutoRace Property ElderRace AutoRace Property BretonRace AutoRace Property DarkElfRace AutoRace Property HighElfRace AutoRace Property ImperialRace AutoRace Property KhajiitRace AutoRace Property NordRace AutoRace Property OrcRace AutoRace Property RedguardRace AutoReferenceAlias Property RenGuardFollowRef  Autobool guardfollowingplayerint x = 0int add = 0bool do_once = falseEvent OnInit()    RegisterForSingleUpdate(0.1)        myGuardActorArray = new Actor[30]    myCivilianActorArray = new Actor[30]    do_once = false    endEventauto State StartHereEvent OnInit()    RegisterForSingleUpdate(0.1)        myGuardActorArray = new Actor[30]    myCivilianActorArray = new Actor[30]    do_once = false    endEventEvent OnStoryCrimeGold(ObjectReference akVictim, ObjectReference akCriminal, Form akFaction, int aiGoldAmount, int aiCrime)     Actor VictimNPC = akVictim as Actor    Actor CriminalNPC = akCriminal as Actor     ;Debug.Messagebox(CriminalNPC + " got " + aiGoldAmount + " crime gold for victimizing " + VictimNPC)    endEventEvent OnUpdate()        GotoState("")        if (do_once == false)        myGuardActorArray = new Actor[30]        myCivilianActorArray = new Actor[30]        do_once = true    endif        if Game.GetPlayer().GetParentCell() != prevplayercell || prevplayercell == none        prevplayercell = Game.GetPlayer().GetParentCell()        x = 0                while x < myGuardActorArray.Length            myGuardActorArray[x] = none            x = x + 1                    endWhile                x = 0                while x < myCivilianActorArray.Length            myCivilianActorArray[x] = none            x = x + 1        endWhile                    endif        playercrimegold = currentcrimefaction.GetCrimeGold()        if playercrimegold > prevplayercrimegold                int golddifference = playercrimegold - prevplayercrimegold                ; player committed a crime        ;Debug.Notification("Player committed a crime and added " + golddifference + " to their bounty.")                x = 0                while x < myGuardActorArray.Length            if Game.GetPlayer().IsDetectedBy(myGuardActorArray[x]) == false                myGuardActorArray[x].StopCombat()            endif            x = x + 1        endWhile                x = 0                while x < myCivilianActorArray.Length            if Game.GetPlayer().IsDetectedBy(myCivilianActorArray[x]) == false                myCivilianActorArray[x].StopCombat()            endif            x = x + 1        endWhile            endif        prevplayercrimegold = currentcrimefaction.GetCrimeGold()        Actor randomActor = Game.FindRandomActorFromRef(Game.GetPlayer(), 8000)        if randomActor.IsGuard() == 1                currentcrimefaction = randomActor.GetCrimeFaction()                x = 0        add = 1                while x < myGuardActorArray.Length && add == 1            if myGuardActorArray[x] == randomActor                add = 0            endif            x = x + 1        endWhile                if add == 1            int arraylength = 0                        x = 0                        while x < myGuardActorArray.Length                if myGuardActorArray[x] == none                    arraylength = x                    x = myGuardActorArray.Length                endif                x = x + 1            endWhile                        myGuardActorArray[arraylength] = randomActor        endif            else        x = 0        add = 1                while x < myCivilianActorArray.Length && add == 1            if myCivilianActorArray[x] == randomActor                add = 0            endif            x = x + 1        endWhile                if add == 1            int arraylength = 0                        x = 0                        while x < myCivilianActorArray.Length                if myCivilianActorArray[x] == none                    arraylength = x                    x = myGuardActorArray.Length                endif                x = x + 1            endWhile                        myCivilianActorArray[arraylength] = randomActor        endif        endif        x = 0            bool keepcrimereporton = false            while x < myCivilianActorArray.Length        Race myrace = myCivilianActorArray[x].GetRace()                if myrace == ArgonianRace || myrace == ElderRace || myrace == BretonRace || myrace == DarkElfRace || myrace == HighElfRace || \                myrace == ImperialRace || myrace == KhajiitRace || myrace == NordRace || myrace == OrcRace || myrace == RedguardRace            if myCivilianActorArray[x].IsDead() == false && myCivilianActorArray[x].IsPlayerTeammate() == false && Game.GetPlayer().IsDetectedBy(myCivilianActorArray[x]) == true                keepcrimereporton = true                x = myCivilianActorArray.Length            endif        endif                x = x + 1    endWhile        x = 0            while x < myGuardActorArray.Length        if Game.GetPlayer().IsDetectedBy(myGuardActorArray[x]) == true || Game.GetPlayer().GetDistance(myGuardActorArray[x]) <= 400            keepcrimereporton = true            x = myGuardActorArray.Length        endif        x = x + 1    endWhile        SetPlayerReportCrime(keepcrimereporton)        if keepcrimereporton == true        ;Debug.Notification("Turned on crime reporting on player...")        else        ;Debug.Notification("Turned off crime reporting on player...")    endif        if Game.GetPlayer().IsSneaking() == true || Game.GetPlayer().IsWeaponDrawn() == true                x = 0                while x < myGuardActorArray.Length            if Game.GetPlayer().IsDetectedBy(myGuardActorArray[x]) == true && guardfollowingplayer == false                RenGuardFollowRef.ForceRefTo(myGuardActorArray[x])                guardfollowingplayer = true                x = myGuardActorArray.Length            endif            x = x + 1        endWhile    else        RenGuardFollowRef.Clear()        guardfollowingplayer = false    endif        RegisterForSingleUpdate(0.1)        GotoState("StartHere")    endEventendState

» Tue Jun 19, 2012 7:42 am

You'll need to move your OnInit() event inside your auto state, or I don't expect it will fire.

I expect using GotoState("") would work, but I haven't tested it and it's not mentioned in the wiki documentation, so unless you want to test that explicitly I'd add another state instead of using the empty one.

Cipscis

» Tue Jun 19, 2012 2:53 am

I'd prefer to have an explicit Empty/NoOp state to switch too, but I guess just using the default "" should work too.

A minor optimization would be to combine a few loops, like the re-initialization ones, besides less code is always a good idea.
Edit#2: Another very minor optimization would be to have only one declaration for "x" -- It's quite possible the compiler does this anyway but still, you could save a few bytes stack space

.

Edit#1: OnInit cannot be inside any state other the default one, it will run only once per character anyway.

» Tue Jun 19, 2012 12:33 am

You'll need to move your OnInit() event inside your auto state, or I don't expect it will fire.

I expect using GotoState("") would work, but I haven't tested it and it's not mentioned in the wiki documentation, so unless you want to test that explicitly I'd add another state instead of using the empty one.

Cipscis

OnInit is inside both the StartHere and empty states, and the compiler yelled at me when it was only in the StartHere state.

» Mon Jun 18, 2012 11:29 pm

Oh, so it is. Missed that... What error does the compiler give you if you remove the other OnInit?

Cipscis

» Mon Jun 18, 2012 9:59 pm

Oh, so it is. Missed that... What error does the compiler give you if you remove the other OnInit?

Cipscis

When I remove it from the StartHere state - no error. When I remove it from the empty state, it says:
e:\program files (x86)\steam\steamapps\common\skyrim\Data\Scripts\Source\RenCrimeSMCGQuestScript.psc(49,0): function oninit cannot be defined in state starthere without also being defined in the empty state

» Tue Jun 19, 2012 1:44 am

Huh... I didn't expect that at all. Interesting... I'll add it to the wiki's documentation.

Cipscis

» Mon Jun 18, 2012 5:37 pm

This will work:

event OnInit()endeventauto state Activeendstatestate Emptyendstate

Edit: Another way to potentially remove stack space/allocations -- avoid function calls.
All the Game.GetPlayer() calls probably require their on stack, and since that reference is never going to change, just use a field/variable and call it only once.

» Tue Jun 19, 2012 8:20 am

Huh... I didn't expect that at all. Interesting... I'll add it to the wiki's documentation.

Cipscis

It seems either the State changes, or the fact that I changed RegisterForUpdate to RegisterforSingleUpdate (and subsequent more RegisterForSingleUpdate at the end of the OnUpdate) has removed the bloating issue. Thank you all your help, Cipscis and Xetrill.

» Tue Jun 19, 2012 1:45 am

Great! It'd be nice to narrow it down to one or the other. I expect that the state changes are most likely to be the fix, but it'd be nice to confirm it.

Cipscis

» Tue Jun 19, 2012 3:06 am

Hmm would be good to know what it did in the end, other people are surely going to run into the same issue.

I guess the the CK wiki needs a Best practices for Papyrus page.

» Mon Jun 18, 2012 10:41 pm

Great! It'd be nice to narrow it down to one or the other. I expect that the state changes are most likely to be the fix, but it'd be nice to confirm it.

Cipscis

I'll remove the RegisterForSingleUpdate and see what happens. Most likely it was the State changes, but we'll see in just a second.

Edit:

Odd. When I removed the RegisterForSingleUpdate and changed it to RegisterForUpdate, the script simply stopped running...

I'll keep fiddling with it, but my guess is that changing the states helped to avoid needless multithreading / multiple instances per script.

» Mon Jun 18, 2012 4:59 pm

Your problem was almost certainly the incredibly short time on RegisterForUpdate combined with a very long OnUpdate function. Your OnUpdate couldn’t finish before enough time had passed to send another OnUpdate event, so the events kept piling up. Since the save system has to save everything (including script events that haven’t yet run) you ended up with a massive save. This is called out in the http://www.creationkit.com/OnUpdate_-_Form#Notes. What most likely fixed the issue was switching to RegisterForSingleUpdate, since the events won’t stack on eachother (as long as the register is at the end of your OnUpdate event).

In the future, if you run into this, try typing in “http://www.creationkit.com/DumpPapyrusStacks” into the console (with http://www.creationkit.com/INI_Settings_%28Papyrus%29) which will dump everything Papyrus is currently running, or wants to run, to the script log. In your case I would expect to see thousands of running and pending OnUpdate events.