[this post is actually a snippet from the readMe/description for my updated XML file for TESvSnip v4.3.1s2.. both available here: http://skyrim.nexusmods.com/mods/21202. Here you will find more information on the Skyrim plugin data structure, which I plan to post in other threads eventually (such as record-types which have localization). And YES this version of Snip is SAFE to use, despite some people's ignorant and/or spiteful contradictions (also explained on the above Description page).]

== The following MAY contain data which seems to be legacy data leftover from some previous game's data structure (Oblivion/Fallout/whatever), cannot be accessed by the public version of the CK, or for whatever reason is changed/destroyed by the CK (sometimes caused by bugs, such as in TACTs). I say "may" because not all these record types necessarily HAVE the specific subrecords - especially if the CK was used to save the data (thus updating to the current standard format).

- I came to these conclusions primarily because these blocks are either destroyed, changed, or replaced when edited in the CK; and usually only a relatively few Vanilla forms of their kind have this legacy data while most are in the 'new' format (sometimes even reflected in the different formID numbering, presumably the legacy data having been created at the beginning of Skyrim's development several years ago, thus having a 'lower' formID #).

Editing ANY part of the following records in the CK triggers the legacy-updating when saved.. which is actually desirable (ie- the data is now stored in the 'new Skyrim standard'). In other words, if you edit some data for any reason whatsoever, then save it in the CK, the entire edited record is updated (ie- deletes/changes those legacy/etc blocks).

***** I do NOT recommend Snip-editing these record-types in their Vanilla state (eg- manually snipping over a record directly from Skyrim.esm, then editing it as-is). The CK should first be opened, the data edited (even if just temporarily changing a minor aspect, such as a single letter in the name), and then SAVED IN A PURE-CK plugin BEFORE editing it in Snip (or ANY 3rd party software).. This will ensure that one doesn't edit a 'worthless' or anathema subrecord, and then wonder why those changes magically disappeared later "for no reason" or why it simply "doesn't work". (25 total, in order of appearance)

CLAS, RACE, SOUN, TACT, ALCH, WTHR, CELL, WRLD, NAVM, ACHR, REFR, PGRE, PHZD, INFO, QUST, PACK, CSTY, WATR, IMGS, IMAD, AVIF, DOBJ*, SCEN, MATO, SNDR

(alphabetical: ACHR, ALCH, AVIF, CELL, CLAS, CSTY, DOBJ*, IMAD, IMGS, INFO, MATO, NAVM, PACK, PGRE, PHZD, QUST, RACE, REFR, SCEN, SNDR, SOUN, TACT, WATR, WRLD, WTHR)

[* - DOBJ record exists in Vanilla, but the CK writes all the data in this record as zeros, PLUS any changes made in the CK. CK-made DOBJ records have an EDID subrecord added (whereas Skyrim.esm's doesn't); this COULD potentially signal that the subsequent data is to be ADDED (ie- not used as an override record). If it doesn't, the original (+changes made manually) would have to be spliced over every time the plugin was saved in the CK. If not respliced, this would (if applicable at all) in turn destroy the entire game by affecting actor's movement, and various other fundamental aspects of mechanics and basic functionality. Since I have no idea what 'Default Objects' do, I see no apparent reason to edit or include this record-type in a plugin; so I recommend not touching it at all unless for experimental purposes. For experimental purposes, the DOBJ data is accessed in the CK through the Gameplay Pulldown, 'Default Objects'.]

ACHR - SCHR subrecord is deleted when record is edited and saved in the CK
ALCH - ENIT subrecord contains legacy data which the Skyrim game-engine doesn't use (addiction data; 1 formID, 1 float)
CELL - some data in interiors' XCLW and XWCU subrecords are reset to 0 when record is simply reopened in the CK (then saved); LNAM and XNAM subrecords are deleted when record is edited and saved in the CK; TVDT and MHDT seem to be unused legacy data which apparently cannot be accessed in the CK (though deleting them have no apparent affect)
CLAS - checkboxes are NOT saved by the CK (recharge/training)
CSTY - CSMD subrecord deleted when record is edited and saved in the CK
DOBJ - In a snipped-over DOBJ record, CK overwrites ALL the data with 0s (zeros), then adds the changes made in the CK (accessed under the Gameplay pulldown); if the original is not included in Skyrim.esm the game's physics and fundamentals don't function properly (that's why the CK can NEVER save Skyrim.esm, un-ESM-flagged, as a viable ESP)
IMAD - contains several subrecords which seemingly cannot be accessed in the CK
IMGS - ENAM subrecord is deleted when record is edited and saved in the CK
INFO - DATA, SCHR, QNAM, NEXT subrecords are deleted when plugin is saved in the CK
MATO - contains Gamebryo-specific data which seemingly cannot be edited in the CK
NAVM - ONAM, PNAM, NNAM subrecords exist only in records contained in a cell which the CK cannot access (00000025 NavMeshGenCellDUPLICATE001) [when this cell is duplicated in Snip, and all the formIDs made unique, the CK autoconverts all the NAVMs to REFRs/primitiveMarkers; I believe these are "cutout" navMeshes for various objects/statics/furniture/etc. I think it's possible that these subrecords aren't even used anymore if the actual NIF/HVK data is read in lieu of them.]
PACK - SCHR, TNAM, SCDA, SCTX, PFOR subrecords are deleted and/or replaced when record is edited and saved in the CK
PGRE - SCHR subrecord is deleted when record is edited and saved in the CK
PHZD - SCHR subrecord is deleted when record is edited and saved in the CK
QUST - SCHR, SCTX, and QNAM subrecords are deleted when saved by the CK (SCTX actually has Oblivion-like script functions still; eg- 0008D5DA); aliases must be individually edited and saved to be updated in the record
RACE - Vanilla DATA subrecords have 36bytes added to them when saved in the CK, which is seemingly inaccessible in the CK; the DATA subrecord also has a block which is always reset if User opens the record in the CK then saves it (any tab, without editing), this is the NoCombatInWater flag and it gets Checked by the CK (presumably by a bug); contains data which seemingly cannot be accessed in the CK
REFR - XMBP and SCHR subrecords are deleted when record is edited and saved in the CK
SCEN - SCHR, SCTX, NEXT (when in a specific position), and QNAM subrecords are deleted when saved by the CK
SNDR - FNAM subrecord is deleted when record is edited and saved in the CK
SOUN - FNAM and SNDD subrecords are deleted when record is edited and saved in the CK
TACT - PNAM subrecord value reset to 0 when record is edited and saved in the CK
WATR - DNAM subrecord contains data which seemingly cannot be accessed in the CK
WRLD - RNAM and OFST subrecords are deleted when record is edited and saved in the CK; MHDT subrecord contains data which seemingly cannot be accessed in the CK (though manually deleting it has no apparent affect)
WTHR - DNAM, CNAM, ANAM, BNAM, ONAM, NAM2, NAM3 are deleted when record is edited and saved in the CK; contains data which seemingly cannot be accessed in the CK
(for more specific info on these, see the XML file in the above link for my notes)



== The following FOUR record-types contain data which is COMPRESSED by the CK, as well as being found that way in the Vanilla ESMs. If this data is saved uncompressed (eg- by Snip while compression is disabled), the game and the CK can still read it perfectly fine. If an uncompressed plugin is opened by the CK, then immediately saved, it re-compresses the plugin and may harmlessly shuffle some of the data (for unknown reasons, perhaps to make it harder for 3rd party apps to "clean" plugins).

CELL, LAND, NAVM, NPC_

I do NOT recommend attempting to use compression on any record-types other than these four. Personally, I recommend disabling compression altogether in Snip, then CK-save the plugin before final testing, and compare the before/after to ensure all data is intact (THEN do final testing in-game). If you do attempt compression (especially on records not intended for it), as always, please experiment and test fully before releasing anything publicly; and keep your backup plugins well organized (Snip v4.3.1s2 auto-backs-up so you have no excuse if YOU mess something up by doing whatever).


All the rest of the data-types should be stored by Snip as binary-identical to the same CK-saved plugin. Exceptions to this are anything which has scripting attached to it in any way (containing a VMAD subrecord)..

***** For whatever reason, in VMAD-bearing records, the CK actually SHUFFLES the order of any filled properties in the raw data - each and every time the file is saved in a new CK session. To see for yourself; save a plugin (with something scripted and having multiple properties), exit the CK, restart/reload the CK, and immediately save it again (without editing anything) - those two files will have the VMAD data different.. the second-one-saved has been shuffled, and possibly other data in the plugin shuffles as well.


I call this the 'CK SHUFFLE' and it appears to be harmless, in any event. A ramefication of this though, is having difficulty identifying 'dirty edits' and dupe records.. since it's possible that the CK will shuffle data in ALL records it saves, including those dirty records. In other words, they won't be identical to the Skyrim.esm record anymore - and will show as different in a binary comparison from plugin to plugin; even though ALL the data is actually the same (just stored in a different order). As of 10-14-12, both TES5Edit and Gecko fail to recognize CK-Shuffled data as a dirty edit; hopefully this will be resolved eventually (though I suspect that will take a VERY long time).

Additionally, the order in which a group of records appears in the plugin is also sometimes shuffled by the CK (eg- the first record stored may become the last, bumping all the rest 'up one').

[RACE records have various subrecords shuffled, even when saved in the same CK session]

*****
Because of this CK-Shuffling of data, the CK's compression algorithms use different 'key' data-blocks from one file to another; therefore the resulting file may be larger or smaller than other versions of the SAME file saved in a different session of the CK. Depending on the content of one's plugin, this differential may be HUNDREDS of bytes or even kilobytes (such as with my SPODUM Mannequin mod - which has over 60 new actors, all having scripts). Again, this is CAUSED BY THE CK, without User having made ANY changes to the plugin - it happens simply by opening the file, saving it, exit, backup, reopen, resave, and compare. For an explanation as to why this is, see the above link's Description page.
*****

Wow that's impressive. To bad you could not make a utility in Visual Basic or C++ that scanned a mod and gave option to remove unwanted junk

I updated the original post to reflect more accurate info (eg- SCEN records).

silhouett - There are at least two projects (Edit and Gecko) already working to create your requested "Clean Plugin" feature.. so I won't be doing it myself. Even if they weren't already working on it, I still wouldn't touch it with a stick - personally I believe that no 3rd party software will ever be able to correctly identify ALL dirty edits in a plugin with 100% accuracy.. if any of them do, it most probably won't be ready for public usage for MONTHS AND MONTHS (those projects are months away from being viable apps as-is, then tack on THIS drama..).

To remove unwanted junk, I recommend using http://skyrim.nexusmods.com/mods/21202. This version of this app is safe to use, contrary to some people's false or uninformed claims.. see the Description page for details. While there is no 'one-button' cleanup solution, one may still view the data and redact as one sees fit. Do so at your own educated peril, since deleting the wrong things or manually placing things in the wrong order may cause your plugin to stop working (and if loaded/saved in the CK may become irreversibly corrupt).

Or people could just use TES5Edit, which is not "months and months away" but is available right now, and is perfectly safe.

^This. Why bother with anything less than TES5Edit now that it's publicly available?

I seem to recall somebody stating TES5Edit was a "pipe dream". Ha! It's dreamy.

OP: http://code.google.com/p/skyrim-plugin-decoding-project/downloads/list, open your plugin and R-Click it when it's all loaded, then select "Apply FIlter" and set up your checkboxes http://i.imgur.com/4x4J8.png. When all is filtered, R-Click your plugin and "Undelete and disable deleted references" then "Remove Identical to master records", save/quit and you're done. You plugin will then be squeaky clean. If you see, in the right panel, that NavMeshes were removed when cleaning, you'll want to save your plugin in the CK to rebuild NAVI.

I've never understood how a utitlity can tell what's "dirty".

I've had a couple of unwanted edits but utilities have always reported my mod as clean.

I think I'll start looking at tes5edit now it's safe and stuff

There are multiple kinds of bad edits:

1) Unintentional changes to objects

These unwanted edits can't be fixed automatically because there's no way to tell them apart from intentional changes. Certain special cases might be possible to fix, but as those cases are found they get their own name.

2) Dirty records

This is a very specific term meaning a record that is in a mod but identical to what is in the mod's master. These can happen by simply selecting an object, moving it slightly, and then choosing Undo to put it back. It still get's flagged as changed. It can also happen when you duplicate an existing item. The original might gettagged as changed even if it isn't. Having these extra objects in your mod increases its size and can lead to unnecessary conflicts with other mods. Fortunately, the fix can be automated because it just means deleting the duplicated change.

3) Marking original objects as deleted in a mod

This can lead to crashes if another mod tries to reference the same object. The fix for this can be automated too. These references can be undeleted, marked as disabled, and potentially moved downward an insane distance (just to make sure it doesn't appear).

While I haven't looked at any code regarding how the apps actually do it, here's my educated guess: The app would decompress to memory/swap the data from both Skyrim.esm (and Update.esm ideally) and the plugin to clean. It would then do a binary comparison to see if any records are identical - which would be flagged as dirty and deleted.

The problem I've been trying to get people to realise is that Skyrim is NOT like any previous games, in the respect that the CK shuffles data; specifically the order in which script properties are stored. In addition to the above simple process, special code needs to be added to the one-click-clean algorithms to chop up the scripted records' data into it's components, then compare THEM (and ALL possible combinations they MAY be stored in).. and not the entire record (as I assume it does now). Theoretically, it can be done - but not until the VMAD records (scripts) are completely decoded. (I decoded most of it, but there are some stubborn blocks that I've just given up on.. hopefully someone else can figure it out)


Example of CK-Shuffling: a plugin alters a Vanilla actor with a script attached to it. The script has 4 properties, ABCD.. and that's the order they are stored in Skyrim.esm and maybe the actual plugin the first time the CK saves it. Now say a week later you change the name of that actor and save the plugin (the script doesn't even need to be edited, just saved in a new session of the CK). The actual data in the plugin may now show the properties stored as DABC, BCDA, or possibly other ways (it's been a while since I first identified this phenomenon so I'd have to go back through my notes, or do a couple quick tests).

Now say another week later, you decide to change EVERYTHING back to the way it was in Vanilla (given that it was a nilla actor and not custom to begin with; in which case it wouldn't be a dirty edit). When the CK saves it this third time, those nilla properties may now be stored in the order of CDAB or whatever else.. obviously not the same as it was in Skyrim.esm. Even though everything is the same as Vanilla, because that order has shuffled, it is now a dirty edit - which cannot be identified by current apps... I tested this with both Gecko and Edit (both are the latest version, downloaded two days ago). The order of properties has no affect on anything, and may explain why Ma Beth chose these blocks of data to shuffle (perhaps to slow 3rd party apps' dev down a bit.. who knows?).

The reason I believe this is important is because of the sheer number of mods which have scripted things in them - NPCs, activators, spells, quests (especially), etc. People using the current versions' one-click solutions are falsely believing their mods to be completely clean (and maybe they ARE, given they didn't dirty up a scripted record)... and if loaded AFTER other mods which intentionally change that dirty edit, will nullify them - forcing people to juggle their load orders. Unless certain people are stopped from spreading misinformation, which is easily verifiable, I forsee a bunch of questions being asked in the near future as to why this that and the other thing isn't working, or why whatever conflicts.

Some people may not think this important, and to some degree I agree - most dirty edits are REFRs and ACHRs (placed objects in worldSpace).. and may not be scripted. But I hope you see my point in that not all dirty edits are that simple.. quests in particular are VERY touchy - they may save to your plugin if you simply OPEN one without editing it (though I'm not sure on that one).


To verify the CK-Shuffle yourself: open the CK, change "playerHouseMannequin" to "playerLouseMannequin" and save it. Exit the CK, restart and reload the plugin (after backing it up, so you have something to compare to of course). Now change the name back to House and save it again - theoretically, the record SHOULD be identical to Vanilla, right? But a binary comparison shows nearly the entire record is different.

I explain how and whythis 'total makeover' happens on the Description page of my http://skyrim.nexusmods.com/mods/21202 'mod'... but the basic gist of it is that the internal compression algorithms identify DIFFERENT 'KEY' BLOCKS (because one property now comes before whichever one did before), and therefore compresses all the following data accordingly. This also happens to be a generalization of why Snip is safe to use (as opposed to the farce of it having non-compatible compression which corrupts data).. because it uses different INTERNAL compression - but the data, when decompressed, is still the same.

I forgot to respond to JustinOther (as if that last post wasn't long enough).. and I hope you don't take this as an inflammatory response - just an explanation/elaboration.

Several months ago I said it was a pipe-dream; and as a completely finished, bug-free app it still is (meaning several months away, which the dev said himself). It isn't safe to use on ALL plugins (just as the old v4.2 of Snip wasn't).. it WILL corrupt certain kinds of data. I assume you've read that other thread, so I won't repeat the specifics.

But by all means, use it to test and experiment with, even release an "Edited" mod - but check your data FREQUENTLY to ensure nothing has been corrupted or lost; and please don't tell people that the app is 100% safe and ready to use, that simply isn't true and anyone can verify that. (I know that's not what you said, but only that it's "publicly available"... but a lot of people trust your word, and may infer more than they should)

I've done all sorts of stuff with TES5Edit and it's solid. I trust it more than any other app save the CK itself If 'cleaning' a plugin, TES5Edit is the right tool for the job as it will nigh invariably perform as expected.

If you've encountered an instance where TES5Edit mangles a plugin, please link it either here or in the TES5Edit thread that we may attempt to reproduce the problem. It might even be that the plugin is already deformed and TES5Edit can't save it. Just 'cause it'll open in TESVSnip doesn't mean everything's okay.

The test plugin is pure CK - just make a simple edit to any BodyPartData. Open it in Edit and it shows a bunch of errors - presumably because it doesn't recognize the subrecords. When that plugin is saved in Edit, that BPTD is all but destroyed. When I saw that, I stopped testing Edit as it was clear to me it isn't ready yet... who knows what else it messes with? Snip has nothing to do with it and wasn't used in the process whatsoever.

[EDIT - and based on the original post, you still trust the CK? heheheh.. now that I know which things it messes with I trust MORE, but still not completely. Granted, most of the above is legacy data which is updated by the CK to the new format.. but if you read through all of it, you'll notice that it messes with stuff that it shouldn't, presumably because of bugs.]

I'm not sure where you're reading into this that I said VMAD support is perfect and identifies dirty records. Right now VMAD support in Edit doesn't really exist, but it won't simply get thrown out either. For you to claim it will shows a gross lack of understanding of how Edit works. Most of it is stored in a giant hex array that Edit is told means something, so it won't ignore it. The records, as I understand it, are already decoded. It's something internal to how Edit works that wasn't necessary in previous games so Elminster will probably need to write the code for it himself.

Also, I am fully aware that the order of the properties isn't consistent when a VMAD block is saved. What you're failing to understand is that Edit has methods for addressing this, and with the "scrambled" order of the TINI subrecords as well (for texture layers on _NPC). In essence, Edit is capable of unshuffling the data for proper comparison. This reordering of things isn't new to Skyrim either. It's been around since Morrowind.

Your body part template example is flat out wrong though. There's one of those edits in the Patch 1.8 version of Update.esm and Edit loaded it just fine without throwing a single error. Frankly I think in your zeal to cover up for Snip's faults you've badly underestimated Sharlikran and Zilav. They're far more capable than you appear to be giving them credit for.

I know you don't give a rat's rip about my opinion on this either, but I've used Edit extensively and trust it completely. It's in good hands. It hasn't corrupted a single thing yet, and two of my mods as well as the USKP are already in the wild with cleaned data. You probably have no idea the trickery I've engaged in with Edit for Oblivion (and Fallout too) that simply can't be done in the CS/CK. It's solid. There's no reason to distrust it.

Also, yes, I still trust the CK as the ultimate authority in much the same way I trust the Oblivion CS to be the ultimate authority for that game. There can be no substitute for official tools. Only supplements.

BTW, you seem to be under the impression that we're suggesting Edit etc will allow violation of the "Rule of One". No such position has been taken. People will still need to juggle load order even with these tools. The only app that will soon be capable of that feat is Wrye Bash - and even it only exploits the Rule of One for its purposes.

You and others have been claiming that the one-click clean in Edit works.. without mentioning any the obvious caveat. It doesn't take any reading into to see that is wrong. [EDIT - I never 'claimed that Edit throws out VMAD data'.]

I address most of the other issues you talk about http://www.gamesas.com/topic/1417195-relz-tes5edit/page__gopid__21657702#entry21657702 and http://www.gamesas.com/topic/1415885-mod-cleaning/.

I agree that the CK is the "authority", but that doesn't mean it should be trusted.

Regarding load order - I never said anything of the sort (about load-ordering not being needed/rule-o-one being violated). I said that load order juggling will get complicated (paraphrased), if/when people with dirty edits NOT picked up by Edit's one-click-clean wonder why their mod conflicts with others. Believing your claims that the one-click works and should be used without worry, these people will probably wonder why these conflicts occur.. and probably start juggling load order to rectify it.

And you're wrong about me not caring about your opinions on this or anything else. While I refuse to take anything you say at face value (without verifying it myself), I apply the same test to everyone.. as I think others taking ANYONE's advice should. The only reason I DO care about your opinions is because they sometimes lead innocent people to believe my work is faulty and will destroy their stuff.. just as I would care that there's a bomb in my living room about to go off.

[EDIT - I don't recall which thread it was in, but you told me that I should use the wiki's data structure as opposed to doing my own research. When I first began updating Snip's XML file, I TRIED using the wiki but quickly found that it was VASTLY incomplete and sometimes even wrong. I don't care if it's the go-to authority, it didn't suit my needs.

Having just checked back to the wiki, I notice it has been updated recently, but is still less complete than my XML/research. For instance, NAVMs are still listed as complete unknowns.. even though I posted nearly ALL their decoded data MONTHS ago, right here in Ma Beth's forum; and are contained in my XML for Snip (though because of the app's current limitations, cannot be displayed entirely in decoded format).

For another instance, why is there no mention of VMAD subrecords in the NPC_ data? You say VMADs (may?) have been decoded completely, please link to the page where that data is shown so I can see for myself. There are a few blocks of data which vary from plugin to plugin, yet I couldn't find anything in the CK which affects it... hopefully your linked page will shed some light on it for me.]

http://www.uesp.net/wiki/Tes5Mod:Mod_File_Format/VMAD_Field.

That's because the cleaning process in Edit does work, but anyone with half a brain knows it can't act on non-decoded data. It's been made crystal clear I don't know how many times that VMAD won't be processed for cleanup. It won't be thrown out though, and yes you HAVE been claiming it would be.

Yes, actually it does mean it should be trusted, because if it can't be trusted then modding grinds to a halt. I'd sure like to know just how Edit could be useful to start placing multiple objects into a cell somewhere without knowing exactly where you're putting it. Sure, it COULD be done, but I think the tediousness of that process ought to prove that the CK is not only necessary, but vital.

See, this is the problem I have with you. You take "one click cleaning works" and conflate that into "you won't have to juggle load order". That has NEVER been true throughout Edit's entire history of existence. That's what Wrye Bash is meant to help with.

Ancient history. The wiki was, at the time, the most complete record of things we had. I even helped post some of that information at the time. That's changed though since Sharlikran's decoding project took off in earnest and now Edit has the most complete decoding of records there is - and the wiki is being updated as time permits them to do so. I'm sure they'll get to posting the NAVM decoding when they have time. Edit however has a pretty strong version of it listed, with all the information shown that's been decoded so far. It's a vast improvement over just a giant blob of unknown hex info.

ShadeMe beat me to it, but I would have pointed you to the TES5Edit source code so you could get it directly from there.

As for the unknowns with no known means to affect it in the CK, it's well known that the CK drops data in with no purpose. A lot of the ones we now know for sure have no impact on things are marked to ignore for purposes of dirty edit identification. The ones that aren't known are listed as such but will NOT trigger a cleanup response.