Release 4.3 of TESVSnip is not changed in any major way.

The setting for allowing compression is fixed, so it now can recompress compressed records that has the flag set. But the compressed data is not binary identical to the how the CK compresses it, but the CK and them game can still read it just fine. Same algorithm, different implementation. TESVSnip uses Sharp Zip Lib.

Editing records that have the wrong definition in the RecordStructure.xml fill WILL corrupt the data. Be aware!
I would strongly suggest against editing CELL, NAVI, WRLD records with TESVSNIP.
On second though, I strongly suggest against editing any records you aren't entierly sure about how they work.

It took the biggest plugin I could find (Moonpath to Elsweyr) and loaded it in TESVSnip, resaved it and did a binary compare with zero diffrences. But all plugins with compressed record will show differences due to the difference in the compression implementation.

TESVSnip is a low level editor, you can very easily break the game. You have all been warned!

I redid the what SLuckyD did in the video above with TESVSnip 4.3 and CK 1.6.91.0 but could not get any data corruption, and no patrol points rotated.

SLuckyD forgot to do one step when duplicating the second NPC in TESVSnip, and that was to internalize the characters name. (Remember that the CK does this for you, but TESVSnip leaves you the choice between putting it in the plugin or in the strings file.) But I tested both without doing that step and with it, and the only difference is that without it the second NPC will get a strange short name.

I also loaded up the plugin again in the CK again after duplicating the second NPC in TESVSnip and resaved it, but still no corruption.
The saved plugins are not binary identical due to the different implementation of the compression, but all uncompressed records are binary identical.

I don't understand what you guys are talking about plugin version being 0.85? It was ages ago since TESVSnip save files with that version number. Are you sure you all have been using at least a remotely recent version?

I did however find a thing I forgotten in the release of 4.3, and that was to re-enable the saving of strings files. It got disabled when I implemented the support for UTF-8 character-set since it was in the changes leihao had done. Going to do a new release with that enabled.

Please give me more scenarios to test.

Out of interest, what is the most up-to-date and correct revision of this file? I'm playing with some python code that uses this, and I don't know of any other relatively complete machine-readable specification of the mod file format.

AndalayBay: yea... I FINALLY figured that out myself the other day (having to change the actual config file manually since the options don't work). I did some testing with the compression ON... and you guys are correct. In limited testing, I found that one byte is always deleted from NPC_ records... somewhere toward the front, but since it's compressed I can't pinpoint it exactly (and don't really want to).

I also found (by doing 1:1 file comparisons.. binary comp I think someone called it) something which totally preempts ANY more conversation (from me anyways.. heheh) about using Snip with compression. While it does compress stuff, and it seemingly works fine in-game; the comparison revealed that the type of compression used is completely different from the CK's. The files are identical otherwise (except the missing byte... but that could affect/chop other records I haven't tested as well).

The compressed data is completely different; presumably since Snip is actually an FO3 app that's been ported forward. Amazingly though - CK reads it alright, works in-game, etc... so there is probably legacy code in Skyrim/CK which can still read it I suppose. ANYWAYS - since the format is different, and corruption is guaranteed, I set my Snip back to decompress-only and will probably never look back.


Arthmoor: I started using Gecko to check the data as well... presumably it uses a different algorithm for decompression (de- since one can't edit or save with it yet). It's got a WAAAYS to go before it's usable; mostly on the data-structure side of things. I assume they're getting the basic program down first, then will go back and add all the descriptions/etc. But what it comes down to for me, and presumably others of like mind, is that I'm not content to use ONLY the CK... and currently I see Snip as my only viable option for the occasional deletion, splice-job, or renaming (etc).


Arafhaida: heheh.. that was just to show that it works. There's plenty more that needs to be done to those NPCs for a decent mod. I actually forgot to delete the subrecord above the container items... I set it to zero instead. No big deal - just a bit dirty. The .85 version is set by the CK - after you add or delete one or more records to a mod w a third party app (without manually changing the record count). As you can see in the video, and my own experience during preCK days, this has no apparent effect in-game... but that doesn't mean something's not lurking.


sirtaj: I dunno if there are other versions floating around somewhere (I haven't checked the github yet... been on holiday away from decent wifi), but I've been editing the recordStructure.xml myself - and I've done a LOT compared to the v4.2 file. I haven't checked to see if the 4.3 version made any updates or not. I also agree that it is DEFINITELY a 'buyer-beware' circumstance if one chooses to use Snip of ANY version.

One thing with v4.2 I found - and maybe this is why some people have destroyed mods while others haven't (compression notwithstanding)... if you alter any data using the non-hex editor window (the one with fancy options and whatnot); it IMMEDIATELY inserts 6bytes of data into the subrecord (I tested NPC_ data and a CELL, saw what it did and was like "ummm... no"). This knocks the REST of that entire record's data off-balance, rendering most of it garbage.

Since finding that out, it's joined the compression in the 'do not open till armageddon' pile; thankfully I never used that thing - I prefer to edit the raw hex data, that way I can see what's going on and am able to better control it. I only discovered it while testing my XML changes; and I don't plan to see what 6bytes are inserted... that's just absurd to me when it given AND can be avoided from the get-go.

But for those of you who are as adventurous as I am regarding Snippage... here's my XML file - I'm gunna get in contact with the 4.3 person to see what's what, and possibly hand it over to be included in the future (or as an optional download). If I knew XML was so easy to learn, I woulda fixed it a while ago. Besides the loads of updated tags and data-types, I included my decoding of the navMesh data (as best as I currently know how), WTHR and NPC_ records are much less 'unknown' as well. Now if I only had a C compiler and a few days to learn some C...

http://dl.dropbox.com/u/67168394/RecordStructure.xml
USER BEWARE! This app may destroy your data - use at your own risk! I recommend using it for viewing purposes only; and ONLY mess with files that you have backups of. I also HIGHLY suggest manually disabling the compression and using ONLY the hex-editor (not the fancy one, as nice and convenient as it may be). Snip has a GNU license, and any edits to this file or any other aspect of Snip should adhere to that license. (I will probably upload the XML to the github as requested in the readMe, but there's more I want to do to it before then.. so this link won't be here long!)

UseDefaultRecordCompression and EnableAutoCompress should both be set to False to disable compression.

[EDIT: type-os]

You going to add what you've discovered over here? http://www.uesp.net/wiki/Tes5Mod:Mod_File_Format

That page is pretty valuable for current and future tool authors, and it would svck to see it neglected.

Nope. Gecko uses the exact same compression/decompression library that Snip uses. However, as I've said 15 times now, we found the error about the size parameter and we don't use that to determine the size of the decompressed data. So we decompress all the data and will recompress it all on save.

And yes, we're getting the MVVM architecture defined first. Our model makes Gecko immune to file structure changes so we feel this is time well spent. But it means Gecko is pretty fugly at the moment. It will be much prettier by the time we're done. It is awfully hard to read right now, but all the groups, records and fields are listed.

Thank you! For folks who might be interested in python code that reads ES files, you can take a look at my very early experiments here - https://github.com/sirtaj/skytool
My goal right now is to use the XML file as a starting point to generate a python object model that can load, edit and save ES files. It's going to take a while to get to that point though.

I was actually mistaken in that statement... which I forgot to mention when I posted that video (which disproves the statement heheheh). The reason my esm-hacked NPC was messed up was because I forgot to delete the AI packages (and MAYBE faction data, but I'm not sure how that could play into it). The AI, presumably, sent the actor wherever... instead of where it should be. Another factor was the unique flag - I found that two unique actors cause very strange things. The video shows basically the exact same test that resulted in NPC-swappage.. but I deleted the AI/unique stuff for the video.

My point is that a pureCK clone of an actor can show these symptoms - if the AI or other 'unique' data wasn't changed. As for turning into another NPC altogether.. I dunno - but what I saw of the double-unique (back when CK was first released), I wouldn't be surprised if that was it. I doubt you overlooked such things; so that being given, yea - I'd say Snip did something somehow... that's why I always disclaim my statements (note the 'specific' above.. meaning an isolated instance; in that case the rest of the statement was errant though.. heheh, ya got me!)

Something I forgot to mention in my last post was about NPCs in particular. I already mentioned this, and I'm sure it's a well known phenomenon: if you clone an NPC (and remove the unique flag) and don't ok/save it before changing OTHER data, that data may not actually be changed. I found this to be the case when decoding the NPC record... specifically when have 'protected' checked, then switch from the Traits to the Stats tab - that 'protected' is not only unchecked but it changes to 'encumbrance 2/200' (or something like that). The encumbrance thing is actually just an esthetic bug which I believe is legacy code... the actual data is still correct and the way it should be (though being unchecked IS real).

I did some testing with pureCK compression; ie- a plugin saved in the CK, copied twice; then each copy opened in a separate session of the CK and saved. I used Total Commander to do binary comparisons of the raw data (compressed by CK). This is also using v1.524 of the CK... too many horror stories for me to update yet (maybe next week).

Here's what the CK changes each new session (without User altering anything whatsoever):
- always auto-forwards the "Next FormID"
- changes the last block of data for Cell Children GRUPs, which seems to be a date/time stamp (but is not readily decipherable)
- sometimes the filesize changes; 1byte, 3bytes, or not at all
- if there are multiple records in any GRUP (except CELL and WRLD GRUPs), the order of the records is shuffled; sometimes the last becomes first, sometimes first becomes last
- script properties inside any VMAD subrecord will shift as well (but the data remains intact)

- I'm not sure yet, but there may actually be more than one form of compression the CK uses. I found that NPC_ records will sometimes be 1byte short (shown in the size-flag, and the actual count reflects that properly). I haven't been able to pinpoint it yet (this seems to be what I mentioned when talking about the Snipped NPCs... so probably is NOT any fault of Snip). But each new-session-save shows completely different data for the same NPC... though some of that is obviously the aforementioned VMAD changes (presumably; I plan to test this more in depth, without a VMAD in the record).

Other seemingly innocuous changes the CK makes on a per-session basis:
- QUST records' CTDA subrecord. eg: 'unknown 1' changed from F5 56 to F6 53
- NPC_ records' DNAM subrecord. 'Base Armor/Unarmed?' is changed (eg: 11358 to 11361)
- sometimes every 'datestamp' of every top GRUP type changed from 08 72 00 00 to 11 72 00 00

So to summarize, I think the shuffling of records for no apparent reason is what throws many people off. What I saw in non-compressed records (which the CK saves as such), all the data was completely intact... just the order is rearranged. Otherwise, it seems that only 'harmless' flags and unknowns are altered... one of which seems to be a time-stamp.

[EDIT: I use the Snip-descriptions for lack of a better way to explain which things change. If you want, I can post the actual data addresses (or whatever they're called in programmer-speak).]

It appears TESVSnip is no longer available for download on Nexus.

You're right. Both the original and wmrojer's TesVSnip Continued say:

File under moderation

This file is currently under review by a moderator and is not open for public access. Please do not contact the staff or the owner of the file regarding this issue unless you are the file owner or you might be penalised by the staff of the site.

In other words - Don't call us, we'll call you.

That might be a little harsh if it is just for the corruption issue. It still has its uses and there are lots of warnings.

It has been pretty well publicized at this point that TESVSnip is broken and corrupts your files, and the latest unofficial update to Snip failed to address this. Taking it down qualifies under malware removal at this point as it is pretty clear at this point the authors are willingly distributing the app in a broken state (see the unofficial update to the tool, it did not fix the corruption issue which was at that point common knowledge). The people who made the mod and updated it have had plenty of warning, so personally, I feel it is a good thing Nexus decided to put the hammer down at this point, it makes it very clear that fixing the file corruption issues should be the number one priority of any future updates to the tool.

It'll also be totally useless for armors and races once CK 1.6.91 goes live for real since the record formats for those were very clearly changed.

I feel sad for the person working on the third party tool, but a third party tool has to ensure a minimum of quality, and that minimum is not corrupt our files.

Yea - I just noticed that it may be shut down permanently (the Snip).. I'm not sure that version was really meant for (or up to the task of) 'public' use. But I doubt it's because it was tagged as 'malware'.... if that was the case - few mods would remain online as MANY of them corrupt saveGame data (eg: erroneous scripting). I think it may have more to do with violation of the GNU license if anything... but that's only my best guess.

Irregardful... I got my copies (4.2 and 4.3) so I'm not worried about it... and 4.2 is still available from the github (and prob'ly elsewhere)..

I'm glad you said it and not me, Dave. I get hammered everytime I try to point that out. It's not like every file Snip touched is now hopelessly corrupted. There are plenty more people out there with hopelessly botched games that had nothing to do with Snip and those mods are still up. OK, I shut up now boss. No need to throw more irrefutable arguments at me.

I never even noticed the GPL violation myself, but yeah, definitely in violation there.

That wasn't why it was pulled though, and regardless of what people seem to want to think, it remains a dangerous program and doesn't need to be sitting up on any mod hosting site pretending otherwise. No, it wasn't me who reported it either, but I was damn tempted to.

With as prevalent as Snip has been, I'd not be the least bit surprised to find that most of the people suffering strange bugs are using at least one mod that's been exposed to it. There are numerous pre-CK mods still floating around out there, none of which should be considered reliable for various reasons.

Six weeks of this non-CK crap was bad enough during Fallout 3. That community never recovered from it. There's been a lot more damage to the Skyrim community over this and it's going to take years to get past it.

TESVSnip is more like a pre-Tes5edit tool, IMO. The tool of choice to make mods pre-CK was SkyEdit, and I don't think that was a dangerous program.

Another thing I thought of.. and this is now my best guess.. is that it may be in violation of the Skyrim EULA. Pertaining to the parts about 'thou shall not decompile or reverse engineer'. SkyEdit's under moderation too, as is the 'orig' TESvSnip (not for nothin). If this is the case, it may preclude any work being done on ANY app (such as Gecko, TES5Edit, etc).

I think the only way Snip4.3 would violate the GPL would be if the source hasn't been posted (to git presumably, but I haven't checked). Of course it could just be someone who complained about all this drama, and Nex being obligated to investigate I'm sure, that's where we're at now... regardless the reasons they may have.

But since other known-buggy/destructive apps are still floating around (like the v3.7.3 of the 3DSMax plugin), I doubt it's because of any damage that may occur in the wrong hands... remember I've proven that not ALL mods are necessarily corrupted - those files are still in my Dropbox for the non-believers (whereas I still haven't seen evidence to the contrary, ie: a pureCK plugin and it's 'destroyed' version). Keep in mind that NIFs are 3rd party... so I doubt they have 'jurisdiction' over those types of apps.


On a semi-off topic note (as if we haven't been already... heheh)... I found that doing certain things (in the CK) to quest fragments in a pureCK plugin REALLY messes stuff up. It could randomly delete one of the fragment stages (I think that's what they're called); and other damage that I really can't even pinpoint. All I know is that when this just happened to me (again, pureCK), the onActivate event in a non-related script SOMETIMES doesn't fire.... wow is that strange.

To see for yourself, make a copy of any pureCK plugin that has a custom quest. I deleted a stage I didn't need anymore, since I added it expecting to but never did. Doing that ONE thing, it messed up the whole fragment. Another test I did to replicate it, by editing one of the fragStages' comments (;3 changed to ;100).... that caused the whole fragment to mess up, as well as causing the onActivate stuff and other drama.

I really should start that thread about this kind of thing (corruption of data, regardless of the app) - as this one is pretty Snip-specific.

If they're pursuing all 3rd party tools via that clause in the EULA, that would svck. Snip was targeted for a very good reason. So far as I'm aware, nobody has ever had a complaint about SkyEdit causing the same kind of damage. I've never used it myself simply because its record definitions were not complete so it could touch half of what I would have needed to edit anyway, and didn't have a cleaning function.

I have had the occasional incident where the CK tells me something failed a "validation" but have never been able to pinpoint a way to reproduce it and stopping to exit and reload always fixes it. I have however had occasion to delete stages on multiple occasions, some with fragments, and never once had corruption result from that.

If you're manually editing the data the script explicitly tells you not to, that's not the CK's fault.

No of course not... it was pureCK. It may not happen everytime, as I've done similar operations successfully until now. But that validation thing is what I'm talking about... I assume you had the foresight not to save when seeing that error before restarting the CK. I tried it both ways and it didn't matter, the frag source was altered and it gets compiled automatically when you exit the quest window.


I've just been running through the list of utils on the Nex to see what's been updated or new; and there are several things in moderation status... including ALL the different Snips (there's actually like 5 or so). But other things that (theoretically) would violate that clause are still up - such as NPCEditor and saveGame stuff. I'm not concerned... but if I were a dev on an app with tons of time invested I would be; as well as if I didn't trust Snip as much as I do heheh.

[EDIT: SKSE would theoretically fall into that category as well, since it requires reverse engineering/etc... So I think it's just amounting to a witch hunt, but again - I could care less, but not much heheh. Especially since I don't use SKSE yet.]

No, I let it compile and save. It never corrupted the actual script. It's just something internal to the CK that it has a cow about. Plus, even if it DOES manage to mangle it, the CK is kind enough to stuff an unaltered copy of it in one of the backup folders. As I said, reloading fixes it and you can move on. It's been awhile since the last time I saw it though, so perhaps whatever the issue is has been worked on recently.

I don't think there's some kind of mass sweep over the reverse engineering thing, we'd have heard something about that already. I have no idea why anything other than Snip would have been moderated.

I think OBSE was endorsed by Bethesda when the Steam version of Oblivion was patched to load the relevant DLLs.

the version of skyedit that was on nexus was removed because the uploader did not have the author's permission. You can find that here:

http://www.uesp.net/wiki/Tes5Mod:SkyEdit