Forms unsafe to touch with TESVSnip

Post » Thu Jun 21, 2012 4:15 pm

Oh joy. Looks like I should rebuild Diverse Guards Skyrim, then. I used Snip in the early days to cut and paste NPCs (although no NPC was ever edited in Snip). It's odd that no-one has reported any problems that weren't attributable to something else, though. Just lucky, I suppose. Urgh.
User avatar
CHANONE
 
Posts: 3377
Joined: Fri Mar 30, 2007 10:04 am

Post » Thu Jun 21, 2012 4:19 pm

Don't misunderstand me... by claiming that the CK is what corrupts things, I mean that under VERY specific conditions - CERTAINLY not always. Try this: create a backup copy of any 100% flawless mod that has a custom interior with custom navMesh (I hadn't tested it with exteriors so I dunno if it applies). Now do what I call a "selective-recast autogen"... ie: select a big 'structural' static (such as a floor kitPiece), then recast-autogen navMesh (this should only autogen navMesh on top of that static, while it SHOULD leave all else).

The CK immediately corrupts all the existing navMesh - specifically the water/preferred flags for each tri. Not ALL tri's are affected... and I'm actually not sure if this affects navMesh that has no water/preferred flags to begin with. One CAN correct this, but it's a pain in the b's (find and correct the tri's that were changed) - and not everyone may be aware of it until in-game testing... that's how I discovered it.

Outside of that, I haven't had any major problems with the CK. All the session/procedure-specific proof I showed in the NavMesh Bug threads MAY be chalked up to being caused an aspect of the Bug... so I can't offer that as real CK corruption until we know NMB has been fixed (and if those sess-specific issues persist).


I DO experience crashes of the CK when I work on my scripting - which I do extensively. Since it crashes after about a dozen 'compiles', I stopped loading any mods while doing it. I load ONLY the CK... no mods, no skyrim.esm, no NOTHING.... crashes after about a dozen compiles. This is garbage-collection bug, which is obviously specific to the CK's Papyrus routines. Months ago, CK would crash randomly - but as others have already noted, I think that was fixed in the CK's 1.524 update (which is the ONLY CK update I'm aware of).

"Assert File: C:\_Skyrim\Code\TESV\BSCore\MemoryContextTracker.cpp Line:151" is the actual error that ALWAYS occurs. [choosing abort/retry/ignore all cause CTD]

To prevent crashes, or session-specific issues, I now ALWAYS use the CK to do exactly what I want, save it, then get out. In compiling scripts, once they're in a test-able state, THEN I load the mod, assign/fill the properties accordingly, save and exit. I haven't had ANY problems with any of my mods since doing this; and I agree it may be overkill or unnecessary, but I know it works. And I think it's noteworthy that I haven't EVER had any problems with Papyrus whatsoever... it's simply been a matter of learning the new functions & syntax... and dealing with its idiosyncrasies (eg: thread limitations).

I agree with Arthmoor that the CK is the standard - it's the only app sanctioned by the game's developers. But we all have to remember the sheer amount of hackage they had to perpetrate on it to release it to the public... several MONTHS worth (after releasing the game itself). Soooo much could have (and obviously HAS) gone wrong in that process that I almost recant all I ever said preCK about them taking so long to publicly release it.


NOW - regarding what I was saying about the CK being what corrupts NPC_ records: Using Gecko AND Snip, I've been comparing the data of pureCK mods vs Snipped mods (uncompressed data, and supposedly 'corrupt without question' as some continue to claim). My limited testing has shown NO difference in the actual data whatsoever... unless Snipping something out of Skyrim.esm (or caused by the CK during a later phase). This is NOT to say that plenty of instances CAN be corrupt... but again, my orig statements were that not ALL Snipped mods are inherently corrupt.

Try this: open CK, clone any Vanilla actor whatsoever... but be sure to remove the 'unique' flag (as leaving it always causes in-game drama... since it's not actually unique anymore). Place one of that object somewhere in the renderWindow, place & link a couple patrolMarkers so it doesn't 'wander', save the plugin.
- Create a backup of that pureCK plugin file
- Snip-edit the plugin... not even editing is needed - just save the plugin in Snip (but change the edID if you want to go the extra mile)
- Now compare the files. You'll see one is like twice the size as the other (uncompressed)... but in either Snip OR Gecko, ALL the actual data is exactly the same
- Now create a backup of the Snipped plugin file and load up Skyrim.esm (in the same Snip session)
- Directly Snip over any Vanilla NPC_ you want, change your ACHR (placed NPC_ ref) to point to your Snipped-clone (the NAME subrecord), save it
- Test that plugin in-game... it should work fine. (ie: the actor will have the correct name and should be patrolling as expected)
- Backup that plugin, open it in the CK. Say 'yes' to correct the record count. View and compare the file (WITHOUT SAVING IT)... it now has the proper record count, and the 'file version' is the ONLY other data that has changed. In-game this file should still work fine. When compared, the data now shows that the file version has changed to .85 (which doesn't seem to make a difference).
- Now SAVE that file in the CK. Going deeper into the data, you'll notice that the Snipped-clone NPC_'s FULL subrecord has had the last byte deleted (but the size-flag-bytes correctly reflect that). In-game, this file will show two ascii characters instead of the proper name. PROOF OF CK CORRUPTION. This can actually be corrected if the last byte is added back (in Snip).

BUT other damage may have been done as well, as the CK changes some various 'flags/unknowns' (as Snip labels them) - and even changed the order in which the actual NPC_ record appears (mine bumped this record to above/before the pureCK-clone). It also changed the VMAD subrecord so that one property appeared before the other two - but all the data itself was the same. But testing in-game, you'd be lucky to even SEE this actor (disappears completely.. as in it has NOT wandered).

"But Snip is what caused that" you're probably screaming at your screen. NO it didn't... backup that last file and change the NAME subrecord back to the orig pureCK-clone. It should work just fine with your originally placed actor.

Again, I haven't gotten to making the hard data into a presentable/postable form yet... I've been working on decoding more of the NPC_ data structure (and actually have found quite a bit that Snip either has mislabeled or never labelled at all).

I also have YET to see a single instance of ANY record or subrecord that was incorrectly 'stamped' with the wrong size (data either being added or deleted and not being accounted for). As AndalayBay said, each record/subrecord has two bytes that follow the record's name (eg: CNAM, REFR, NAVM, etc)... this is converted to a short which indicates how big that record should be - and has ALWAYS been correct in my limited testing. (which disproves the claim the Snip ALWAYS corrupts or causes this) I'd still like to hear which record whoever has seen this happen to (ideally with the method to reproduce it)... in my prelim opinion, it sounds like someone translated the data wrongly (eg: "13 00" taken as indicating 13 bytes long, but is actually 19).


Once again, I couldn't get Snip to actually save a plugin with an intentionally compressed-flagged record... no matter what changes/options I made to Snip. It DISPLAYS it as being flagged, but saving then exiting and reloading in Snip - that flag is gone. Could anyone tell me how to go about doing that so I can see this for myself?


Not for nothin' - but wasn't 'Voldermort' correct in his claims that there is in fact a navMesh bug? I believe he got the bad wrap he did because he spoke out about it in a way that could have been formulated/delivered with more prudence and politeness... to put it mildly. I'm not THAT familiar with 'he who probably shouldn't be referred to' but I did see the videos months ago and know the basic premise of his argument.

[EDIT: type-o's]
[EDIT2: I put the .85 thing on under the wrong step.. corrected now. CK changes it to .85 WITHOUT actually having to resave the file.]
User avatar
DAVId Bryant
 
Posts: 3366
Joined: Wed Nov 14, 2007 11:41 pm

Post » Thu Jun 21, 2012 11:16 pm

It's pretty easy to tell when your mod causes voids to appear in dungeons, then mysteriously doesn't do that anymore once you've rebuilt it :tongue:

Since there are no reliable data anolysis tools out yet, it isn't really possible to produce hard evidence one can turn into screenshots to satisfy the silly rule on the internet of "pics or it didn't happen". Using Snip to confirm Snip broke something isn't really going to be all that useful. You have to observe how your mod behaves and pay attention to user reports of issues. After awhile, it becomes too large a problem to ignore.

Or, you could bust out a hex editing program and manually check the data, but gah, that's just asking for torture.

That the CK has bugs isn't really at issue here. NONE of what you just listed results in corrupted files, so I don't know why people keep raising the strawman of "it has bugs" as meaning anything in this.

Thanks for the answer, I don't think I have corrupted files.

Regarding CK, we keep talking about him, because, errrrr...... Because its the tool we all use? And since it has a lot of broken, missing and buggy functions we have to use third party programs. If CK worked properly in the first place none of this would happen, but CK fails to do simple tasks like duplicating a worldspace, from the skyrim.esm or from another esp, the result is this:

http://www.mediafire.com/imageview.php?quickkey=iw8mfqa76f3zl5w

So yeah, all comes down to the buggy CK not making what he is suppose to do. And don't get me wrong, I'm thankful that Bethesda gave us this, but I would much prefer that the buggy parts didn't come at all, if its not for us to use, why include them?
User avatar
Laura Richards
 
Posts: 3468
Joined: Mon Aug 28, 2006 4:42 am

Post » Thu Jun 21, 2012 3:20 pm

[Not even going to bother with repeating myself since SOMEONE keeps insisting I'm full of it without having the balls to say so]

Not for nothin' - but wasn't 'Voldermort' correct in his claims that there is in fact a navMesh bug? I believe he got the bad wrap he did because he spoke out about it in a way that could have been formulated/delivered with more prudence and politeness... to put it mildly. I'm not THAT familiar with 'he who probably shouldn't be referred to' but I did see the videos months ago and know the basic premise of his argument.
Yes, he was, and nobody was ever disputing that he was... well... except for ONE person... but I'll spare him the humiliation of claiming there was no bug when it was patently obvious it existed.

His problem wasn't in claiming there was a bug, or in producing a video to prove it. It was the fact that he insisted on going about it by being a complete [censored]. So when Bethesda got sick of dealing with him and nobody else bothered to report it because the community got around it themselves, he just interpreted that to mean everyone was trolling him and he was being unfairly sidelined.

When we got the CK beta, the navmesh bug was one of the first things some of us ran into. The intermittent nature of it was problematic and made testing for it difficult at best. I even resorted to citing "Voldemort's" video on the subject as confirmation of the findings I had come to on my own. There were people from the very beginning who were going to be content with the use of 3rd party tools to correct this and were immediately pushing the ESM solution as the proper fix.

I'm sure by now everyone knows how much hell I've raised over this issue and how I refused to accept a solution that is clearly NOT coming anytime soon. Bad enough it became the norm in Fallout 3, but 100x worse when people were seriously suggesting using FO3Edit on Skyrim files to produce the necessary ONAM records. I don't care which side of the issue you're on, you don't use tools designed for a completely different game to solve problems like this.

Fortunately Bethesda has seen fit to fix the bug and it's not an issue anymore. I think it goes without saying though that the amount of pressure needed to keep the bug visible should never have been necessary.
User avatar
Beast Attire
 
Posts: 3456
Joined: Tue Oct 09, 2007 5:33 am

Post » Thu Jun 21, 2012 8:35 am

I am sure glad that AndalayBay and her partner are working on an alternative even though you can never use it as it it will clearly violate the Bethesda Standard.

Errr... No. It will not violate the Bethesda Standard. It will simply provide the rest of the tools modders need that don't come with the CK. Tools like mod merging and cleaning. I wouldn't be surprised if the CK originally had these functions and Bethesda had to strip them out because they were part of a third party licence agreement.

-----------------------------------------------------------

SLuckyD - I'm a little concerned about your testing methods here. When you save the plugin from Snip and it's uncompressed, what is the compression flag set to? If that flag is still 1, then the CK will corrupt the data. No surprise - you're telling it the record is compressed when it's not.

I'll see if I can dig up the instructions on how to put Snip in compression mode so that it saves the plugin as compressed. That will help, but won't fix the problems entirely, as I've explained before.
User avatar
Neko Jenny
 
Posts: 3409
Joined: Thu Jun 22, 2006 4:29 am

Post » Thu Jun 21, 2012 1:36 pm

Merging with the CK can be accomplished using the undocumented version control system.

Cleaning would have been nice to have though. Or better yet, a CK that doesn't make dirty edits to start with :P
User avatar
Amiee Kent
 
Posts: 3447
Joined: Thu Jun 15, 2006 2:25 pm

Post » Thu Jun 21, 2012 3:44 pm

Merging with the CK can be accomplished using the undocumented version control system.

Only if the plugin is a master though.

Cleaning would have been nice to have though. Or better yet, a CK that doesn't make dirty edits to start with :tongue:

Heh, yeah well we can't part with tradition now!
User avatar
bimsy
 
Posts: 3541
Joined: Wed Oct 11, 2006 3:04 pm

Post » Thu Jun 21, 2012 2:10 pm

Errr... No. It will not violate the Bethesda Standard. It will simply provide the rest of the tools modders need that don't come with the CK. Tools like mod merging and cleaning. I wouldn't be surprised if the CK originally had these functions and Bethesda had to strip them out because they were part of a third party licence agreement.

Sincere apologies AndalayBay. I didn't expect you to take it like that. I do not think your work will violate the Bethesda Standard. That was foolish sarcasm aimed at Arthmoor as I think he is being inconsistent in his opinions. Earlier he said that anything developed by a third party would be risky to use. I deeply appreciate all of your hard work and I look forward to seeing the results. I for one, do definitely intend on giving it a full tryout as I am not that concerned with risky. I already think my mod is somewhat at risk everytime I open it in the CK.

The whole point of all of my wasted discussions was just to try and say that the game and the mods are always going to have some flaws and bugs in them. Nothing is ever going to be perfect. And I think I have had more problems from using the CK than I ever had from using Snip. It is just an opinion and is not intended as an endorsemant of Snip or a condemnation of the CK. They are both impressive pieces of software to me, although the CK is clearly on a whole different plane.

Arthmoor, you are right in that I did imply in my post that the botched upload was the result of a crash in the CK, but that was not my intent, just a poor arrangement to the words. There was no crash that time. I agree that a crash would make more sense and it would at least have given me some warning that something out of the ordinary had happened. I instead saw all indications of a successful upload. My suspicion is that the glitch must have occured immediately prior to the upload sometime between my save and the actual upload itself. Perhaps as part of the file access process when I started the CK back up to do the upload. I always test the file just before I do the upload and it was perfectly fine or as close to it as it is ever going to get.

Yes it would be better to have a CK that does not make dirty edits in the first place. Funny, but that was sort of the whole point of my original post in the first place. You said the CK does not corrupt data and I said that it does and gave some examples and that seemed to be against the rules somehow. At least it led to you implying I am a moron who doesn't even know that his hardware isn't running right or how to install an operating system. You have no idea how experienced I am and it is a little offensive for you to just assume someone is ignorant, stupid or lying because they have experienced a problem you know nothing about and have decided it therefore could not have occured.

It is funny how different people can have such a different experience using the same piece of software. The game is like that too. Amythyst Deceiver is having very little trouble with the CK and evidently has very few crashes. SLuckyD has so many crashes he makes his edits, saves and gets out of the CK as fast as he can. I, on the other hand am somewhere in between. Some things cause crashes more than others. I especially have troubles with large duplications for some reason, even with 16 gigs of ram and 6 gigs of vram -- assuming memory to be the critical resource there. Everytime it crashes, I cringe because I am concerned about what just got all effed up. I usually releived to find nothing, but every once it a while it does something unusual like deleting an NPC or maybe just the merchant links to the work area. Sometimes those things are hard to find and then it is difficult to know just what the cause was. Fortunately, the incidence of crashes has dwindled tremendously since the last update. When I said it will crash all by itself, I mean after an extended period of leaving it running and unattended. 8, 10 ,12, 24 hours or more - usually after an extended work session. I leave and return to find it has gone unresponsive while I was away. Working just fine when I left it and not responding when I return. That happens more often than not when the circumstances are right - always after an extended period of inactivity. And no, my computer is not overheating, either. Needless to say, I do not leave my seat without saving my work and I save often just in case. And I make a backup copy of my current workfile on a daily basis and also before I begin any session. On several occaisons, not too many, maybe 2 or 3, the CK has corrupted the file so thouroughly, I have been forced to resort to the backups.

So anyway, I am not trying to be argumentative. I was just trying to discuss the idiosyncrasies of working in the CK and modding for Skyrim and provide a few simple observations. From here on, I will probably just keep my thoughts to myself.
User avatar
Eve Booker
 
Posts: 3300
Joined: Thu Jul 20, 2006 7:53 pm

Post » Thu Jun 21, 2012 1:08 pm

Sounds to me like there are different meanings being ascribed to the word "corrupt" here, with some equating the term to "dirty edit" and others not. I'm not equating the terms. From what I can tell, "dirty edit" means a situation where the CK has changed some value, but in a way that is indistinguishable from a legitimate, user-intended edit - ie, it results in a valid plugin and loads into the game just fine, but has effects that were not intended and may cause conflicts with other mods or the vanilla game. It is often (usually?) the case that this is caused by the user of the CK doing something that they didn't realise would manifest as a change in the plugin. Eg - bring up an NPC, change something, decide you don't like the change, put it back the way it was - this will result in the NPC appearing as edited in the plugin, even though it is identical to the vanilla NPC.

I'd consider corruption to be something that introduces data into the plugin that the game (or CK) cannot interpret, or (worse) has a legitimate but completely wacky interpretation.
User avatar
Charlie Sarson
 
Posts: 3445
Joined: Thu May 17, 2007 12:38 pm

Post » Thu Jun 21, 2012 10:25 pm

I think I agree with your definitions. I don't think every dirty edit is corruption as most of them clearly are not, In fact I think the majority of dirty edits are probably going to be harmless and never adversely impact the game at all. However, I am not sure that a dirty edit cannot result in a corruption. Maybe not. I would consider any file that will not allow the game to load and run in at least a usable fashion to be a corrupt file. I would consider something that the user causes unintentionally by editing the vanilla game in the manner you descibe to be a dirty edit. However, if that dirty edit results in the file not being usable, then I would want to call the file corrupted although I could see someone saying it isn't really corrupted if it is recoverable. In the end, I don't think that really matters too much as that particular issue, whether we call it corruption or simply a dirty edit, was really caused by the operator and not the program. It would be nice if the program would clean that up when it is simply a dirty edit as you describe and I consider that to be a flaw in the program. A feature so obvious that it is unlikely to have been overlooked. I am sure there is a perfectly good reason that is simply beyond my ability to discern, but it would seem to be an easy process to allow for. If the operator decides not to edit that object, the CK should provide some simple method for removing the reference. I suspect that Bethesda did not develop that feature in the CK because they used some other application to find and remove their dirty edits. An application that they could not or would not share.

Thanks for trying to clear things up, but I think the difference of opinions is running more over the question of whether the CK has a bug or bugs that can cause file corruption without contributing operator error, which is, in my mind, rendering the file unusable in its present state even if it is recoverable. I believe I have seen that circumstance occur on a few occaisons, I don't want to speak for him, but I think what Arthmoor is saying is that he believes what I experienced was casued by some factor or factors that are not attributable to the CK.
User avatar
Samantha hulme
 
Posts: 3373
Joined: Wed Jun 21, 2006 4:22 pm

Post » Thu Jun 21, 2012 3:10 pm

A dirty edit is definitely not corruption. It's a record that appears in your mod that is identical in every way to the record in one of the mod's parent masters. Skyrim.esm or Update.esm for the most part. With just that mod loaded, they are entirely harmless. Where the problems arise are when dirty edits override other mods intentional changes. Such as a dirty landscape edit reverting a cell back to the vanilla settings, resulting in a land rip where it meets the remaining edited cells.

More often than not, these edits sneak in without anyone noticing. It's been a bothersome problem ever since Morrowind and really hasn't improved much.

Bethesda doesn't run into this problem because by definition Skyrim.esm cannot have a dirty edit. It's the parent file, and their version control system merges all edits into the parent. So changing a sword that does 12 damage into a sword that does 12 damage has no meaning in their case. That edit never exists.

Yes, I was basically getting at some other factor being involved in the corruption of your file. I know far too many people who leave the CK open for the whole day, idling away without incident. I do this myself, as the accumulated hours on Steam will attest. It's possible that when you told it to upload the file hadn't been fully saved out yet, and perhaps never got closed properly. Windows isn't supposed to allow that sort of thing to happen, but then it's Windows. A codebase that makes the CK look like a children's story.

On the occasions when the CK does crash for me, it's just an inconvenience. Nothing has ever been damaged by that, and I wouldn't expect this to be the case unless it crashed while saving. Which is why there's a backup folder full of recently saved copies of your mod.
User avatar
Deon Knight
 
Posts: 3363
Joined: Thu Sep 13, 2007 1:44 am

Post » Thu Jun 21, 2012 1:41 pm

Merging with the CK can be accomplished using the undocumented version control system.

Can you PLEASE elaborate? Thanks.

I'm so stuck because I can't merge that I'm frustrated to death, problems:

-If I make my old esp master then I can't load skyrim.esm and my old mod, it gives me a error "too many masters"
-If I work in the latest esp I have to duplicate all work of the old so it appears in the new one (tons of work and impossible to track correctly)
-If I work in the old esp the land doesn't show ingame
-Can't use tesvsnip
-Can't duplicate because of ck problems

So please, if CK allows to merge please tell me how, thanks.

EDIT: I think it's this link right? Looking at it, but what I want is esp merging.

http://www.creationkit.com/Version_control
User avatar
Jonathan Montero
 
Posts: 3487
Joined: Tue Aug 14, 2007 3:22 am

Post » Thu Jun 21, 2012 5:21 pm

This is so frustrating, now I got this error:

http://www.mediafire.com/i/?z8bojddeuohxgbf

I've transformed one esp to esm so I could latter merge into it, but it won't work either.
User avatar
Ludivine Dupuy
 
Posts: 3418
Joined: Tue Mar 27, 2007 6:51 pm

Post » Thu Jun 21, 2012 3:07 pm

Good news, I think I've done it, if I don't encounter problems I will update the wiki to help others.
User avatar
helen buchan
 
Posts: 3464
Joined: Wed Sep 13, 2006 7:17 am

Post » Thu Jun 21, 2012 4:03 pm

Well, and don't take this as valid advice since I've never actually TRIED it, you could do all your work via version control so it gets merged in properly. Then when you're done (since the navmesh bug is fixed) you could simply have Wrye Bash flip the bit to turn it back into an ESP. All of your edits should still work and the game will not care about the ONAM records the VC system adds to the file. Obviously only try this on a backup copy.
User avatar
Makenna Nomad
 
Posts: 3391
Joined: Tue Aug 29, 2006 10:05 pm

Post » Thu Jun 21, 2012 4:20 pm

I agree with Arthmoor regarding the dirty edit vs corrupt... and exactly how he explained it. That having been said, THAT is the primary reason why I do strategic ops when using the CK... quick in, quick out - to avoid dirty edits that require later cleaning. Since I started doing that, I NEVER have dirty edits (in my 'real' mods).

But if I KNOW I'm going to fool around with an entirely new 'thing' (eg: editing a Vanilla cell to add all kinds of new stuff) I use a backup file... which inevitably has those dirty edits because of the early 'planning stage' (btw: deleted refs stay deleted, not like Obliv's CS). Once I've ironed out what I want, I exit CK and start over from scratch - knowing exactly what I want to do and what to avoid. This may be overkill - but it works for me... and has actually saved me plenty of time already.

So hopefully I just adequately refuted the misconception (a la ch0k3h0ld) that my CK crashes with every breath I take... I've just heard the stories and try to avoid it, and what I do is successful for that purpose. My experience with the CK has been quite pleasant actually - and the 'random' crashes I haven't actually seen before... only when compiling scripts - which is GIVEN if enough compiles are performed. This has nothing to do with the amount of memory or how much of it is used... it's a 'garbage-collection' issue; meaning certain things are stored in memory by the CK which SHOULD be flushed periodically but aren't; eventually leading to a CTD.

Regarding that sentiment, I think the statement that Windows is to blame for possibly altering a file in use by the CK is erroneous: the CK should be flagging that file as read-only during its processes, should it not? (that would prevent other programs from altering it; though I could be mistaken, as I'm no programmer)

Now about my 'testing methods' (a la AndalayBay)... the only reason I do what I do is because I don't know how to decompress records (either in my mind, or to write an app which does). To date, the only app I trust to decompress the records into a 'viewable' format is TESvSnip, and now I've started to use Gecko to double-check this (presumably it uses a different algorithm to decompress for viewing). Since Gecko is in an early enough stage, I feel can't completely abandon Snip yet - but that day will almost certainly come. I try to be as impartial and open-midned as possible in EVERYTHING I do (in life), especially using such a questionable methodology for the issue at hand - any ideas for an alternative?

Some of you may be 'happy' (deep inside.. heheh) to know that I have actually found an instance of data that the Snip-CK dynamic causes outright data 'corruption' (in that the data which SHOULD be present, has been changed or deleted). I need to do more testing to see if there is a pattern though.. right now it's seemingly 'random' (though isolated to persistent REFRs). Saving something in Snip (decompress), then saving in the CK (recompress - but NO intentional changes made in the CK whatsoever, just a load-n-save) has caused this: two of four persistent refs (patrolMarkers in this case) were Zrotated +6.283185 EACH, for no apparent reason. Obviously it's correctable if User is aware of it.

As you can imagine - this is an actual, tangible problem; though in the case of my test-mod it means nothing.. rotating the patrolMark the way I set it up had no apparent effect in-game. All other corruption I've seen and documented (yet to be posted) has been in the form of the 'flags' data... which are ambiguously harmless. BUT - again, this was actually caused by the CK not Snip - regardless if whatever Snip did triggered the CK to f it up (the non-CK-saved plugin works fine and still has the correct data).

Again, I still plan to do up another thread (with a more-searchable-for name) - which outlines, educates, and discusses corruption in its various forms. I made several video clips to show what I do and what happens... one of which I've finished and uploaded to youTube. This doesn't really address actual/meaningful corruption, quite the opposite - it proves that a Snipped plugin CAN still function 'perfectly' in-game (disproving the 'Snip is always evil' theory)... but you MUST keep in mind that this is a very SIMPLE test-mod. There are PLENTY of circumstances that I haven't added to the mix yet (which IMO would be poor technique this early in the troubleshooting/anolysis process).

http://youtu.be/IHvuaCTGpUM

I plan to Dropbox the test-ESPs I used for these videos so you can look at them and see for yourself if you want. It'll all be in that as-yet-uncreated thread.
User avatar
Karen anwyn Green
 
Posts: 3448
Joined: Thu Jun 15, 2006 4:26 pm

Post » Thu Jun 21, 2012 9:09 pm

I was gunna "edit" that last post, but it was long enough as-is, soooo.... About the compress/decompress stuff:

I still haven't been able to save something in Snip as compressed. I tried flagging the records as compressed and not... no dice. I tried both of those using various 'options' in Snip. I think I may have shown that in the linked video - but will certainly be illustrated in future videos.

I still haven't seen a single instance of Snip actually compressing data - or corrupting data which has been 'mistakenly' flagged. I'd like to know how those of you who say this is the case accomplish this. DON'T mistake my statement (as one or more are prone to interpolate) as meaning 'Snip doesn't do whatever'... simply that I haven't seen it and cannot FORCE it to do whatever.

I also have yet to see any data or procedure which reproduces or proves the supposedly corruption caused by the decompression/yada yada. Being primarily left-brained.. skepticism is my default position until I feel something is true. I NEVER deny something unless I know it too be true. Athough I am human and could have made a type-o or mistake.. so jump on that lightly if you're going to.. and LINK to your accusations so /we know where my divergence was.

User avatar
Kara Payne
 
Posts: 3415
Joined: Thu Oct 26, 2006 12:47 am

Post » Thu Jun 21, 2012 11:23 pm

Well, and don't take this as valid advice since I've never actually TRIED it, you could do all your work via version control so it gets merged in properly. Then when you're done (since the navmesh bug is fixed) you could simply have Wrye Bash flip the bit to turn it back into an ESP. All of your edits should still work and the game will not care about the ONAM records the VC system adds to the file. Obviously only try this on a backup copy.

That's what I did, and it works, no problems so far. I have to get some time to write a tutorial for others.
User avatar
Andrea Pratt
 
Posts: 3396
Joined: Mon Jul 31, 2006 4:49 am

Post » Thu Jun 21, 2012 5:19 pm

I agree with Arthmoor regarding the dirty edit vs corrupt... and exactly how he explained it. That having been said, THAT is the primary reason why I do strategic ops when using the CK... quick in, quick out - to avoid dirty edits that require later cleaning. Since I started doing that, I NEVER have dirty edits (in my 'real' mods).
Probably overkill :P

I do my "strategic ops" by opening a second session of the CK with just Skyrim.esm and Update.esm opened. Then I'm free to poke and prod that all I want while at the same time having the primary session I'm doing the actual work in untouched until I'm ready to touch it.

Some dirty edits you simply can't avoid. Try landscaping without the CK spreading filth across several surrounding cells you KNOW you never touched. A problem that's maddening with Oblivion. Paint a bit of ground well inside the cell borders of a single cell, find out the CK/CS has plastered 6 other cells nearby as edited even though they aren't. All the strategic ops in the world can't help this, and it's pretty frustrating. Scrubbing them out with the Details screen doesn't always take either.

And let's not even get into the mess it makes with editing navmeshes and dirty edits. UGH. Though not corrupt, I don't much care for having dozens of cells I never touched in the file to cause conflicts with other peoples' mods simply because one has to finalize the cells they do edit.

Regarding that sentiment, I think the statement that Windows is to blame for possibly altering a file in use by the CK is erroneous: the CK should be flagging that file as read-only during its processes, should it not? (that would prevent other programs from altering it; though I could be mistaken, as I'm no programmer)
I was always told that it's the API's job to ensure the file is locked so that nothing bad happens to it and that your application has no control over this, but I'm not a Windows API guy so I have no idea what really goes on down there.

Now about my 'testing methods' (a la AndalayBay)... the only reason I do what I do is because I don't know how to decompress records (either in my mind, or to write an app which does). To date, the only app I trust to decompress the records into a 'viewable' format is TESvSnip, and now I've started to use Gecko to double-check this (presumably it uses a different algorithm to decompress for viewing). Since Gecko is in an early enough stage, I feel can't completely abandon Snip yet - but that day will almost certainly come. I try to be as impartial and open-midned as possible in EVERYTHING I do (in life), especially using such a questionable methodology for the issue at hand - any ideas for an alternative?

two of four persistent refs (patrolMarkers in this case) were Zrotated +6.283185 EACH, for no apparent reason. Obviously it's correctable if User is aware of it.
I wonder if this contributed to the error I keep getting with the released version of OCS and that road marker outside Whiterun. That error wasn't present until the firs time I exposed it to Snip but I hadn't connected the events with an actual unwanted change.

One would figure than in general, rotating most markers isn't a big deal. 6 degrees isn't much, but hell, if this is a cumulative rotation effect, I've probably got some crazy [censored] going on by now. Which is why I'm rebuilding the whole mod.

I still haven't seen a single instance of Snip actually compressing data - or corrupting data which has been 'mistakenly' flagged. I'd like to know how those of you who say this is the case accomplish this. DON'T mistake my statement (as one or more are prone to interpolate) as meaning 'Snip doesn't do whatever'... simply that I haven't seen it and cannot FORCE it to do whatever.
That's because it isn't actually compressing it. I thought AndalayBay made that clear. The problem stems from then loading it into the CK when you've TOLD it the data is compressed and the CK goes "Ok, I trust you" and then... BLAM. The deception Snip played on you costs you work.

Your testing method is flawed because you're introducing the very application we've validated has problems as part of your procedure. The moment you did this, all bets were off on the validity of anything following that step.
User avatar
Kristina Campbell
 
Posts: 3512
Joined: Sun Oct 15, 2006 7:08 am

Post » Thu Jun 21, 2012 6:10 pm

regarding exterior tamriel navmeshes, in my experience i have avoided all unnecessary edits to surrounding cells (and the immediate cell itself) under the following conditions only:

if all you are doing is dropping a door into tamriel to point back to your mod cell, you don't actually have to finalize that cell. the navmesh does not need to turn green in order for that teleporter to work. the NVMI alone is enough to guide NPCs to and from the door teleporter. i have tested this thoroughly over the past 2 and a half months and monitored user input with no incident.

now, this ONLY works if there are no other vanilla door teleporters on the same navmeshed cell, but you can add several of your own all without finalizing.

this will allow you to create a 100% non-vanilla mod, as the only edit in tamriel is in the (0,0) persistent child GRUP. for those using esm, this will take with no incident since adding REFRs to a (0,0) cell requires no ONAM. also the NVMI's stack, they dont overwrite so no ONAM needed for that either.
User avatar
Enie van Bied
 
Posts: 3350
Joined: Sun Apr 22, 2007 11:47 pm

Post » Thu Jun 21, 2012 3:25 pm

Merely as a curiosity, and not for serious consideration: I have only been experimenting with making a couple of very small noob-level mods for Skyrim (Like dressing my Housecarls in weightless Barkeeper and Wench outfits when they are off duty, and such.) Yet even I noticed the NPC-issues with snip, but I also realized, that for a truly tiny thing like that, it was quite easy to simply use a hex editor to remove a completely useless World Group from the mod (as such a group is completely self-contained), and while I was at it, I also successfully removed Update.esm as a master (had to adjust a header for record size), just for kicks. :smile:
User avatar
Heather M
 
Posts: 3487
Joined: Mon Aug 27, 2007 5:40 am

Post » Thu Jun 21, 2012 12:01 pm

*grumble grumble*

Here is how you turn compression on in TESVSnip:

This quote is from Maegfaer:

You can turn on compression in TESVsnip by manually editing TESVSnip.exe.config and changing the following:


False


To:


True


The bloody button in TESVSnip to turn it on simply doesn't work, hence you need to do this manually. After that you can check which records need to be compressed in TESVSnip under 'Options' -> 'Compression Settings...' and then the settings will stick.

Not saying that this solves all issues, but I do regularly edit our stripped down Skyrim.esm (removed all worldspaces and cells) and I have compression for every form type turned on. Works fine. And they actually do get compressed this way, since the file size gets like 4 times smaller than with compression turned off. I wouldn't know about this second layer of compression you are talking about, I'm not an expert.




As he says "The bloody button in TESVSnip to turn it on simply doesn't work, hence you need to do this manually."

THIS WILL NOT FIX THE PROBLEMS IN TESVSNIP. It helps. Now when you load the data in the CK and have the compression flag set to true, it truly is. That will cut down on the corruptions. HOWEVER TESVSNIP WILL STILL TRUNCATE DATA.

I'm not going to argue this anymore. I've repeated myself what feels like fourteen times now. If you guys want to keep playing about, feel free. :tongue:
User avatar
Jeff Tingler
 
Posts: 3609
Joined: Sat Oct 13, 2007 7:55 pm

Post » Thu Jun 21, 2012 9:22 pm

I'm not going to argue this anymore. I've repeated myself what feels like fourteen times now. If you guys want to keep playing about, feel free. :tongue:
Gooood.... GOOOOOOOD! Let the grump FLOW through you and make you stronger!
User avatar
Jah Allen
 
Posts: 3444
Joined: Wed Jan 24, 2007 2:09 am

Post » Thu Jun 21, 2012 8:48 am

I'm gonna make a mod in TESVSnip that adds weekly gay pride parades in Riften
User avatar
Shelby Huffman
 
Posts: 3454
Joined: Wed Aug 08, 2007 11:06 am

Post » Thu Jun 21, 2012 12:31 pm

http://skyrim.nexusmods.com/mods/18761.

Wonder if he managed to fix this problem? Tool isn't really worth using until it is fixed.
User avatar
Heather beauchamp
 
Posts: 3456
Joined: Mon Aug 13, 2007 6:05 pm

PreviousNext

Return to V - Skyrim