That philosophy works when you're editing something where giving it your own script is an option. This is not the case if you are making changes to vanilla quests, dialogue entries, or scenes. The fragments in each of the boxes you're presented with have to follow a certain setup that you have no control over.

+1 For this to be a sticky, excellent description and research Lojack - And well, I know it means more work for the mod manager teams but Wrye Bash having the ability to delve into BSA's would be very nice.

Isn't Ethatron on the team yet - If he is willing of course, bash is quite a commitment.

Still haven't figured out the issue with your script mod yet - like Arthmoor said though, replacing the script should work fine in general (and does, with Loose Files obviously).

Question: what utility did you use to create the BSA? I know some tools (FOMM for example) make BSAs the game doesn't like.


Ethatron's got enough to do as it is, with his multitude of utilities he maintains, but he is going to make a few changes to DDSopt so Wrye Bash can interact with it better and deal with BSAs that way (basically in a similar manner to how we use 7zip to handle normal archives).

Another master of the code (shadeMe) is also helping out, he's taking a shot at lifting the 255 character limit on the INI entries. If we get that working right, then the best method of BSA Load Ordering will be to register all BSAs, and control their order by the order their listed.

Oops, sorry, thanks for the correction .

I've been doing my experiments with the ini file too:
One result: I set the SResourceArchiveList to this: "!,",#,$,%,&,',(,),*,+,-". Then I hooked the CreateFile and GetFileAttributes functions and in those mapped the symbols back to the actual bsa filenames (i.e. if Skyrim tries to open # it really gets Skyrim - Animations.bsa). Almost all printable symbol should work, so with this trick can represent over 200 bsa files by 2 character each which we can easily fit into SResourceArchiveList and SResourceArchiveList2.
Of course this is very hacky, requires a skse plugin (except with Mod Organizer ) and the "obfuscated" list needs to be generated automatically but still...

EDIT: Second information was wrong.

And they told me Archive Invalidation doesn't exist for Skyrim . . .

Quick question: How should BSAs be ideally loaded? I'll see if I can implement that along with the INI string limit fix.

It doesn't.

I would say by date if they're not associated with an esp, and by esp load order if it is.

a ) record all active ESM/ESP (Oblivion though doesn't support BSA<->ESM connectivity, don't know about Skyrim)
b ) for all BSAs find the ESM/ESP which is a prefix-name ("Skyrim - Textures.bsa" -> Skyrim.esm")
c ) order BSAs without companion ESM/ESP by .ini order in list.a
d ) order BSAs with companion ESM/ESP by ESP/ESM-order (can be date, can be pluginlist priority; in case of == continue using the BSA-date) in list.b
e ) append list.b to list.a

Then first-found rules works.

I also tried to hook the CreateFile function, however while I saw Skyrim accessing its standard .bsa files, I didn't find any CreateFile versus plugin .bsa files.

If your hook controls file loading, you could hypothetically speaking make all bsa loads fail as "file not found" (or, better, all non-registered bsa loads), and represent their content to the game as loose file loads.

The downside of this approach is that you need to build a complete map of files represented inside those .bsas, so you can handle conflicts properly. I am also not sure whether you can implement this as a file object to the game or whether you have to have unpacked the files from those .bsas.

And, if you have that map, you can refine this such that only conflicting bsas need surrogate handling.

+1 for this to be stickied.

Invalidation, Validation, Alteration, Tweakeration, whatever you wanna call it -- same difference. And yes, it does. That's the purpose of editing the Archive section in the ini files. And there has been many reports of meshes/textures wacking out like it did for Oblivion due to it.

Been said mostly already. Ideally, it will work like this:

• All BSAs registered in the INI, in the same order they are registered
  
• All BSAs non-registered in the INI, but with an associated ESP causing them to load - these in chronological order (like ESPs, but not tied to the Load Order of the ESPs)
  
• Loose Files

This gives the most control over which resources are used in game, while still allowing use of Plugin BSAs and Loose Files easily (and not in a confusing manner). Since the Skyrim BSAs are loaded via the INI, and that order is controlled by the order they're listed, we're good there as well.

But, since you probably can't make the Plugin BSAs load before Loose Files, I'm guessing our workaround will be just to load ALL BSAs through Skyrim.ini, and so you're left with:

• All Registered BSAs, in the same order they're listed in the INI
  
• Loose Files
  
• All Plugin BSAs, again by chronological order (but we'll use the INI to load all BSAs instead).

Yes, I agree with this too as I see no reason why they should not override them.

I guess a temporary solution would be to make any (texture) replacers into a *.bsa file with the Archiver.exe included in the game's main folder and an *.esp with the same name and make sure the names are in alpha-numeric order in the order you want to load them. I think this is better than unpacking everything as packing them gives the option to compress them thus using less space.

And if you got too much time and are a total nerd like me, unpack for example all texture stuff, drop them all in one folder in the order of which files need to overwrite last, separate them if it's all larger than 3 GiB making sure there's a maximum of 3 GiB per *.bsa you're going to make. Then use Archiver.exe to make the *.bsa's using compression so that 3 GiB becomes 2 GiB, put in Data folder and add names to Skyrim.ini. Just be careful not to make a recheck in steam as it'll consider the file incorrect and re-downloads it, overwriting the one you made.

Believe what you want, but AI is a different animal altogether. Right now for Skyrim, it is merely understanding the load order precendence. AI was an outright bug.


http://devnull.sweetdanger.net/archiveinvalidation.html#AIExplained

Oh whoops. Yeah, that's true; didn't realize it was something that never got added to the Wiki. Joel mentioned it as a thing for us to test during the Beta; I guess it wasn't on the Wiki since it hadn't been tested before the beta or something?

Anyway, I'll think about where to put that, though that could be tricky.

You are completely incorrect on a technical level. The issues described herein are completely different from Archive Invalidation issues, and conflating the two will only lead to confusion. Continuing to claim they are the same is only going to confuse users, so please stop.

. I want this thread to be a place where useful information is maintained.

As for the difference:

ArchiveInvalidation was Oblivion,etc's method of telling the game to load a Loose File instead of using one in a Vanilla BSA. It was bugged, and had many problems - it wasn't reliable, and really didn't work at all for modding purposes.

The best solution to the problem was ArchiveInvalidationInvalidated - registering a dummy BSA in the game's INI, so that when the game looked for a resource, it wouldn't find it in that BSA, which then FORCED it to load from Loose Files.

It was only a problem in strange circumstances (Loose Files would USUALLY load, but sometimes wouldn't). ArchiveInvalidationInvalidated solved all that, and has been in use for Oblivion for many years now.



The issue we have with Skyrim ISN'T this. The issue(s) we have, as has been stated already:
1 - Loose Files don't win, Plugin BSAs do
2 - Plugin BSAs don't have a reliable Load Order - they load alpha-numerically, instead of something we can control (without renaming BSAs+ESPs)

You know I noticed on the Steam Workshop (before it went down today) that many new mods were adding zzz1 or zx in front of their ESP files. This isn't a good idea of course but I have seen it begun to happen. It's so vital they correct this. It is only the first week out but if you're deep into modding a week might as well be a month.

Yeah, that's only a bandaid over a dismembered arm. If no official fix comes out, you'll just end up with wars seeing how many z's you can prepend to your plugin name, we'll end up with 'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzBetterPotions.esp' and such, lol.

I don't think I'm alone when I consider that wanting loose files to load over BSAs is how we think of ArchiveInvalidationInvalidated working and hence the shortening of that concept to that compound word. I think the confusion comes when this is further shortened to 'Archive Invalidation is not working' which to me means the same thing. Yes I admit it is lazy language, but I don't think people are posting that they think about it in such exact terms - especially with the history of BSA redirection and a lack of names for the issue.

I did some minor clean-up folks. Back to your regularly scheduled program...

Wow. That was a good read.

A couple of questions you might be able to answer:

1. Didn't the devs mention something before release about being able to have mod-specific ini changes? Might be possible to register any plugin BSAs by doing that. Pretty sure I didn't imagine it, but I didn't look into it yet...

2. I have an issue when running my female mannequins mod with CBBE. For some reason, when CBBE is installed, the files inside my BSA aren't recognised (the game/CK can't find them). CBBE doesn't use any BSAs and nothing has the same filepath/filenames as my mod. Normal bodies and UNP both work fine. It makes no sense and I can't work it out. Got any ideas?

http://www.creationkit.com/INI_files#Mod_Defined can override Skyrim.ini, but I think they would only be able to overwrite the entire line.