Initially this was going to be a response to http://www.gamesas.com/index.php?showtopic=1343725 about BSA Load Order, but this actually is very important - it has an effect on everyone, not just hard core modders.

Disclaimer: The game doesn't actually work exactly like this. This is how it logically works. As in, if you had to describe the decision process the game goes through, this is it. Realistically, working this way would be a waste of resources and processing time (why load up every file, when you already know a different one is going to "win"?). It's just much easier to understand if you think of it like this. A good description of how it would actually work is http://www.gamesas.com/topic/1345724-important-bsas-and-you/page__view__findpost__p__20282031.

Ok, I've gone through and anolyzed Skyrim's process for loading resources (all the meshes, textures, etc in the Data folder). For the most part, it works fine, just like it did in Oblivion. There are some improvements (two BSAs with the same resource will no longer cause a crash), but there's also a major drawback now.

Contents:

• Resource Load Order
  
  • Registered BSAs
    
  • Loose Files
    
  • Plugin BSAs
• What This Means For You
  
  • Mod Makers
    
  • Mod Users
    
  • Mod Managers
    
  • Bethesda
    
  • If All Else Fails
• Other Questions
  • Replacing Scripts
    
  • Directory Thrasing
    
  • Are You Sure?
    
  • What are all these terms you use?
• A Quick Workaround
  
• Tools
  
  • DDSopt
    
  • SKSE

1. Resource Load Order
Skyrim loads its resources from the Data folder. Makes sense. These resources can also be packed up into a nice neat little archive, called a BSA. These are gamesas's custom archive files, and internally they mirror the layout of the Data folder. They're nice, because they keep the Data folder uncluttered, and it makes it really easy to remove every file from one mod.

Ok, but what happens when the same resource exists in more than one place? For example http://img401.imageshack.us/img401/9967/dummyn.jpg, which is the texture for the map you see all the time in castles, with the little flags and stuff sticking out of them. This file exists in Skyrim - Textures.bsa. That's the original file. If you download a texture replacer, you'll get it as a loose file probably, and if you download HD Skyrimmap from Steam Workshop, you'll get that file in a BSA. So which file actually gets used? Well, which ever one is loaded last. The game loads resources in three stages, with the last files loaded being what you see in game:

Update: Further investigation reveals that only Registered BSAs and Loose Files are loaded when the game first loads. That means anything in a Plugin BSA won't take effect yet - but you'll only notice that if you're trying to replace a Main Menu asset, like http://skyrim.nexusmods.com/downloads/file.php?id=7801. Once you start a game, either by loading a saved game or starting a new one, then the Plugin BSAs are loaded. So in the example of Brumbek's Main Menu Logo replacer, if it's loaded via a Plugin BSA (like it is when installed via Steam Workshop), you won't see the spinning logo until you load a game, then exit back to the Main Menu.

Registered BSAs:
First, the game reads your Skyrim.ini file. There's a section that tells the game which BSAs to load. I looks like this for most people:

[Archive]sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsasResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - Voicesixtra.bsa

The game will load the BSAs in order as listed, BSAs in sResourceArchiveList first, and sResourceArchiveList2 second. So if you manually register a texture replacer BSA (like the HD Texture packs), but put it at the beginning of the list - they won't load! Because the Skrim - Textures.bsa will override them of course. If you put it after, or at the end of the list on the second entry, then they will load.

Easy enough.
Note: In Oblivion, there was a 255 max character limit for this entry. I didn't test it this time around, but I think it's a safe bet to say that limit still exists, otherwise there wouldn't be two entries for BSAs. This character limit is why registering BSAs is a bad practice - get enough in there, and you won't be able to put any more in it.
Update: Tannin42 has confirmed that this is the limit for each entry. That means we've got a maximum number of BSAs that can be registered between these two entries.
Note 2: These are the only two entries that work, also. I tried adding a sResourceArchiveList3, etc, but they don't load BSAs, so we only have two 255 max character entries to work with.

Loose Files:
This one's simple. If the file exists in the Data folder, it loads after any of the registered BSAs, so it'll always win. This is how most texture replacers work, and this is what mod managers like Wrye Bash or NMM are made for. Managing the Loose Files so you don't have to track them yourself. It's easy to keep track of which file came from which installer when you only have 3 mods. But when you get up to 150 mods, each with their own resources, a lot of them overlapping, a mod manager just takes the headache out of it.

Note: In Oblivion, Loose Files would only win if their modification date was "newer" than the BSA that also contained the file. This is not true anymore, Loose Files always override a registered BSA, but never override a BSA loaded via Plugin.

Plugin ESPs
When the game loads an active ESP, it also looks for a BSA of the same name (for the most part, some additional variation allows one plugin to load multiple BSAs, but that's not important for this discussion). So dummy.esp will make the game load dummy.bsa. This is true even if the ESP does absolutely nothing (for example, the official HD Texture packs do this, in Oblivion, the Shivering Isles assets were loaded like this). Cool! We found an easy way to make our replacement texture only load when we want! Just deactivate the plugin and they don't load, activate it and they do!

Yes, but...

This first point is an old one. Due to the way the game indexes plugins and stores the data in your save game, you can only have 256 active mod files. That's because objects are referenced by what's called a FormID, which is a 32-bit (4-byte) unique number, usually written as a 8-digit hexadecimal number. The first two hex digits are used to identify which mod the data came from. Two hex digits can represent a maximum of 256 unique numbers (0-255). Now, Skyrim.esm and Update.esm will always fill up two of those. Another one that's not obvious is your saved game itself - this one is always assigned the ID of hexadecimal FF, or 255. That leaves room for 253 other plugins. And here we get to our point: dummy plugins that don't do anything at all, but exist solely to make the game load a BSA? Yep, those take up one of our plugin slots. This might not seem like a big deal, but once you start trying to play with 300+ mods, you'll be faced with the hard decision of deactivating some mods. Luckily, dummy plugins you can just unpack the BSA and use the resources as Loose Files, then you don't need the plugin anymore. Other plugins can be merged via Wrye Bash into the Bashed Patch, further cutting down your mod count. And hopefully an equivalent of TES4Gecko will come out eventually, which will be able to merge mods into a single plugin. This last one however has an impact on your saved games, so it's for the more advanced user only.


And now for the second point. This here is something new, and it's why I've written this post for you guys.

What happens when two Plugin-loaded BSAs have the same file in it? For example, I did my testing with HD Skryimmap above, plus another BSA I created with the same texture, but I scribbled graffiti on it so I could tell which one was loading: Vanilla (Skryim - Textures.bsa), HD Skyrimmap (hd_skyrimmap.bsa), or mine (dummy.bsa). Well, the results are Bad News™.

When the game is loading up the BSAs associated with plugins, it doesn't do it in a way that would make sense for modding. If it loaded them similar to plugins, where modification time determines Load Order, that would make sense. That would mean that a plugin with a BSA that was modified on 2/1/2012 would load after a BSA modified on 1/1/2012, meaning that the "newer" BSA would win, and you'd see its resources in game. Nope! Sorry, doesn't work that way. Ok, how about tying the BSAs in with Load Order? That would make sense, even if it takes away control from the user on which resources they want to see (it couples Load Order with Install Order). In this scenario, if hd_skyrimmap.esp loaded after dummy.esp, the hd_skyrimmap.bsa resources would load later, and win. Nope! Also, not the case.

Well, how does it work then? Kinda silly, and not helpful at all for modding. BSAs (only the ones associated with active plugins), are loaded in alpha-numeric, case-insensitive order! What!? Why!? Well, if you know about filesystems, it makes sense: when a program asks the filesystem for a list of files in a specific directory, the usual response is a list of files in alpha-numeric order. On Windows (read: the filesystems that Windows uses), filenames are case-insensitive as well. So Skyrim will ALWAYS load hd_skyrimmap.bsa AFTER dummy.bsa, meaning I can never get my dummy ESP's BSA to override and win. Well, I could rename it to zDummy.bsa and zDummy.esp (in fact, I did, for testing). That makes my dummy textures load, and they're used in game.

Ok, but that's NOT a good way to change load order, because renaming plugins causes all sorts of other problems. Remember how we talked about FormIDs before? Well that unique number assigned to each plugin in your saved game is assigned to the plugin name. So if you change the name, the game will throw out any data associated with it, since it's associated with the plugin called "dummy.esp", NOT with the plugin called "zDummy.esp". This is how you end up losing all of your items you stored in a chest in a player created home when you update the plugin. Because the author changed the name from myhome_v1.esp to myhome_v2.esp. Also, other plugins that either patch yours, or build on it, reference your plugin by filename as well! So now by changing the name, you've ensured that any plugin that needs yours as a master won't work anymore.

Another small strangeness associated with Plugin BSAs - they aren't loaded until you load either a saved game, or start a new game. That means if you're still at the Main Menu for the first time, any replacers loaded this way haven't taken effect yet! Woah! Not a big deal, unless you're trying to replace the Main Menu Logo...http://skyrim.nexusmods.com/downloads/file.php?id=7801. That mod works fine when installed as Loose Files, or if the BSA is registered in your Skyrim.ini, but it only works as a Plugin BSA after loading a saved game.
Note: Makes sense really from a design standpoint - it means the game doesn't have to search through all of your BSAs when it first starts up. That cuts down on the initial startup time.


2. What This Means For You

Mod Users:
Did you understand all of that? Great! That means I'm not a horrible writer after all. Didn't understand it? Don't care? Didn't read it? That's fine, my advice is the same no matter what:

Update: shadeMe has released a SKSE plugin that provides an alternate method of controlling BSA Load Order. Check out Section 4 "A Quick Workaround" for more information.

If you don't want to go the SKSE route, unpack all BSAs associated with mods. Don't register them in Skyrim.ini, I've already talked about the 255 character limit for those entries, and you'll run out of space quick if you have a lot of mods. With each mod, unpack its BSA, then zip those Loose Files up along with any documentation and plugins into an archive. Then you can install and manage that archive with your favorite mod manager. If you want, you can also examine the ESP. If it's an empty ESP just there to load the BSA, toss it, you don't need it anymore. If you don't know how to check for empty ESPs using TESSnip or a similar utility, another way to do so is right click on the mod in Wrye Bash and select 'Mark Mergeable...'. If the plugin is empty, Wrye Bash will say so in the results of that scan.

And if you need a good tool for unpacking BSAs, check out http://obge.paradice-insight.us/wiki/DDSopt.

Mod Makers:
Don't package replacement files into BSAs. I know, it's hard to do that with the CK, since it gives you no control over which files it packages. Not to mention the fact that you can only upload files to Steam Workshop as an ESP/BSA pair. Don't take this as Steam hating, but right now Steam Workshop is too messed up. This issue with BSAs is one problem. Another issue (that will hopefully get fixed), is that Steam Workshop will redownload a mod if the modification date changes. That means if you change the Load Order of a mod, Steam Workshop downloads it again! Waste of bandwidth, plus it messes up your Load Order all over again.

Don't package any files into a BSA if you can avoid it. It'll be easier in the long run. This is, of course, modder's preference. But, even NEW resources loading as a BSA - they can't be overriden by Loose Files anymore. This means if you discover a problem with a mesh in your mod, you can't just release the replacement mesh and have it override the BSA mesh. You have to repackage the whole BSA again. Not a big deal for smaller mods, but when your BSA is 500 MB, that's a pain to upload, and a pain to download for just one mesh fix.

Don't change plugin names unless you have good reason to. There are times when changing plugin names makes sense. It's NOT a good idea to change the name just because the version changed, or you added stuff, or you want the associated BSA to load last.

If Steam Workshop is a must for you, encourage your users to unpack the BSA, and leave the plugin deactivated. That only works for dummy ESPs though. If your plugin actually does something other than load the BSA, now they'd need to unpack and delete the BSA. But...now Steam Workshop will want to download it again, so they'll have to unsubscribe, which means if you update the mod, they won't get updates. Ugh!


Mod Managers:
Come up with a good workaround to all this mess. My plan for Wrye Bash is multi-staged:

• When installing mods via BAIN, (optionally) unpack any BSA in the package, and install just Loose Files instead. This will require more processing on BAIN's part, since BAIN needs detailed knowledge on the contents of the BSAs now. That means when scanning a package, it'll have to extract any BSAs, then parse them for a file listing, plus CRC information.
  
• If a BSA that isn't registered in Skyrim.ini is found, offer to unpack it into a BAIN package, then delete the BSA. Additionally, after doing so, if the ESP that loads the plugin is an empty one (has no records), offer to delete the plugin as well.
  
• I'll be working with ianpatt on this one, and it'll probably take quite some time. Interface with Steam Workshop via its API to access mods on the Steam Workshop. This way we can bypass the Steam interface altogether, download mods to a temp directory, unpack their BSAs, and make a good BAIN package out of it. Updates will still work this way, since Wrye Bash will be able to check with Steam and download new updates, then apply them in the same way to the custom BAIN pacakge.

Hopefully Dark0ne and the NMM mantainers can come up with a solution as well, so their users can play a modded Skyrim the way it's meant to be played as well.

Bethesda:
Please, please fix the issue with Plugin BSAs Load Order! Probably the best method for modders is make them load chronologically by file modification time, basically like how Load Order is determined for plugins. I'd plead to NOT make them load based on their plugin's Load Order, because then you're tying Install Order (what resources you see) in with Load Order (what changes to the game are made). We enjoy having our Install Order and Load Order decoupled, please keep it that way!

Also, it's still important to be able to override these Plugin BSAs with Loose Files if we want to! That's more complex, sure, but a good idea would be that if a Loose File is "newer" than a Plugin BSA, use the Loose File instead of the BSA's file. Edit: Actually, it'd be easiest to just make Loose Files always override. Then the only thing to sort out is making sure the Plugin BSAs load up in a logical order.

And another issue that's come up: replacement scripts in Plugin BSAs don't actually replace the original script. If it's loaded as a Loose File or from a Registered BSA, it'll replace the script, but not from a Plugin BSA. Obviously with Steam Workshop using BSAs, that's something that we'd love to have fixed as well.


If All Else Fails:
Well, I suppose we could write a SKSE or SD plugin that hacks the BSA loading code, and implements a saner version ourselves. Still, it'd be WAY easier to just have gamesas fix this.

3. Other Questions

Replacing Scripts
There's been a couple reports that trying to replace a Vanilla Script file with one in a Plugin BSA isn't working. New scripts seem to work fine, it's just if you're trying to replace one of the original scripts. A good example of why you might want to do this: Vamprism overhauls. You'd want to replace the script(s) that control Vamprism, and if you release this mod as a BSA, those replacement scripts will be in your BSA. So far the results of testing indicate that the replacement script will be used if it's either a Loose File or in a Registered BSA, but not a Plugin BSA.

Some more investigation still needs to be done:

• Verify that this only affects Vanilla scripts, and if so, is it all of them, or just some of them?
  
• Is there some way to get the replacement scripts to load, even when used in BSAs?

Directory Thrashing
What is it? http://wiki.tesnexus.com/index.php/Oblivion.exe_file_and_directory_thrashing. Basically, simply having too many ESP, ESM, and BSA files (even inactive) in the data directory would cause Oblivion to go crazy, bringing your game to a crawl as disk I/O went through the roof. It's why Wrye Bash has the Auto-Ghost feature.

Does it affect Skyrim? I don't know, I didn't really feel like making 400+ plugins to test this out. Not to mention that there was never a reliable number anyone could pin down that triggered it. But, if it does affect Skryim, there's another good reason to not use BSAs at all. Loose Files didn't trigger it for Oblivion, which is why a lot of modders didn't use them (BSAs).

Are You Sure?
Yes. I did all of my testing with a fresh install of Skyrim, updated to the current 1.4 patch. I then installed http://steamcommunity.com/sharedfiles/filedetails/?id=7540 from Steam Workshop, which modifies the map texture file I mentioned. Then I unpacked that BSA, edited the textures in Paint.NET (spray'd graffiti on them), and packed those into another BSA, called dummy.bsa, and made a dummy (blank) ESP to load it, dummy.esp. I then went on to test every conceivable (to me) combination - redating BSAs, rearranging plugin Load Order, registering the BSAs in various points in the lists in Skyrim.ini, even renaming files to make them sort differently alphabetically at one point. Another point brought up - I completely exited the game each time between changes. That makes sure the texture is in fact loaded from the files, instead of already existing in the RAM (in which case, the game wouldn't load it, it's already loaded). I wrote it all down as I went to make sure I didn't leave anything out. I'm very sure this is what's going on (not that it's much different from previous games).

Additionally, after Brumbek's issue with the Main Menu Logo replacer came up, I did some further testing with http://technet.microsoft.com/en-us/sysinternals/bb896645 running to log what file accessing Skryim was doing. It's helped nail down some specifics.

What are all these terms you use?
I used a bunch of terminology here, most of it old, but there's a couple new terms I made up for this explanation.

• Plugin - any .esp or .esm file.
  
• ESM - Elder Scrolls Master. Usually saved as a .esm file, which has the same format as a .esp file, only one bit (as in, one bit of a byte) is different.
  
• ESP - Elder Scrolls Plugin. Most mods are ESPs, but some are ESMs. Their format is the same, but the game handles them slightly differently in some specific cases.
  
• ESS - Elder Scrolls Save. Your saved games. This is loaded after all the ESPs and ESMs are loaded, so that its changes (like what items are in which containers) can override what the plugins say.
  
• BSA - Bethesda Softworks Archive. Sort of like a .zip file, but specific to gamesas's games, and optimized (sort of) for their specific usage.
  
• Load Order - The order that your mods (ESPs and ESMs) are loaded. When conflicts occur, the last loaded conflict "wins".
  
• Install Order - The order that files are installed into your Data folder. When multiple packages install the same file, the last installed package "wins".
  
• BSA Load Order - The order that the game engine loads BSAs. When conflicts in data files occur, the last loaded BSA "wins".
  
• Plugin BSA - A BSA that the engine loads because an ESP/ESM with the same name is active.
  
• Registered BSA - A BSA that the engine loads because it's listed in Skyrim.ini.
  
• Loose File(s) - A resource file in the Data folder that's "loose", as in not packaged into a BSA.
  
• FormID - A unique ID assigned to each "record" of a plugin. Each object in the game is defined by a record, and each record has a FormID. A FormID is 32-bits, with the first 8 bits defining which ESM or ESP the record belongs to.
  
• SKSE - Skyrim Script Extender. A nice utility that everyone should be using. It aims to extend the functions available to scripts written for Skyrim. It's also useful as DLL loader, if one wants to inject some code into Skyrim's game engine.
  
• SD - Script Dragon. Another DLL loader, but with a different goal in mind than SKSE. Both can work along side each other.

4. A Quick Workaround
Thanks to shadeMe (author of the CSE and other great tools), we have a fix for the 255 character limit on Skyrim.ini string entries. It's in the form of a http://skse.silverlock.org/ plugin. I'll walk you through setting it up and using it to get around these BSA loading issues.

1. Install SKSE
Visit http://skse.silverlock.org/ and download the latest stable version of SKSE. Inside the download there should be a few files: an EXE, a couple DLLs, a readme, and the source code for SKSE. If information in the readme contradicts what I say here, follow the readme - it'll be more up to date than what I have here. Just extract the EXE (skse_loader.exe) and the DLLs into your Skyrim root folder (this means the same folder that has TESV.exe and SkyrimLauncher.exe). When you launch the game, instead of using the Steam interface, first just make sure Steam is already running, and use the EXE provided by SKSE to launch the game.

2. Install Nitpick
Download: http://dl.dropbox.com/u/2584752/Nitpick.dll - For Skryim version 1.4.21 only
Download: http://skyrim.nexusmods.com/downloads/file.php?id=9591 - All Skyrim versions.
Source: https://github.com/shadeMe/Nitpick | https://github.com/lojack5/Nitpick
This is the DLL provided by shadeMe. Put the DLL here:

skyrim\Data\SKSE\Plugins\Nitpick.dll

You can create the folders if they don't exist.

3. Setup your BSAs
This is the trickiest part. If you Register a BSA, and it later gets loaded by a Plugin, the Load Order of that BSA effectively gets reset, and it's the same as if that BSA was never loaded via Registering it. This means any BSA you want to register in Skyrim.ini must not be loaded via a Plugin later on. You have two options here:

• If the ESP used to load the BSA is empty, you can simply deactivate the ESP or delete it.
  
• If the ESP used to load the BSA is not empty (for example, the Open Cities ESP), then you will need to make sure the ESP will no longer load the BSA. The best way to do this is to rename the BSA. Be sure you're not subscribed to the mod in Steam Workshop anymore, otherwise it will download the BSA again.

3. Edit your Skyrim.ini
Open the Skryim.ini in your My Games\Skyrim folder, and find this section:

[Archive]sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsasResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - Voicesixtra.bsa

Here, you want to add in the BSAs you want to load. Remember that BSAs are loaded in the order they are listed, so the textures and meshes you want to have higher priority should be listed later.

With the Nitpick SKSE plugin provided by shadeMe, the limit on these entries has been extended to around 32KB - basically, you'd have to be insane to be able to ever reach this many characters.

4. Pro's and Cons
Pro's:

• Since all of your BSAs are loaded via the Skyrim.ini now, Loose Files will again override your BSAs. That means for example, that you can use the HD Texture packs in conjunction with other user released HD Texture packs.
  
• Ordering of BSAs is easy to understand - they're loaded in the order listed in Skyrim.ini

Con's:

• To get the most benifits, all BSAs should be Registered BSAs. Right now, that means manual editing of Skyrim.ini, and manually editing it each time you want to change the BSA Load Order. In the future, Mod Managers will be able to make this process simple for you (I have plans for Wrye Bash to do it, but I can't speak for NMM).
  
• You need to rename the BSAs. This is required so the game doesn't load them as Plugin BSAs. This of course, brings the problem of Steam Workshop downloading the mod again once it detects that the file is missing. Again, this can be alleviated in the future with another SKSE plugin that would trick the game into thinking the BSAs aren't there.
  
• It relies on a specific game version. This means once a new patch is released, this plugin will have to be updated with the new location in the EXE to edit. I'll be working on some methods so this won't require a new plugin, but at most an INI edit for the plugin. This of course will mean that it's very possible the plugin will be updated before SKSE can support the new game version.

Tools
DDSopt
Use this for unpacking BSAs. You can also use it to create new BSAs as well as optimize textures. Ethatron recommends this in place of BSAopt, since it's all inclusive and uses the same code as BSAopt. You can still use BSAopt if you like, but if you ever find the need to use DDSopt then there's no point in keeping both of them.

http://www.skyrimnexus.com/downloads/file.php?id=5755 | http://obge.paradice-insight.us/wiki/DDSopt

Skyrim Script Extender
You'll need this if you want to use http://skyrim.nexusmods.com/downloads/file.php?id=9591. It's also used for other great mods, and with the CK out, it'll only get more important. The only disadvantage is that after each game update, SKSE must also get updated to work with the new EXE.

http://skse.silverlock.org

This should be stickied.

Thanks for gatthering all this information!

Lots of very useful information. Thank you!

These kinds of threads are why I love this community. Thanks a lot!

And yes, this should definitely be stickied.

Thanks for the info! You can register over 80 if you give them 1 letter file names. a.bsa,b.bsa,c.bsa,d.bsa,etc.

Thanks a lot, Lojack. This is a brilliant service you've done for the community. I shall make sure kaburke is aware of this immediately.

Solving the load order issue seems quite urgent. Here are my proposal:

1. Hook to FindNextFile.
I suppose that is the API they use to get the ESP listing over the Data directory. Hooking this method would provide us an "easy" way for
solving the problem.

2. NTFS hard link.
Mod manager could be able to make an hard link of the esp/bsa couple at install time with a name that suits the load order. (i.e prefix the file name with 'z's as many as needed). This won't mess up with SW because you don't touch the original .esp/.bsa couple. However this only work at install time ( if you don't want to mess up your savegame )

my 2 cents.

Great post! Mirrors my own findings, so can confirm. Thanks for taking a leadership position on this. I would forward this DarkOne. Not sure for SW (maybe GStaff?)

edit: Ninja'd by DO!

Thanks Lojack

great info and thanks for debugging the Skyrim BSA loading and how to handle it..

Bad news. Thanks for the info.

Thanks so much for the work and all that useful info, lets hope it gets to the right people at Bethesda and Valve. Moderator please sticky this.

Thanks guys If anyone finds any new information, please report it here! I'll update the OP with new findings as they're...found. And ideas to combat this are welcome too! Ideally gamesas will fix it, but a common utility that everyone uses instead would be the next best thing.

You sir, are silly.

Awesome, I may borrow from you guys if it I like your fixes better than mine

1. Not a bad idea. I haven't poked too much into the actual Windows API calls/etc that Skyrims uses to walk the Data directory, but hooking those calls should actually be easier than replacing the game's loading code. Only thing to factor in those is getting it so Loose Files can override Plugin BSAs again.

2. Not a fan of that one myself, too messy. Plus, it'd be very tricky making sure the Plugin file names are kept the same, so you don't lose save game information. Also, you can't be 100% sure the game is installed on an NTFS partition. Some people install to an external HD, which sometimes are still formatted in FAT32 (why!?!) - so no Junctions/Hard Links.

Thanks for the ideas.

And I PM'd GStaff, so hopefully we can get some official fix for this.

http://www.youtube.com/watch?v=xiwtXHPwGPE

You'd obviously run out of English characters after 36 and would have to use alternates

+1

This is great research, and good information.

That said, I personally would prefer a different strategy, to the one you mentioned, for managing mods and Steam Workshop in its current form.

First, Steam Workshop, in its current form, will cause problems for people with a lot of mods. It's survivable, perhaps, for people that live within the 50 mod subscription limit of Steam Workshop, but I am going to neglect them.

For people using Wrye Bash and BAIN:

Steam Workshop can still be used to download and build mods. But they should never be activated.

I totally agree with your strategy of offering to unpack the BSAs. But I disagree with your strategy of deleting them. No, that's not entirely true: I agree with your strategy of deleting them, but only as a coping mechanism for people that have their save games tied to activated plugins, with mod managers in their current state of existence. This will work, but we can do better.

For the long run, it would be best (for the modder and the user that wants to precisely control their in-game experience), for .esp files that have bsa with matching name to never be activated. (This is a lie, in some circumstances, so I'll come back to this.)

One big problem with Steam Workshop is that it does not currently offer any kind of staging area, where mods can be downloaded to some place other than the game's data directory. So that means that if we want to compensate for this lack, we need to treat the Data directory as a staging area. This is not ideal, but we play the cards we are dealt... Anyways, one variant for dealing with this is:

unpack the original .bsa files
rename the original .esp files before activating them
never activate the original .esp file

This causes an issue for save game management, but it's nice and modular. And, hypothetically, people will eventually start new games. (And, maybe games can be repaired.)

A related issue, by the way, is that I would prefer if I could save the mod configuration associated with a save, so that I could revert to my previous set of mods when I loaded another save. This is a huge issue, though, and a thorough implementation would require support from skse, would require an entirely new kind of mod manager (maybe think of using something based on git as a mod manager, along with having "plugin timestamps" represented in a file), and could require a lot of disk space to deal with some of the storage requirements. This is not anything I expect to see this year.

But wait, this is not our only option!

There is going to be one .bsa which is loaded last. Think of zzzzzzzzzzzzzzzz.bsa {BAIN}.bsa ~LINE-IN-THE-SAND~.bsa, files with unicode characters or code page characters in their name (and it's "lovely" that windows is so "consistent" about such things), and so on..

Hypothetically speaking, it's only bsas for plugins which later get introduced that are loaded after this last one [or that cannot be distinguished from mods that would load after this last one] that need the above "rename the .esp before activating" or "unpack the bsa and delete it" treatments. If (and this is something of a big if) we have proper tools for managing this .bsa then it can hold all "conflict override choice" files, and everything else with "just work".

(Proper tool might include a shell extension for windows, so that .bsa files can be treated "almost like a subdirectory" -- perhaps like modern versions of windows already do for .zip files. But an algorithmic bsa builder might be sufficient).


Anyways, I'm hoping that if we choose an appropriate "last bsa file" approach, that the number of mods that need to be managed in a destructive manner would be minimized.

Thanks, Lojack! A clear explanation and very helpful!

Thank you lojack for this very important work. I've already PMed GStaff regarding load order for plugins and Steam Workshop, but have got no reply yet. I'll see about closing my thread on this issue as you've covered everything neatly here.

I do, however, think that it would be better if loose files always overrode BSAs. Having their behaviour modification date dependent would be too complicated. IMHO.

Hopefully Beth and Valve will take note and fix things.

Very informative. And very silly on the developer's part. I'm very curious to see how this will all play out, I know that Robin and the NMM will be the first to adapt and respond to this situation.

I'd estimate BGS and SW taking a few months to work around this.

I wish modern instruction manuals were as clearly written as this article. I actually enjoyed reading this, believe it or not. Thank you very much for all the effort you put into this research!

Save Games won't be corrupted generally by changing plugin names, and data won't usually be lost. Especially if the plugins are empty. The data loss generally occurs when you are using items (or storage areas) from a mod and the name of the plugin changes.

Can't you already manually do this in Wrye Bash? I know you can save configurations; I have 4 myself. I seem to remember trying to load configurations from save game dependency lists with Oblivion as well.

Awesome post. Thanks for all that work. Now how in the world do we get Bethesda to fix this?

I had also thought about that approach initially when coming up for a plan for Wrye Bash. It'd be easy enough with Wrye Bash - BAIN already knows which files should be winning, just make a BSA out of that and then ensure it loads last. But, I decided against it because:

Uninstall a package, using Loose Files:

• Determine if the file(s) should be replaced by a lower Install Order package.
  
• If so, extract those files from the lower Install Order packages (this is the slowest part) and install them.
  
• If not, just delete the files

Uninstall a package, using a master BSA that's always loaded last:

• Determine the new files that should be in the BSA.
  
• Fastest way:
  
  • Unpack the current BSA (slow for a large BSA).
    
  • Replace/remove applicable files.
    
  • Repackage into BSA (slow, again).

And of course similar flow for installing a package.

So overall, the time involved with installing and uninstall packages with the BSA approach is going to take more time. Going with Loose files you pay up front with unpacking the BSAs once, but going with the master BSA approach you pay for it each time you make a change. Also, you'd have to ensure no Loose Files fall into the mix, since they're always being overridden by Plugin BSAs right now (hopefully that will change).

Most of the other problems to consider (can't have a BSA > 2GB, etc) is solved by using BSAopt to create the BSAs. Another thing to consider is BAIN's philosophy: it's just a managed manual installer - with a giant BSA, it's hard for the user to see what's installed. Still, it's not a bad idea, just not what I'm going to go for with Wrye Bash.

Exactly. But in general it's a poor idea to rename plugins unless you have to: it avoids that loss of data, and BOSS can still recognize the plugins and sort them properly.

Sort of. You can 'Load Masters' on a save game to activate/deactivate the appropriate master files. You can also use Save Profiles to have a different Load Order based on your profile. What you can't do though is change the Install Order based on the Profile.


Good point on the Loose Files, I updated the OP to reflect that Loose Files should always override BSAs, regardless if they're Registered BSAs or Plugin BSAs.

Well, thanks I don't write really, and usually I'm worried that I'll get too long winded or technical, so that's nice to hear.

Thank you Lojack, very good information

LoJack --- Is this correct or does it actually SEARCH for the textures in the opposite order UNTIL it finds the texture it is looking for ?? ( seems it would make more sense for the game to go through the list backwards until it finds the texture it is looking for and then load the one it finds first instead of searching entirely through every file each time !! )

Also IIRC it also searches RAM first and if the texture already exists in RAM it does not even search the files ( this is why it is important to exit the game when testing so that changes will show up properly otherwise RAM may not update and you'll get different results than when actually playing -- at least this was the case with OB so if you had 2 mods that each used a different version of a texture with the same name sometimes you'd get the vcorrect texture and other times you'd get a different one if the second model had recently loaded and was stil stored in RAM.

While the end result is the same in most cases whether it actually loads every file first and then overwrites again as it finds newer ones it seems that would cause alot of problems and slowdowns vs. traversing the lists backwards until it finds the first instance of the texture (which would be faster and consume less resources !