FindFirstFileA/FindNextFileA is not guaranteed to return the files in alphabetical order. It usually does for ntfs partitions but with fat32 it will return the files in the order they were written to disc. IF bsas are loaded in the order that FIndFirstFile/FindNextFile returns them there is no predictable order at all!

Regarding non-english versions: Afaik skyrim should support non-latin characters in file names, Oblivion did. The encoding doesn't really matter as the filename in this case is only used as an array of bytes that will probably be passed into CreateFileA unmodified. There it will be "transformed" back into the correct filename.
Besides, Windows does also support multi-byte charactersets in the ...A variants of functions. They may not be unicode but they still support non-latin characters like Kanji.

Edit: Btw, in case anyone cares: the skyrim launcher DOES use unicode, only the game doesn't.

You're right, that's what I was thinking. Just adding the conflicting files might turn out to be less overhead in most cases. Still, if we can get an official fix, it won't be necessary.

Crap, I didn't know that about the Windows API in relation to FAT! That's definitely not good (at least for anyone still using it, which they shouldn't be).

All true for most points, except ESPs and ESMs. This is because plugins.txt is used to determine which files to open, and Skyrim will refuse to load a file listed in there, no matter which encoding you've used to write it, unless that's Windows 1252 (ANSI). IIRC, Oblivion wasn't quite that limited, but Skyrim is (spent many weeks trying to fool Skrim into doing it, but to no avail).

Very well written , and interesting read. Thank You ! for taking the time to do all this. I have said many times.I will not give up my Tri-fecta...
Bash
Bain
BOSS

The 3 B's. So far it looks like we had it way better in Oblivion....sigh. Already came to the conclusion that I was going to Unpack everything and Bain it all. Unfortunate since most all mods have a BSA.

This so needs to be a Sticky !!

Speaking of an SKSE plugin implemented to fix this, would that be able to also extend the max character limit for the bsa ini setting?

@Lojack: Massive thank you! Very informative! But I’ve got a case study that seems to throw a monkey wrench into your findings. Take my “http://steamcommunity.com/sharedfiles/filedetails/?id=7652” mod.

On a fresh 1.4 install with no other mods, try downloading my mod from Steam Workshop. Activate it. Your load order should now be:

Skyrim.esm
Update.esm
HighRes…1.esp
HighRes…2.esp
MainMenuSpinningSkyrimEmblem.esp

FYI, my blank .esp points to my .bsa which only contains two files:

meshes\interface\logo\logo.nif
textures\interface\objects\logo2_n.dds

However, the game still refuses to load my emblem mesh upon starting the game. Rather, it loads the default mesh from Skyrim – Meshes.bsa. BUT! If you load a save game then quite to the main menu, you get a red exclamation (missing mesh) icon! This indicates it is now trying to load from my .bsa I believe, but it breaks down I believe because I made that .bsa using FOMM when trying things to get my mod to work. A user reported that my first upload did have the spinning emblem AFTER quitting to the main menu.

So very strange. To add MORE confusion, now unpack my .bsa as loose files. Still KEEP my .esp ACTIVATED and run the game. Low and behold, the emblem spins because it is using the logo.nif in the loose files, even though “MainMenuSpinningSkyrimEmblem.esp” is ACTIVATED! Can anyone else confirm this? I hope it is not just me.

Interesting, I'll monkey around with your mod, see if I can come up with answers.

I'll say though, I'd avoid FOMM for BSA creation, the first tests I tried it wasn't working out right, the game wouldn't load anything in those BSAs I made with FOMM.

I enjoyed reading this thread, and I don't even have Skyrim Thanks Lojack. Your dedication to this community is awe-inspiring.

Brumbek: Ok, checked it out, here's what I'm pretty sure is going on. I'll update the OP as well with this info.

First, like I said, FOMM isn't good for creating BSAs for Skyrim. Whenever the game tries to load your BSA, all I get is a giant exclamation point "missing mesh" mesh instead of yours.

Ok, so the strange inconsistent behavior between just launching and after loading a save / starting a new game?

It looks like Skyrim is only loading assets in Registered BSAs and Loose Files when it first starts up. So that means if you install your mod as Loose Files, you'll see it take effect right away. If you install it as a BSA loaded via a plugin, you won't.

It isn't until you actually start a game (Continue/New/Load), that the game loads up any Plugin BSAs. Makes sense if you think about it, saves on startup time for the game. So installed via Steam Workshop, you don't see the effect at first, because Plugiin BSAs aren't loaded yet. Then once you start a game and exit back to the Main Menu, your modified logo is now loaded, now you see the effect.

Only usually for NTFS? Now that sounds like fun.


I suspect that some people will use it for drives that need to be written by multiple OSs. Though putting games as disk intensive as the TES games on an external drive (the usual place for FAT32) strikes me as not such a good idea to begin with

Does the game load all of the actual ESM/ESP files at startup? Or are those too only brought forth once you choose to load or start a new game?

Agreed on the first point, totally agree on the second point - which is why I made my comment I agree FAT32 is used a lot for multi-boot setups and stuff though.

Couldn't tell you on that one, I haven't done any specific testing. I can tell you that the game at least reads the header (TES4 record) of each plugin, because it then uses that to determine which .STRINGS (and .DLSTRINGS and .ILSTRINGS) files to load. I know this because you can change the Main Menu strings using these files, and you can cause a CTD at that point if you have a Plugin missing a Strings file (even if that Plugin's Strings files don't change anything on the Main Menu).

Well, I'm glad I could contribute very slightly to the research...of course this means my mod will never work as intended on Steam Workshop...cries...unless there's some hack made to load plugin data right away, but I suppose only main menu mods would need that.

So grr...at least we understand what is happening. Thank you very much for the help! I appreciate it!

Lojack, your awesomeness is beyond description. It goes to 11. Thanks for the huge amount of work that went into that post.

So what I've learned is...

BSAs are evil. Stay away from them like the plague.

If the limit is placed by the size of the temporary string buffer the code uses, yes.

Edit: Nevermind. Realized I was responding with answer to wrong question.

In any case I tried and I can hack the FindFirstFile/FindNextFile API to return the files ordered by lastWriteDate.

00000203 145.37548828 [1848] FindFirstFileA(H). lpFileName: DATA\*.esp. cFileName: HighResTexturePack01.esp. ftLastWriteTime: 47797524.00000204 145.37573242 [1848] FindNextFileA(H). cFileName: HighResTexturePack02.esp. ftLastWriteTime: 47797560.00000205 145.37596130 [1848] FindNextFileA(H). cFileName: hideout.esp. ftLastWriteTime: 477975d8.00000206 145.37617493 [1848] FindNextFileA(H). cFileName: Acquisitive Soul Gems.esp. ftLastWriteTime: 47797614.00000207 145.37635803 [1848] FindNextFileA(H). cFileName: smakit_house_markers.esp. ftLastWriteTime: 4f35756c.00000208 145.37655640 [1848] FindNextFileA(H). cFileName: portal2mod.esp. ftLastWriteTime: 4f357619.00000209 145.37675476 [1848] FindNextFileA(H). cFileName: dd - realistic ragdoll force - realistic.esp. ftLastWriteTime: 4f35762f.00000210 145.37696838 [1848] FindNextFileA(H). Finish. 

So here I make steam loads the .esp in the right order.

00000211 145.37704468 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\Skyrim.esm. Found: Skyrim.esm 00000212 145.37709045 [1848] FindClose. HANDLE: 017bdb08 00000213 145.37716675 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\Update.esm. Found: Update.esm 00000214 145.37721252 [1848] FindClose. HANDLE: 017bdb08 00000215 145.37725830 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\hideout.esm. Found: hideout.esm 00000216 145.37731934 [1848] FindClose. HANDLE: 017bdb08 00000217 145.37736511 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\HighResTexturePack01.esp. Found: HighResTexturePack01.esp 00000218 145.37741089 [1848] FindClose. HANDLE: 017bdb08 00000219 145.37747192 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\HighResTexturePack02.esp. Found: HighResTexturePack02.esp 00000220 145.37751770 [1848] FindClose. HANDLE: 017bdb08 00000221 145.37756348 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\hideout.esp. Found: hideout.esp 00000222 145.37760925 [1848] FindClose. HANDLE: 017bdb08 00000223 145.37767029 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\Acquisitive Soul Gems.esp. Found: Acquisitive Soul Gems.esp 00000224 145.37770081 [1848] FindClose. HANDLE: 017bdb08 00000225 145.37776184 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\smakit_house_markers.esp. Found: smakit_house_markers.esp 00000226 145.37780762 [1848] FindClose. HANDLE: 017bdb08 00000227 145.37786865 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\portal2mod.esp. Found: portal2mod.esp 00000228 145.37789917 [1848] FindClose. HANDLE: 017bdb08 00000229 145.37796021 [1848] FindFirstFileA[017bdb08]. lpFileName: DATA\dd - realistic ragdoll force - realistic.esp. Found: dd - realistic ragdoll force - realistic.esp 00000230 145.37800598 [1848] FindClose. HANDLE: 017bdb08 

here I see Skyrim is accessing the esp files in the same order I specified.
So, assuming Bethesda is not doing any further sorting we can use this approach either on NTFS and FAT32, if no official fix is provided.

Have you tried it at all to see if it affects the loading of the BSAs? If you want, just send me your hack and I can test it out. Hopefully changing the order Skyrim sees the ESPs in will indirectly affect the order the BSAs are loaded.

I got a new problem for you. I made a mod that just modifies a single script. As scripts are now an external resource they have to be packaged into BSA files if you want to upload them to Steam Workshop. So I have my mod made up and packed correctly, but it doesn’t work.

If I have the compiled script as a loose file it loads fine. If I force the BSA to load from skyrim.ini it also works fine. If I load the BSA via its ESP it fails. I have tested to see if the BSA is loading by placing a modified texture alongside the script in the BSA and the texture is loading but the script will not.

I don’t know if I am doing something wrong and I can’t find any information on it. If you want to see for your self:
http://steamcommunity.com/sharedfiles/filedetails/?id=7999

Progress Update:
Using RYO's hook into the filesytem didn't work out like we hoped. It makes the game read ESPs in chronological order, but internally that data's already sorted that way, so no change there. We'd hoped that doing so would make it also try to read the BSAs in chronological order, but it didn't. We're still looking and poking around, but no breakthroughs yet.


On another, very interesting note: I've made a useful discovery. When Skyrim looks up Skyrim.ini settings, it doesn't go to your Skyrim.ini in your My Games\Skyrim folder first! First it does a check similar to how it looks for Plugin BSAs - it checks for an INI with the same name as your Plugins first! Unfortunately, it searches in alpha-numeric order again, so we can't have a Load Order for INI Edits, but it does mean you could do something like this:

Your mod (better_shadows.esp) requires the user to edit their Skyrim.ini and make the following change:

[Display]iShadowMapResolutionPrimary=4096

Previously, you'd have told the user in the instructions to manually edit this.

What you can do instead, is make a better_shadows.ini file and install it along side your ESP in the Data directory, with the above lines in it. When Skyrim looks for the setting, it'll use the setting in your INI instead! Oh, and these INI files are only used if the associated plugin is active.

Before anyone asks, I already tried this to trick the game into loading Plugin BSAs as Registered BSAs. It works, but you still have 3 problems:

• There's still the 255 character limit
  
• If you have that same entry in multiple INI files, the last one in the Data folder will win - it doesn't merge the results (that'd be odd anyway). That means you'd need some utility to manage it, just like you'd need if you wanted to make the changes directly to Skyrim.ini
  
• The INIs are checked in alpha-numeric order (again)

Now, it is useful as a way of applying INI Edits with less errors (no typing), and being able to revert back to the user's default settings simply by removing this Plugin INI file! Cool Beans!



@Paradox 42: I'll check it out and let you know.

Oh, that is very interesting indeed! Good find, Lojack (again)!

But am I correct to understand that ini-edits in the data-folder override normal skyrim.ini-settings?
Say there's bettershadows.ini, but thre's a different value for the same setting in Skyrim.ini, which one wins?

The last one loaded from the Data folder wins - the Skyrim.ini setting from your My Games folder is loaded way earlier.

Funny enough, you can have a Skyrim.ini in your Data folder as well, since Skyrim.esm is treated just like another plugin in this case.

That was the first thought I had too.

Thank you for this incredible post, Lojack. For Oblivion I got in the habit of always bundling my resources as .bsa files, but I suppose (for the time being at least) I will go back to loose files to prevent these shenanigans.

You could solve some of this by adding a feature to BAIN that prefixes each file with it's ID Number based on load order. Though I guess this would cause issues with savegame compatibility.

Read http://www.gamesas.com/topic/1345708-understanding-the-new-way-of-handling-scripts/ - in short, you shouldn't be modifying original scripts, you should only add new ones (possibly extend old ones) - and in your .esp alter entities that referred to the old script to refer to the new.