Very informative. Thanks for the digging, Lojack!

Thanks for tracking down and organizing all this info.

Thank you Lojack for writing this up.
Just to confirm your suspicion: The character-limit for sResourceArchiveList and sResourceArchiveList2 is indeed 256 bytes in skyrim.

Yes but then using SW is out of the question since changing the BSA name would lead Steam to redownload them the next time you launch the game !

Awesome! Not the fact, the information . Thanks for confirming that, I'll update the OP.

Nope for counting every character counts -- ie. spaces punctuation periods use up the count as well

So as DW mentioned you are closing in on the limit in the first line already with about 50 characters left !

Isn't SW inherently incompatible with this method of BSA activation since it uses the esp as a bsa activator?

Lojack- more like Kojack

You cracked the code. now all my trials make sense. combined with the fact that only bsa made with the CK will work. I've had nothing but failures trying to get bsa made with OBMM/FOMM - maybe want to note that.

I like your approach that this is a bethesda problem, but maybe doing this was a way to avoid the crashes that used to come when two bsa had the same file in it?

What is the best way to ask bethesda to fix this issue?

thanks

That issue was important to fix, but they dropped the ball in fixing it IMHO.

Best way of contacting Bethesda is through Gstaff - Lojack said he did this, but I don't know if he ammended his PM with my suggestion for loose files to always override or not though.

Poor Gstaff must be getting so many PMs, he still hasn't read mine.

As far as I remember, BSAs contained a version number. So if OBMM/FOMM create BSAs with Oblivion / Fallout 3 version numbers, it does make sense that Skyrim would refuse to load them.

Thank you so much for taking the time to put this information together for us. It's greatly appreciated!

Other than FOMM, is there another tool available for unpacking BSA's?

OBMM/FOMM don't make good BSAs for Skyrim. Use BSAopt (or one of the other similar branches of the tool) instead.

As for specifics:

What definitely needs fixing: providing a way to control WHICH Plugin BSA wins when conflicts occur. Right now, it's simply the alpha-numeric sorting of the BSAs. So technically you can control their order, however not in a good way. Renaming BSAs would require renaming their ESPs, which leads to issues I already mentions.

The other thing that I've requested to be changed is the role of Loose Files in the mix. Right now, Registered BSAs have lowest priority, followed by Loose Files, followed by Plugin BSAs. That means Loose Files can't override files in a Plugin's BSA. I request that instead Loose Files always override, so the order would be Registered BSAs->Plugin BSAs->Loose Files.

I just sent him a link, plus a small bit of extra information. I probably should recommend the specific solution as well, I suppose.

I bet with SW and the CK just coming out, he's got to sort through a mass of PM's to pick out stuff that's actually high enough priority to respond to.

BSAOpt. Don't have a link sorry, but it's on Nexus and the OBGE Wiki.

EDIT: Dammit, Lojack, that's the second time you've ninja'd me this week! Getting rusty...

Not real familiar with Fallout so not sure if it is the same as FOMM but I used http://fallout3.nexusmods.com/downloads/file.php?id=34

Well I don't want to harass the staff at Bethesda, but I'd think more than one PM would help. ... and this is all compounded by Steamworks resetting the timestamps on esp. A fix may be a while.

Maybe if we could recruit the CK beta testers to send in their support!

This hierarchy would at least solve the BSA redirection problem. I'm also wondering if the now missing "load loose files on the Skyrim Launcher was part of this problem.

If we could suggest how to solve the multiple BSA problem and they could implement it - I'd think having the dummy esp load order determine which wins would be the most modder friendly.

http://skyrim.nexusmods.com/downloads/file.php?id=247

What happens if you have a 'dummy' plugin that isn't a dummy? Then you've got resources tied to records, which is less flexible. Better they give as flexible a system as possible for us to impose any restrictions we want upon, rather than us having to hack away at a restricted system to give the flexibility we want.

There are two separate issues for modding Skyrim ATM:

1. BSA load order. This is an issue for Bethesda to fix. Lojack has notified Gstaff and provided an optimal solution (IHHO, but IMHO too).

2. Steam Workshop using timestamps for version checking. This is an issue for Valve to fix. I've notified Gstaff and hopefully he can pass that on to Valve.

Unless anyone knows of a better way to contact Valve, I don't think there's much more we can do.

I did some hacks with FindFirstFile / FindNextFile and here are my findings.

WARNING: big text file there! http://dl.dropbox.com/u/954487/skyrim.txt

here is the extract of interest

00010295 980.02227783 [5976] FindFirstFileA[01a94338]. lpFileName: DATA\*.esp. Found: Acquisitive Soul Gems.esp00010296 980.03222656 [5976] FindNextFileA[01a94338]. cFileName: dd - realistic ragdoll force - realistic.esp, ftLastWriteTime: ef6eba6c00010297 980.03247070 [5976] FindNextFileA[01a94338]. cFileName: hideout.esp, ftLastWriteTime: b545fc0000010298 980.04028320 [5976] FindNextFileA[01a94338]. cFileName: HighResTexturePack01.esp, ftLastWriteTime: 49fc2a0000010299 980.04064941 [5976] FindNextFileA[01a94338]. cFileName: HighResTexturePack02.esp, ftLastWriteTime: 6dbf700000010300 980.04095459 [5976] FindNextFileA[01a94338]. cFileName: portal2mod.esp, ftLastWriteTime: e25f2f0700010301 980.04119873 [5976] FindNextFileA[01a94338]. cFileName: smakit_house_markers.esp, ftLastWriteTime: 7b6078c500010302 980.04144287 [5976] FindNextFileA[01a94338]. Finish.00010303 980.04150391 [5976] FindClose. HANDLE: 01a94338

as you can see esp files are loaded in alphabetic order (case insensitive) as OP pointed out. I believe we can hack this process
by ordering by ftLastWriteTime quite easily.

I tried but was unable to write a plugin for skse and wrote my own loader.

QUESTION to skse people. How do I write a plugin to Hook to a Windows API?

QUESTION for non-english people: I noticed that Skyrim uses FindFirstFileA instead of FindFirstFileW, so Skyrim might not support japanese/chinese/cyrillic (not ANSI) file names. Do you confirm?

But I don't want to pack them into a BSA again - I just want to unpack the BSA...

Oh dear god, please don't tell me they're still not using Unicode?

You can unpack only with BSAOpt IIRC.

GStaff's good about getting back to us about real issues, plus knows enough about modding internals to understand all the technical jargon. I think patience here is the way to go. Plus, this thread's only been up what, 8 hours or so?

BSA Redirection isn't needed at all anymore for Skyrim. Loose Files always override files in registered BSAs. The #1 issue is what happens when two Plugin-loaded BSAs have the same resource - deciding which one to load. The #2 issue is that Loose Files can't override a file in a Plugin-loaded BSA, although this one can be worked around.

Which is exacerbated by the fact that EVERY Steam Workshop mod uses a BSA, and a lot of Steam Workshop mods are texture replacers - so you run into the situation quite often with multiple BSAs containing the same file.
I'm personally going to hold off on a plugin to solve the problem until we get some official word back.

For ANSI stuff: yes I can confirm that. Not to mention plugins.txt isn't encoded in a UTF-8 or other similar encoding, it's encoded in ANSI. The Russian version of the game is actually a different build because of this, Russian characters aren't supported by the Windows 1252 codepage, so they need a different EXE that uses the Windows 1251 codepage.

Dear god, they STILL aren't using Unicode.

And BSAopt can output to a .BSA, just add the '.bsa' on to the output file name. Took me a while to figure that one out myself.

Ah, yes, got it sorted. Interface was confusing me a bit. If you only want to unpack, you pick a folder and just say Use Folder, otherwise it wants a file name for the new BSA.

Very good. Thanks guys.

And I've also linked a post in the beta forum to this as well, for added visibility. It won't get lost in the sea of posts as easily this way.

Also, a very excellent write-up and testing protocol. I don't think I'd have had the patience for all that, but your findings mirror my own less than stellar testing.

I think you have assumed, here, that ZZZ.bsa (or whatever name) is primary data. But we don't need that.

I was thinking: build ZZZ.bsa from known conflicts based on lose files.

If that's still too slow [and eventually, I am sure that it will be] there are a variety of ways to manage the needed time. For example, note that the bsa file format can be used inefficiently. (You could append a "zip" file header to a .bsa file that would let you unpack it using zip methods.) But probably a better approach is to just have the user decide when and if to do rebuilds (much like bashed patch), instead of tying rebuilds to package install/uninstall.

Then again, if Bethesda will fix things so that loose files get loaded last (which I think will also matter for sane behavior of Creation Kit), all of this should be unnecessary...

Thanks for all your research on this LoJack, and brilliantly written post