This is a continuation of a discussion I started in the OBSE thread.

Basically, I asked the OBSE team if it was possible to have OBSE give Oblivion the ability to utilise multiple CPU cores and IanPatt replied that it may very well be possible. I then continued discussing this possibility with about half-a-dozen other forum members until Scruggs suggested that this discussion be moved out of the OBSE discussion thread.

The relevant parts of the discussion in the OBSE thread is quoted below. Please excuse me if the layout seems a bit messy. I had to use Code boxes because I got a "too many quotes" error when I tried posting this.

http://www.gamesas.com/bgsforums/index.php?s=&showtopic=1069884&view=findpost&p=15563264

Me again. I probably already know the answer to this question but I'll ask anyway.Is it possible and/or feasable to make OBSE patch Oblivion to be a true multi-core application?

http://www.gamesas.com/bgsforums/index.php?s=&showtopic=1069884&view=findpost&p=15564905

It actually might be possible to make some of the more CPU-intensive parts of the render setup multi-core, but making the game "true multi-core" like Lost Planet or something similar would be a several-month project *with* the source code.Either way this would be done via a plugin, not as part of the core.

http://www.gamesas.com/bgsforums/index.php?s=&showtopic=1069884&view=findpost&p=15565272

@IanPattWell, it's already more than I expected...but is it worthy enough to give someone a try?

http://www.gamesas.com/bgsforums/index.php?s=&showtopic=1069884&view=findpost&p=15565279

@IanpattThat was a surprise. I thought the answer would be no. Anyway, I think making the "more CPU-intensive parts of the render setup multi-core" would be adequate.Why would you prefer multi-core capabilities to be part of a plugin rather than part of the core though? Furthermore, I wonder who would be willing to make such a plugin. I might become inclined to do it myself except I don't know anything about creating OBSE plugins.@BenrahirConsidering the fact that Oblivion only really uses one core most of the time and the fact that running QTP with RAEVWD can cripple even the most powerful consumer-level PCs available today thanks to the first core of the CPU becoming overloaded, I would say that adding multi-core functionality to OBSE is a worthy objective.

http://www.gamesas.com/bgsforums/index.php?s=&showtopic=1069884&view=findpost&p=15565355

@BenrahirI'm sorry but I think it is [i]you[/i] who is mistaken. I am not asking Ianpatt to make "the game "true multi-core" like Lost Planet or something similar" (second part of his sentence). I am asking for him "to make some of the more CPU-intensive parts of the render setup multi-core" (first part of his sentence). I'm making the assumption that changing certain parts of the EXE (guessing mainly the parts concerning AI, scripting and polygon rendering) to use multiple cores is doable whereas going through the entire EXE and making every single part of it recognise multiple cores would be a enormous task (requiring months of work IF you have the source code).

http://www.gamesas.com/bgsforums/index.php?s=&showtopic=1069884&view=findpost&p=15565874

@JdeRauYou already use Oblivion Stutter Remover I presume because it improves performances on multicore system through implementing spincount for existing threads.As to changing parts of the code to be executed in a separate thread I frankly don't know if it's doable, changing a system that wasn't designed for multithreading into a multithreaded system can be quite tricky, even with the source code :P.Is there some place to discuss/read about all the info the obse team has about the Oblivion engine(IDA database? :?)

http://www.gamesas.com/bgsforums/index.php?s=&showtopic=1069884&view=findpost&p=15566477

@JdeRauOBSE is meant to be an extension for scripting and for additional plugin, not be the additional plugin itself.=>Modular is better :).

If it is possible it would mean the end of the world as we know it.....

Wait, what's this all about?

It's about creating a plugin for OBSE that allows certain CPU tasks to address multiple threads (ie multiple CPU cores).

So I'd be able to run Oblivion distributing the tasks over multiple cores.... If the theory proves true, that is?

That's correct.

You asked why team OBSE say plugin rather than core, my guess would be that making "some of the more CPU-intensive parts of the render setup multi-core" might upset single core computers, in which case OBSE would become unusable for anyone still using single core processors. Putting this capability into a plugin would allow all to continue to use OBSE, while only those who would benefit from multi-core (most people nowadays of course, but not all) would need to install the plugin. I hope someone with know-how produces such a plugin, thousands of people would get excited about it.

I wasn't expecting this to create as much of a reaction as it did, which was probably a total misread on my part. Discussion does belong in another thread. Anyway, to clear some stuff up, anything involving Bethesda's code is probably non-multi-proc-friendly. This includes AI, scripting, loading, everything gameplay-related. Once their code goes off in to some of the middleware (netimmerse mainly) then some work can probably be done in parallel. This works since there's effectively a handoff of a static scene graph for processing in a standard, (relatively) easily understood way. These processes are also both extremely slow on the CPU end, making them good targets for optimization, and usually operate on a tree structure, making a multithreaded implementation simple.

This would be done in a plugin since the probability of introducing stability/compatibility issues is *extremely* high. Also keep in mind that I wasn't saying I planned on doing this, just that it's possible and seems feasible within the extremely limited extents I've brought up. In no way would this make all of Oblivion multi-thread friendly, just some parts of the middleware that are currently painfully slow.


Oblivion was designed as an Xbox 360 game, which is a 3-way + hyperthreading system. I don't know how the game is tuned on the 360 or how much of the other cores are used by the existing mt support, but they may have hit a point where making the game more multithreaded would have given diminishing returns. Also, since they don't support mods on 360, they only needed to tune for the content actually shipping with the game. Something like RAEVWD wasn't even vaguely close to their target.

Giving the Oblivion engine proper multi-threading would create a lot of excitement for Oblivion players, especially if the high-end PC owners will be able to play with QTP3 and RAEVWD at the same time (OMG!).


This comes from TweakGuide.com's Oblivion tweak guide:

Multithreading Tweaks:bUseThreadedBlood=1bUseThreadedMorpher=1bUseThreadedTempEffects=1bUseThreadedParticleSystem=1bUseMultiThreadedTrees=1bUseMultiThreadedFaceGen=1iNumHavokThreads=5iThreads=9iOpenMPLevel=10All of the above settings seem to relate to the use of the GameBryo engine's multithreading capability. Multithreading splits tasks into 'threads' where possible, and runs them in parallel across both cores of multi core or HyperThreading (virtual multi core) CPUs to improve performance. Note that raising the values of the iThreads, iNumHavokThreads and iOpenMPLevel settings very high doesn't automatically mean it uses that many threads - it all depends on how many threads are actually possible based on the information being processed. Experiment with these variables but if you experience problems reset them back to their defaults.

This seems to imply that Oblivion does support multiple cores. There is also the quote by Dogswood in the OP which says that Oblivion does support multiple cores, just not very well. Therefore, I don't think the statement that Oblivion's code is not multi-core friendly is completely correct.


That sounds like a plan. Now we just need someone to implement it. I agree with your first post on this subject that making the whole game multi-core friendly would be too much work. Even just the part mentioned here would probably be a satisfying improvement over what we currently have. That and it might not be too difficult to add some more threaded applications since the architecture should already be there (don't quote me on that though).


Fair enough! I formed my own reasons but I agree that it should be part of a plugin. I don't know if I implied anywhere that you should be the one to add multi-core capabilities but I'm not expecting you to do so. Amongst other things, it seems that your team has enough work on their hands just getting OBSE v18 out of it's beta status.


That may be so but considering the differences between a PC and the Xbox 360, I think that may be too simplistic a comparison. During most of the game's development, PCs only had one CPU whereas the 360 had 3 cores it's whole life. Also, I can't completely believe the statement that Bethesda didn't take into account the fact that mods would be created for the PC version (I can believe that they didn't plan for RAEVWD though).

As a quick test, I decided to compare Oblivion with Fallout 3 (which is a true multi-core game). My PC is a Pentium 4 3.0GHz HT with 2GB RAM and a Geforce 7950 512MB AGP graphics card. I started both games with no mods installed, using their own script extender in each case, set their video options as high as they can go except for a few cases (Oblivion's trees and grass were as low as possible, Oblivion was using bloom lighting while FO3 used HDR and both were set to 800x600 with no AA) and I looked at the sky as soon as I could. Oblivion's framerate was 30 FPS (clear sky) while Fallout's was 40 (clear sky with HDR sun in view). I'm not completely sure what to make of these results though. My guess is that this proves that Oblivion's CPU part of the equation could still use some optimisation, maybe in the threading department.

I'm not sure how much warning Bethesda would have gotten that dual core CPUs were comming but you yourself said that it would take a few months of work to change the program to completely support multi-threading. On top of all the work that would still have to be done with Oblivion at that point, I don't think Bethesda would have managed an April release if they tried adding full multi-threading. The next best solution, which they used, was partial support. I wish they chose the second option though (delay the release). Then we wouldn't need this discussion.

This Multi-Core stuff sounds really interesting and I never fought about fixing it up. I have looked at the Shadow-Engine ones at Core-Code Level and found some interesting things, but I never got it to work in game, to get everything to cast shadows. Oblivion and Fallout do seem to have the exact same Shadow-Engine. I will try to find the Multi-Core stuff this week and see what I can come up with. Maybe that we can use the Fallout optimisation stuff to make Oblivion use Multi-Cores better.

• bUseThreadedBlood - Does nothing. Isn't even read by the runtime.
  
• bUseThreadedMorpher - Is read by the runtime, but isn't used for anything.
  
• bUseThreadedTempEffects - Is read by the runtime, but isn't used for anything.
  
• bUseThreadedParticleSystem - Is read by the runtime, but isn't used for anything.
  
• bUseMultiThreadedTrees - Is read by the runtime, but isn't used for anything.
  
• bUseMultiThreadedFaceGen - Appears to do something. (probably generation of head meshes for NPCs in the background, as well as remorphing when people start wearing helmets)
  
• iNumHavokThreads - Is passed to Havok and probably does what it sounds like.
  
• iThreads - Is read by the runtime, but isn't used for anything.
  
• iOpenMPLevel - Is read by the runtime, but isn't used for anything.

I would bet that these .ini options are either 360-only, or completely unused. The game does use some threads when loading objects and setting them up in the scenegraph, which you can see in one of the debug text modes (current BSTask count).


Accounting for and spending a significant amount of development time to improve the performance of are two vastly different things. It doesn't make financial sense for them to overengineer their engine in order to support user-made mods.


Which, at the time, would not have significantly improved the performance of the game due to the low number of multi-core systems on the market and therefore wouldn't have made them any more money. In general, companies don't spend lots of money unless they get something (aka more money) out of it. (obvious oversimplification with many counterexamples, but in this case the cost:reward ratio is horrible)

Wow! I guess that tweak guide is as outdated as some people say. I need to make some changes to my website. Hopefully there is a way to extend the threading capabilities beyond what you mentioned here. Otherwise, that suggestion you made about the middleware may be the only way of doing multi-threading (correct me if I'm wrong).

How did you figure this out though (but then again, I am talking to someone who disected the game's EXE)?


I guess that is correct. Point awarded.


You are correct there. Another point to you.

@Sjors_Boomscors
You seem to know what you are talking about so thank you for taking the time to try figure this out. If you manage to figure this out, I imagine you would become one of the most renowned Oblivion modders ever. I would probably suggest that you start off with the suggestion IanPatt made about the middleware as this seems to be the easiest way presented so far to implement multi-threading:

After this has been investigated, I think you can then try making AI, polygon rendering, etc multi-threaded.

IanPatt, is there any additional information you can provide Sjors about this middleware that could make this task easier?

Read the thread due to my intense interest, not my expertise, which is zero.

However, would it be at all possible to attempt to have the script engine run on a separate core? Or, more generally, rather than full multi-threading, is there anything susceptible to running on a different core that would not cause too much trouble?

I do know that if there were any differentiable exe's running simultaneously, one can assign separate core affinities for each as a cheap multi-core hack, eg with http://www.prnwatch.com/prio.html, but Oblivion seems monolithic with all run by the oblivion.exe.

At the moment, I use Prio to isolate oblivion.exe to its own core, which at least means there is no competition from background OS threads.

Hope I am making sense; sorry if not.

Oblivion does use multiple threads. Much of what OSR does is oriented around improving Oblivions ability to multithread. For instance, Oblivions vanilla heap isn't bad until multiple threads try to use it at once; heap replacement is only needed for multithreading.

But yeah, it would be better with a lot more stuff moved to a seperate core. Specifically, the call of 0x0070C0B0 at 0x0040CCD3. The calls at 0x0040DF06, 0x0040DEA1, 0x0040DC2A, 0x0040DBDC, 0x0040DBA5, 0x0040D8D8, and 0x0040D6F7 can also take a little time, but the one at 0x0040CCD3 is really big one in my test case.

Moving scripts to a separate core probably can't help performance much, since not much time is spent in scripts anyway. And presumably there will be overhead for any attempt to move stuff between threads.

http://en.wikibooks.org/wiki/X86_Disassembly should give you an overview.

I'd use it, though I wonder if it could be used the handle Quad Core 64bit CPUs better,.... FO3 had an issue where if you had more than two cores it liked to freeze. Unless I missed a thread disproving that elsewhere (I got the STEAM version Standard Game ... might end up getting GOTY later on so I can get one of the CANON endings... grrr.)

I am going to try a few experiments with moving portions of Oblivions graphics code to another thread. Dunno yet if it will work decently or not.

Oblivions threadedness or lack thereof is rather confused. Or at least confusing to me. The task manager says that it never uses any significant portion of a 2nd CPU. At default Oblivion.ini settings it tends to suffer greatly from its heaps lack of willingness to handle requests from multiple threads simultaneously. It's littered with calls to Interlocked* functions.

My overall take on Oblivion multithreadedness is:
1. It does not multithread significantly during normal gameplay
2. It does multithread certain things significantly, like loading new cells
3. When it multithreads stuff, it tends to do so badly. OSR partially fixes this.
4. Certain parts of it were intended to be more multithreaded than Oblivion ended up being. In some cases because the extra-multithreaded stuff didn't get finished, in other cases because the people using those parts didn't understand how they were supposed to be used.

Odd, because I never had any issues with it and I'm running a quad-core CPU.

I guess that's why FO3 performs so much better for me than OB though since people are saying it handles multi-core CPUs better. It's never bogged down even once for me.

The same thing happened to me with FO3 because of having a quad core cpu and I had to change some ini settings to get it to work correctly without crashing. From what I understand even Fallout 3 doesn't handle multithreading correctly either and there can be problems on systems that have more then 2 cores. The reason why is because Fallout 3 was made to use 2 cores and doesn't use the other cores in the system. I think the reason why games can have problems with systems that have more cpu cores then what they were made for is because the program doesn't have proper thread management so then it will have problems when there are more cores then what it was designed to use.

PS. You can remove problems with having more then 2 cores by changing the affinity of Fallout3.exe so that it only uses 2 cores.

I salute the valiant endeavor and look forward to hearing more, though I am guessing it will take quite some time. Here's wishing best of luck from the peanut gallery.

@Hlafordlaes
In the last two comments made concerning this matter in the OBSE thread, QQyix and Scruggs mentioned that scripts rely on the fact that they aren't executed in parallel so it would probably be best that script processing remain on one core at all times. Whether it stays on it's default core or gets moved onto another core though is another story.

@Skyranger
Thank you for taking interest in this. I knew that you would probably be the best person to sort this out.

You may also want to take note of IanPatt's comment regarding middleware:

I don't know anything about this though. It seems like IanPatt disappeared before he could explain this in more detail. Anyway, I'll be carefully watching to see what you can come up with.

@ShadeMe
Thank you for your reply. I was more specifically refering to how IanPatt knew what those INI variables did though.

I tried FO3 on a q6600 and I also did not have any problems.

________

Would it be possible to separate Oblivion generated sounds from the rest? The sounds and musics being processed by another core?

+1


Moving it to another core requires that there is something else to run at the same time.


We talk on a pretty regular basis.


anolysis of the EXE. Usually we don't go in to more details than that on the forums.

Oh if a plugin would be made for OBSE to use Multi Cores it would be awesome!

I didn't think it would be possible for Oblivion but I noticed that Nehrim has this option, so IT IS POSSIBLE! :celebration: And now the difference between Nehrim and Oblivion for me is that in Nehrim I almost never have FPS problems, at least so far.

It would be great for me since I have Quad Core, and for others with multi cores of course.

Is it possible to find out what the Nehrim creators did to be able to use multi cores and make a similar feature for Oblivion?

Ask them?

Really? My system is quad core too. I have minimal problems with Oblivion but enormous ones with Nehrim