Are you talking about the symbol table ?

Symbol tables are always generated when compiling code into object files. Because symbols are needed when linking different object files into one executable. Or into a library.

I think symbol tables can also be used for debugging purposes. But on top of that, there can be more debugging stuff inside an executable. At compile time you can specify what extras you want. And some of those extras will be inside/interweaved with the code. You can't get it out afterwards, except by recompiling.

The symbol table is different. You can "strip" it later at any time. Even endusers can strip it, if they want. The stripping is usually done to save diskspace. Or, if you ship executables to customers, to give them less info to mess with your code. I assume that Bethesda has stripped the TESV.exe before shipping. But that doesn't say anything about which compile flags/options were used.

But I assume you know all that. So I probably misunderstood you.

I think the obvious answer is they actually believed that 10% number for the PC market they threw out there, regardless of how ridiculous it was and they probably had no idea how big the digital distribution market is on PC games. Hopefully their delusions have now been removed considering how big a hit Skyrim was on PC and PC will be a much larger focus for them in the future. Who knows maybe they'll even support more than two cores in their next game? Better to be a decade and a half late than never, right?

That's a good point. Bethesda ought to care a bit more now when they've seen how big the PC sales actually have been.

You're completely right. In addition to just the names, further information about how the functions allocate their local variables on the stack space can be used to rewind into the complete context of any function higher up in the call chain. Also, a map between the relative virtual address space of a module and all its corresponding file names / line numbers enables a debug runtime, like msvcrtd, to automatically resolve the exact location of crashes as well as the content of the variables in the context and the complete call chain at that point in time. So far, the only thing I've found are the file/line combinations for failed asserts, but as far as I remember, they've been disabled and just the text references remain.

This stuff can be done either directly on the user's computer whenever it crashes, however this requires the debug information to not be stripped away. Otherwise, a minidump can be created and be used by the developer in combination with the correct debug symbols for the modules that have been present in memory when the minidump has been created. The latter also requires some mechanism for catching, collecting and transmitting the minidumps. Firefox does this, for example.

What compiler flags have been used can sometimes be determined from the debug symbols, however, it is usually sufficient to just look at the disassembled code and anolyze which things the compiler would've optimized away. This works quite easily once one has debugged on asm level a few times - the flags all have characteristic effects. For example, setting the flag to optimize for small binaries produces other instruction encodings and even chooses different instructions in the same code situation, also proper alignment isn't enforced and so on.

For example, TESV has a huge amount of functions looking exactly like, or very similar to:

void* get_this(void* this) { return *this; }

or

void* get_this_offset_8(void* this) { return *(this + 8); }

Here, the "this" variable is actually passed through ecx, so it's using the __thiscall convention, meaning it is a C++ getter method that hasn't been inlined, even though the whole method can be optimized away so just a single instruction remains. In conservative inlining mode, the compiler calculates how much code would have to be generated when inlining it and if that's under a certain or configurable limit, it will automatically be inlined even if the function doesn't bear the inline keyword. Varying degrees of inlining aggressiveness vs. additional binary size can be configured in all the major compilers.

In Skyrim, the function I described here as get_this() has three copies and is being called in about 30000 different places in the game code. When profiling the original executable, it is also shown as one of the functions the CPU spends the most time in, simply due to the fact that is called tens of thousands of times per second and the unoptimized function body is quite large in assembler code. This is how and why the whole hard-crafted optimization was a viable route to go for Skyrim. Typically, this doesn't work well because optimizing compilers tend to distribute the time the CPU spends very evenly across the whole code, so that only algorithms with bad runtime behavior will stand out instead of unoptimized code constructs.

I agree. And thank you for modifying the OP. It is sloppy code, and I hope they enable some level of compiler optimization in 1.4. Arisu(?) has done good work here.

it was the same with daggerfall, morrowind, oblivion now its skyrim. bethesda is very good in creating open world sceneries and really bad when it comes to optimization. glad they also know it, thats why we get the CK every time

maybe someday we will get an open world rpg with the crytek engine, a storytelling from bioware, and the "you can do what you want approach" from bethesda. that would be fantastic

I'm pretty confident that with a fully optimised skyrim..... ugrids set at lets say 9 or 11 might actually be possible with a significantly HIGHER performance boost due to the huge CPU bottleneck that is currently occurring..

FPS above 30fps isn't a HUGE deal but it's still notable.... personally if i had my choice i'd run anything i could at 120fps or faster provided the screen is capable of delivering...

The boost experienced will effect those the most from my own testing, that are the most CPU bound... in my case on my machine listed in the specs....i get a relatively minor boost... but there is still an improvement..

Meanwhile on a poor little dual core cpu running 1.8ghz, there is a gigantic improvement..... being able to run that game at a little higher setting and maintain the same fps is nothing to scoff at .. specially when that slightly higher setting resulted in 15-18fps and the change over produced a 27-32fps difference which by my calculations is well above 40%

It's all about efficiency.. and proper optimisations.. or even just a little optimisation can make a world of difference. IF my 2 cores +HT threads on my machine listed below.... weren't pegged at 100% cpu usage at all times (their singular values).... i'd know the game wasn't being cpu bottlenecked like it is. Specially provable when upping the frequency from stock to lets say 4ghz REALLY increases the fps in numerous places.

Essentially what i'm saying is that FPS and CPU performance in this game have a direct correlation in situations where your most definitely NOT CPU bound...

A recent review showing crossfire/sli/top Video cards in single and dual/triple or quad sli/crossfire (and yes i know sli/crossfire in numerous cases do not work with skyrim).... show that the high end cards all seem to hit the near exact low/avg/high FPS counts regardless of what is done....

Are you going to continue patch developing ?

My game runs stable without any mod nor any .ini tweak, than again ive a hardcoe PC. But when i start with some HD textures and .ini tweaks it becomes clear that the game wishes a lot of extra optimization, as better usage of multi threading and GPU usage..

Results: Cpu usage= ~25%, Gpu usage= 40-60% (SLI)

Like Captain John Shepard said.....Game is coded terribly. You could disagree....but you would be wrong.

Thanks for the explanations, Arisu.

So if I understand correctly, the issue is kinda like this.
- The engine is written in C++.
- In C++ there is a concept of the "this pointer". This concept occurs a lot in most C++ code.
- Accessing this "this pointer" goes via special short function-calls. These calls are probably hidden inside the C++ language syntax ?
- For some reason, by default these functions are real functions, with all the overhead associated with full function calls.
- Because the Skryrim executable has not been compiled with the proper optimization flags, these calls stayed full function calls.
- Because these calls are everywhere in the code, and happen so to be frequently, they cause a noticable burden. Around 10-30% maybe.
- If Skyrim was compiled with the proper flags, then these calls would have been inline functions.
- And then in stead of a full function call, they would have been one (or just a few) machine instructions to de-reference a pointer. Maybe add an offset. But the body of the inline function would have been an order less effort than the whole effort of setting up a function call itself ?

Am I correct so far ?
Now what you did it:

- You looked at all places where functions call these short this-pointer-related functions.
- You replaced those functions with slightly altered functions. Where you replaced the machine instructions for the function calls, to machine instructions that would do the dereferencing directly.
- Then you put those functions in a dll, and made sure the TESV.exe would call your functions, in stead of the original ones.

Correct ?
If not, I don't see how you could replace those this-pointer-functions with new code, without replacing the TESV.exe ?

I also assume that you could do this, because the code in the body of the this-pointer-functions is smaller (less or equal bytes) than the machinewords needed to do the function call. Right ? (I think I read you mention something like that). I assume that there will not be very many other functions you can replace in this way. Bethesda will need to recompile the exe to improve it further.

Very nice work.
I used to be a programmer myself. C, not C++. And in a Unix or embedded real-time OS environment, not in a MS/Windows environment. I prefered to stay away from "the hardware". We had a tools team, and platform teams to take care of those issues. But I find it very interesting to see what you did. Imho, it requires some guts to dive into someone else's exe. Especially if you don't have the source code. I would never have even started trying.

Meh its best not to try to understand a code when u r not rly into programming.

Copy paste from http://www.cplusplus.com/doc/tutorial/pointers/:

We have already seen how variables are seen as memory cells that can be accessed using their identifiers. This way we did not have to care about the physical location of our data within memory, we simply used its identifier whenever we wanted to refer to our variable.

The memory of your computer can be imagined as a succession of memory cells, each one of the minimal size that computers manage (one byte). These single-byte memory cells are numbered in a consecutive way, so as, within any block of memory, every cell has the same number as the previous one plus one.

This way, each cell can be easily located in the memory because it has a unique address and all the memory cells follow a successive pattern. For example, if we are looking for cell 1776 we know that it is going to be right between cells 1775 and 1777, exactly one thousand cells after 776 and exactly one thousand cells before cell 2776.

What a naive little rock you live under.

Huh...that would be unique in some ways

Gotta say I agree with the OP. Maybe not 100% buuuut the aforementioned mod DOES improve fps by at least 20% and this is from someone fiddling around with compiled code? If someone else can do that to YOUR exe without access to the original code well... You funked up big time and you should just deal with it, shrugging it off or shirking away from facts won't get anything done, nor will blindly backing those who have, in fact, made an obvious mistake. Hoenstly I feel bad for all the people out there who are stuck not knowing what causes stuttering, and low fps rates and that they can be improved by mods, like this and oldblivion, who are unable to play Skyrim to the extent or with the enthusiasm it should and could easily have. All because of simple oversights, or even lazyness? Personally I don't care whether it was an oversight or just lazyness the past is what it is, but I WILL be extraordinarily dissapointed if the Dev's do not take the time to at least look into this and turn SOMTHING out. Our actions define us, man up Beth.

I have a hard believing Bethesda did not leave these optimizations out on purpose for some reason. Its very hard to believe any company selling a commercial quality game would forget something this basic yet so important. I think they need to fix this though if its actually true and its realistic for them to do. Its not reasonable to expect some good programmer like this guy to spend tons of free time disassembling the whole game because of such a stupid mistake if they actually did do that.

Before everyone goes off on the wrong direction, a few clarifications: Arisu (mod author) made this code in about a day, but this doesn't necessarily mean that it was a grievous oversight on gamesas's part.

As it has been explained before, situations like this can happen when the game just fails to compile...and believe me, big projects WILL do this A LOT, and adding more people can INCREASE the chance of this happening...so they turned off various optimization flags and see if it compiles after that, and it worked. After trying to add the optimization back in it could have failed to compile again and again, until they finally decided to live with it. Over the course of several iteration of the project the unoptimized code was marked stable and became standardized. (This is only one possible scenario out of many)

We also don't really know what other effects this change may have. It really should have none except speeding things up, but the way Windows and DirectX handles itself is pretty darn dirty, so there could be side-effects. A few people, I've counted 2 so far out of many, many who reported great findings, are reporting longer load times and we have yet to isolate why (it's most likely not this plugin specifically, something else on their system). Additionally, this code may cause additional problems on weird system setups we never knew existed, but gamesas's Compatibility Lab does.

Like the Large Address Aware flagging, time will tell if this is truly an optimization that should be patched into the main game.

Edit: From http://www.gamesas.com/topic/1322063-rel-tesv-acceleration-layer-thread-no-2/ we're already getting reports of people having issues when starting a new game with delayed scripts and unresponsive AI. Not sure if the plugin is exactly to blame yet or if it's exacerbating an existing issue, but since the plugin was made in just one day, I think it's at least an equal chance of either.

What a dramatic thread. Guys, if you aren't into professional coding, don't even try to speculate about it and give "expert" opinions.

2 OP and co - thanks for the mod! Gave a nice boost in cities indeed.

Disabling optimization may've been a last resort against bugs. Patch changes code in one bottleneck, when opt flags make all the code to be compiled with optimizations, and somewhere in physics/math/threads it may make heisenbugs. If they had a choice between a more or less stable release to 11.11.11 and a +10% faster but randomly crashing code with a serious lack of time to properly debug it, no surprise they'd choose the first option.

Use the patch, be happy. No need for a forced drama even here, seriously.

I disagree, it's definitively a developer-side problem. Look through the object hierarchy some day (the game includes RTTI, so you can easily reverse-engineer it without the source code). That alone screams "last century game design!" - deeply nested multiple inheritance, interface classes having default implementation, WTF?

Bioware and Bethesda are like friendly rivals, they are in the same big industry working on similar games. I wish they got together and made a game, that one would probably be the game of the decade.

There are 2 reasons why Beth release the game like this.

1. They design the game around the Xbox360 and port it to PC and PS3. (this SAVE MONEY)
2. 11.11.11 deadline. (delay would have cost time and MONEY)

It is frustrating to pay for a game and get such a "BETA".

- unoptimized (for PS3 / PC, see reason above)
- bad texture quality (check on "skyrimnexus" and see for yourselves)
- Bad scripting / lack of testing skripts (but that is nothing new for a TES game)
- slow patch progress (give a "modder" Construction Kit and he would fix you a [censored]load of Quests, Perks, weapons.... in 24H flat.)

Oh last but not least (yes i know i am Trolling but i can't resist to think about it)
- I bet allmost all of the Beth staff went on hollyday at 12.11.11, fine with me but please fix your [censored] before doning so.

I code for a living as well.

I think the one thing that stands out as slightly incompetent is that Beth never ran a profiler on the game. I mean when you write a game, that's the first thing you should do! Profile to get rid of any bottlenecks.

Being that the OP ran a profiler and found these issues probably withing 10 minutes, that points out where gamesas failed easily. I mean, the profiler will show you the exact spot where the problem is ! All they had to do was run a profiler at least once! Who doesn't do that to a modern game?



-niko

I like how there's so much fan-boy lapriding despite this major blunder, not to mention all the other bugs and problems with Skyrim.

I also like how many so-called "programmers" fail to realize that everything regarding game development isn't about programming. Code monkeys are just the grunts. Managers, designers and software engineers actually make the decisions.

The problems show sub-par management and design.

I'd like to put forward the possibility that you don't know what you're talking about.

I'd also like to put forth the following. These a facts, and not opinion, as seems to be the basis of most of the posts in this thread.

Fact 1. We don't know why Beth "left" these errors in.

Fact 2. Bethesda may or may not be incompetent. Stating that they either are, or aren't, is opinion and subjective.

Fact 3. Bethesda must have some talent and can't all be idiots, they're the only dev creating truly open world RPG games with the depth of the ES.

Fact 4. There are some very talented and intelligent people on these forums.