I just read this article released in August.

http://www.kotaku.com.au/2011/08/the-skyrim-team-has-a-better-plan-for-squashing-bugs-this-time/

"Before launch, Howard said, Bethesda is testing Skyrim rigorously and more heavily using a system that runs automated versions of the game overnight, searching for bugs and helping the team squash them. “We’re getting better at it,” Howard said."

Pretty sure it wasn't "rigorously" tested in order to meet the 11/11/11 deadline. Or maybe it was and they chose to ignore a multitude of obvious bugs in order to reach the deadline.

Edit: Sorry, just in a BLAH mood after having to reload a save that made me lose an hour of gameplay. Not a big deal but its 3am and I'm cranky.

Agree, but its a Bethesda game, what'd you expect?

All I can say is 'sorry to hear about you're gametime loss, but if you don't like the game don't complain about it and quit'

An automated system to detect bugs?

I think they need to run their automated bug detector through their automated bug detector.

Oh I absolutly love the game, but find that article very amusing.

think there are just a lot of things that can't be found by an automated detector, should've had more actual testers.

thats not what they wrote.

Bethesda: I could come and test your games for you. Just pay me in Doritos and Dr. Pepper.

@ Aryja:
It's a system that automates the game and finds bugs.

Skyrim is the most stable Bethesda game I have yet to play at launch. 200 hours of play time for me and it's only had three CTDs. For contrast that's the same number I had with Mass Effect 2 on the exact same PC. I did have a problem with the game, and my PC, freezing but that was on my end. Once that was solved there have been zero freezes. If Bethesda did that sort of testing for stability it seems to have worked from where I'm sitting.

There is some internal scripting issues with all the different stuff going on. The only one I've run into that's got me miffed is Esbern's potion not giving me the ability it should.

what this person said

It is NOT a lie.







They were running spelling checkers.

*bolts*

I find this one even more disingenuous.

They are also adopting this new EA Sports-style “live update” system that will let Bethesda tweak the data in their game almost immediately. If a quest isn’t triggering because players are doing some unforeseen combination of actions, that might be addressable by tweaking the data tables that dictate the flow of events in the game. If the game is proving to be unbalanced in an unexpected way, that can be fixed on the fly too. And these fixes — Howard prefers to call them “improvements” — will take effect fast. “We can change stats on a sword and it will be available to everyone in a few hours,” he said.

If I understand correctly, they made a bot to run the game continuously that way, they can identify 'random crashes' but not the quest bugs and stuff, I think.

I find Skyrim a lot more stable than Oblivion or even Fallout 3/NV to be honest.

The automated system probably just randomly runs the character around and maybe gives it an AI to simulate fights. The goal is to try and build CTD logs, either with or without detailed crash information, so that they can reduce random crashiness in their system. It would not do much for finding bugs that occur only under certain circumstances, or bugs that affect gameplay without crashing the game.

Its unlikely that their automated system represents behavior similar to an actual player, who will be running dungeons, going on random killing sprees in cities, using a wide variety of situational skills/equipment/consumables, and saving/loading on occasion. It sure as hell doesn't detect things like backwards animation, resistance calculation problems, or damage scaling issues. It probably doesn't track stutters or performance/framerate drops either, although it would be nice if it did.

I just feel like the whole article was fabricated for the players. He just said what we wanted to hear, without there being any validity to the statements.

Still, bugs or not, Skyrim 10/10

If only. I've found quite a few misspelled words with subtitles on.

This makes sense.

I mean, even if they only had a team of 5 testers, surely they would have found and squashed a majority of the quest related bugs. But it seems they only tested it for stability; not scripting errors.

Lies, eh? I don't give a rip about your bad day caused by a video-game

When I read somebody chanting "lies", I feel that it's my privilege to ask the person to back that up with fact instead of opinion.

Care to show how it was a "lie" that they tested as your quote indicates? I'm all ears

He's referring to the quote about the game being rigorously tested and that they're getting better at squashing bugs. While no one can prove that they didn't test the game as extensively as they claim, the pure number of bugs and glitches should stand as a testament that they failed. Skyrim's as buggy, if not more so, than Bethesda's previous releases. They still haven't patched any of the major gameplay breaking glitches a month after release, and don't even plan to start until January. Releasing a game this incomplete and not even bothering to fix it until months after release is becoming a standard for Bethesda, I fear where this will lead the open world RPG market.

Mass Effect 2 never crashed on me. Skyrim has crashed several times until I got the 4gig fix that they created for New Vegas.

They need to run thier spellcheckers though a bug detector then, and buy a +grammar version.

Haven't game companies for over a decade had access to programmes that run the code though themselves compling list of collisions? That sort of thing would be unlikely to find that picking up an item before a quest breaks it, although what bugs me more then they remain in game is the people behind it didn't think "oh well they can pick up these items at anytime because we allow them to go whereever, lets at a line to check thier inventory for the item like we did for the dragonstone", it shouldn't have happened in the first place its just so slap dash features are missing, I'm surpising the script to walk isn't missing from some building. It creates such a massive problem of the game getting in its own way, I explore dungeons but am afraid of what I pick up, especcially books I'd rather not have a long list of incompletable misc quests.

There are two drawbacks when using automated testing programs. First, they are human creations and are themselves prone to general human errors that occur in any piece of software. Second, they are typically programmed to follow happy path routines in that each quest or activity or function is tested to ensure they meet the exact design parameters in a vacuum - typically in sequential order without any deviations or other tasks or actions to interrupt that sequential flow.

Common or significant problems can be added as additional tests; but these are often of a lower priority and an as time permits.

Consequently, game players often become sidetracked and work on multiple quests at once or shortcut quests or unintentionally break the flow of a quest by stealing an item or killing something. The massive number of these what if or alternative user scenarios are seldom (or with much difficulty) added to automated testing routines. This explains why Blood on the Ice is so problematic for people since the start cues are often missed by players. This also explains oddities like not being able to become Thane of Falkreath because you still have an open retrieval quest for another character. Automated programs simply do not operate this way and it requires massive work to build in those scenarios. Until computer AI reaches levels approaching Skynet or even WALL-E, it will be very time consuming and expensive for automated game testing programs will continue to pale to actual human testing efforts.

One thing Bethesda could have done (and maybe they did, I do not know) is to engage in some customer or user Beta testing. I suspect if more time had been spent on that, the lag issue and other annoyances would have been repaired or minimized before launch. In lieu of that, most of the game's issues (apart from the lag) are design shortcomings and not really testing issues. The design scenarios needed to address more alternative scenarios that focused on broken or missing or out of sequence scenarios.

I agree. I've only seen a couple of instances where there were misspellings in the subtitles, but books are just full of misspelled words. I'll just have to assume the person (in-game, in-universe person) that wrote the book just wasn't very good at spelling.

I do think "ZOMG LIES!!!" is overstating the case though.

Agreed. Not lies, but a hollow statement by Bethesda.

Possible. It seems to me that random crashes are less common than Oblivion at initial release, but I'll admit I don't have concrete data to back it up, its just an impression. Oblivion at its current patch level is certainly more stable, but thats an apples to oranges comparison since its at a more mature patch level.

Quest glitches, crashes specific to certain UI processes or situational crashes weren't addressed before release, but they generally can't be found with any automated system.

Performance fluctuations can be investigated using automated tools, but it would be a much lengthier process and might not be practical until after all other parts of the engine and setting are set in stone, ie after it has "gone gold".

The biggest performance-killer for PCs are shadows from what I understand. Having the CPU calculating shadows seems like a very poor design decision. Hopefully Bethesda will consider writing code to offload that burden onto the GPU where it belongs.

I wonder if BGS has considered preferentially hiring mildly autistic people to do the testing. I've seen research that suggests they actually make far better QA testers than your average joe. There's even a company in Denmark (called Specialisterne) that hires mostly ASD/asperger's QA engineers and contracts to do QA work.