QA Testing Bugs [merged similar topics]

Post » Mon Jun 11, 2012 4:07 pm

How many QA's worked on this?

How many copies of Skyrim sold?

There's your answer.
User avatar
Astargoth Rockin' Design
 
Posts: 3450
Joined: Mon Apr 02, 2007 2:51 pm

Post » Mon Jun 11, 2012 4:53 pm

How many QA's? Three. One was going through a divorce, dealing with his mother dying, got diagnosed with a terminal illness, fought off the legions of hell, and was otherwise pretty distracted. The other guy was just lazy and spent most of his work day at the local mall playing arcade games.

That's alot of work piled onto that one remaining guy, cut him some slack.
User avatar
scorpion972
 
Posts: 3515
Joined: Fri Mar 16, 2007 11:20 am

Post » Mon Jun 11, 2012 6:40 pm

My theory is, no matter what the problem is in any game the maker CAN and SHOULD fix any problem that a paying customer has with a game. Bethesda appears to just stop caring at some point.

It's not that simple. Bug has to be reproductible. In a game like Skyrim it can often be a problem. The fix must not break anything else.
There will probably a lot of bug that will never get fixed simply because developers could not reproduce/understand it.
It's not like they have level 100 instant fix perk =).
User avatar
Kevan Olson
 
Posts: 3402
Joined: Tue Oct 16, 2007 1:09 am

Post » Mon Jun 11, 2012 1:15 pm

No matter how big a game is, there is absolutely no excuse for a bug that ruins half the main quest to be present at release. What's more if said main quest is shorter than the average monopoly game and as linear as it can possibly be, and if said bug takes about 30s to fix, once the player has gone through a ton of forum posts to understand what is broken and what they must do...
The only way for QA to miss that is for QA to simply NOT EXIST... there is no other conceivable way, considering that the fix is so easy that even the laziest person would fix it on spot just to get it out of the way...
User avatar
Silencio
 
Posts: 3442
Joined: Sun Mar 18, 2007 11:30 pm

Post » Mon Jun 11, 2012 2:35 pm

Why does there need to be a standard? Bethesda should be striving to release games with absolutely 0 bugs, not games with a max of 250 or so.

A runner shouldn't be striving to run the 100 meter faster than most people. He should strive to do it in EXACTLY 5 seconds.

See what I did there? Ask them the impossible. I know there's no literal "excuse" for making a game with bugs, but if you know even the slightest bit about programming, you know it's frickin IMPOSSIBLE

Hell, just look at Windows! They have been working on the same damn core for around 15 years (don't know the exact number), and yet they STILL have bugs and crashes


That said, having witnessed the PS3 version of my friend, I do agree that they simply should not have made one on PS3. Holy [censored] how buggy can a game get?
User avatar
Julia Schwalbe
 
Posts: 3557
Joined: Wed Apr 11, 2007 3:02 pm

Post » Mon Jun 11, 2012 2:20 pm

Wth would tax records have to do with it? You have no idea how their QA department is structured, whether QA are even listed as employees, temps, whatever, do you?

The idea that you would somehow discover Bethesda's dirty little secret of no or badQA by looking at their tax records is ridiculous, furthermore, it would do you no good at all as you legally have no case whatsoever for action. Consumer protection laws are a great thing, but don't exist to be abused by entitled gamers...who have gotten a huge amount of valid use out of a product, but still feel they are 'owed' something.
Your right, I'm wrong. As a disabled, former Federal "employee", I over-stepped my bounds. I was hoping someone in this forum is a fan that could advise if I should, or should not invest with this company. It seems with the amount of dissatisfaction I am reading here, the choice is quite clear. Thank you for your input.
User avatar
dell
 
Posts: 3452
Joined: Sat Mar 24, 2007 2:58 am

Post » Mon Jun 11, 2012 2:28 pm

Your right, I'm wrong. As a disabled, former Federal "employee", I over-stepped my bounds. I was hoping someone in this forum is a fan that could advise if I should, or should not invest with this company. It seems with the amount of dissatisfaction I am reading here, the choice is quite clear. Thank you for your input.

Using company Q&A practices as a metric to determine the profitability of a company is ... kinda bizarre if you don't mind me saying. :shrug:
User avatar
Emmanuel Morales
 
Posts: 3433
Joined: Sat Oct 06, 2007 2:03 pm

Post » Mon Jun 11, 2012 12:12 pm

Good job. You really opened America and invented a wheel with that idea.

Thanx, I figure I had to come up with something until I caught a break in the Sunday funnies.
User avatar
Glu Glu
 
Posts: 3352
Joined: Sun Apr 01, 2007 5:39 am

Post » Mon Jun 11, 2012 7:33 pm

you know what...just read through this thread....you're all full of [censored].
pathetic
User avatar
tiffany Royal
 
Posts: 3340
Joined: Mon Dec 25, 2006 1:48 pm

Post » Mon Jun 11, 2012 1:07 pm

You know... I played FO:3 and FO:NV several times each... and didn't really have all that many glitches. This game, however, is buuuuuuusted. Or maybe I just don't remember the glitches?
User avatar
jenny goodwin
 
Posts: 3461
Joined: Wed Sep 13, 2006 4:57 am

Post » Mon Jun 11, 2012 10:38 pm

It's one thing to QA test a 4-5 hour COD game thousands of times.

Quite another matter to do the same with a massive open world RPG that provides several hundred hours of content. Bethesda could have hired a QA team of hundreds and had them testing nonstop for years and still never found every single bug. Within the context of huge open world sandbox RPGs, this is hands down one of the least buggy launches IMO.

Keep in mind, I'm not a Bethesda employee and this is my personal opinion.

Agreed completely. I have played over 100 hours, lvl 61 Nord warrior and i have'nt had a single game breaking bug. No crashes to desktop either, i am not saying i did'nt find any bug at all, i did like bodies falling from the sky or npc's spawning in wrong places but there was nothing game breaking.

And remember, its not Dragon Age either which pin point you to a straight path in prebuilt cells, its open world RPG. You can go anywhere you want, you can take any path you want and ofcourse game of this size is bound to have bugs. Trust me if they released the game 1 year later it would still have bugs.

Hats off to bethesda, you have done an excellent job and my thanks for that. Thumbs up.
User avatar
Etta Hargrave
 
Posts: 3452
Joined: Fri Sep 01, 2006 1:27 am

Post » Mon Jun 11, 2012 3:01 pm

Using company Q&A practices as a metric to determine the profitability of a company is ... kinda bizarre if you don't mind me saying. :shrug:
Not at all. I'm just being utterly witless. I could care less what's on ZeniMaxs plate, or Lynda Caters husbands award shelf....pun intended. I mean no financial disrespect to Bethesdas financial/ profit spread. I think it's obvious, if it's factual, that if Bethesda has enough money for QA unprofessionalism, while charging us top dollar for it's...dare I say "products"; then P.T. Barnum was right, for I, as well as many other RPG gamers am a svcker. Anyway, they're millionaires, and I'm a disabled schmuck.
User avatar
Harry Leon
 
Posts: 3381
Joined: Tue Jun 12, 2007 3:53 am

Post » Mon Jun 11, 2012 2:04 pm

It's okay.

Companies like EA salivate at this kind of Market. Bethesda is completely dependent on this particular niche market for profit. EA acquired Pandemic Studios (which had an open world engine that worked decently) and Bioware (Mass Effect) not too long ago. It was about the same time they were working on The White Council, which was also supposed to be an open world RPG.

They may still be doing so, despite reports that they cancelled that project.

EA is a predatory company. They already have an open world engine. Several of them, actually.

EA's corporate model is built on exactly this kind of circumstance: A company has a niche market or idea, so EA competes with that market or bullies the competition out, takes over, and they then have a death grip on that market.

This is exactly the kind of situation EA looks for, and no I'm not simply speculating.



My point is that if Bethesda doesn't learn to put some more of that massive revenue percentage into taking care of its users better, I wouldn't be shocked if they were gone from the world entirely within a couple of years.

There are companies out there right now that are looking at Bethesda, Skyrim, and this niche market that's being served up to them on a silver platter; a gift on par with a voluntary sacrificial meal...

And they're salivating.

My wife is a dedicated Sim player - does not play any other computer games.

Belief me when I say this, people think Bethesda is bad putting out patches, or no patches. EA is a hundred times worse.

To give you an idea - thousands of Sim players reported that their families go invisible, and then completely disappear - never to be seen again.

It just started happening to my wife, and after doing some research, we discovered this has been going on for a little over two years. EA won't even acknowledge there is a problem, let alone fix it. And their favorite mantra - It's your computer, not the game. Uninstall and reinstall the game and all will be well. But, it keeps costs down, and profits high.
User avatar
Shelby Huffman
 
Posts: 3454
Joined: Wed Aug 08, 2007 11:06 am

Post » Mon Jun 11, 2012 11:11 pm

Do you guys realise how buggy this game would've been if there wasn't any QA at all?
User avatar
x_JeNnY_x
 
Posts: 3493
Joined: Wed Jul 05, 2006 3:52 pm

Post » Mon Jun 11, 2012 4:18 pm

$5 says that when they originally started the project they told the level modelers to just start modeling caves and such before they even got a storyline going haha
User avatar
Tinkerbells
 
Posts: 3432
Joined: Sat Jun 24, 2006 10:22 pm

Post » Mon Jun 11, 2012 8:53 am

Honestly movies, games, soundtracks, they ALL have to have deadlines. Ask anyone who ever made one, they feel like if they had 5 more minutes their art could have been better but not everyone can afford to be Valve or Blizzard. For every Starcraft 2 and Half-life 2, you have projects like Chinese Democracy, Daikatana, Heavens Gate and other bloated entertainment products that went WAY over budget and timeline and ended up flopping. Everything like Skyrim needs a deadline for it to be completed at all, and it certainly wasn't "rushed'. Years of work isnt "rushed"
User avatar
Mrs Pooh
 
Posts: 3340
Joined: Wed Oct 24, 2007 7:30 pm

Post » Mon Jun 11, 2012 11:15 pm

Everything like Skyrim needs a deadline for it to be completed at all, and it certainly wasn't "rushed'. Years of work isnt "rushed"

Apparently it wasn't tested very well either.
User avatar
Ezekiel Macallister
 
Posts: 3493
Joined: Fri Jun 22, 2007 12:08 pm

Post » Mon Jun 11, 2012 4:05 pm

i judge QA testing based on how long it takes players to find issues. withing day one people had already discovered the 5th grade math used to develop the enchanting/armor/alchemy system which let people make hugely overpowered items..........that is a major fail since it was spotted by average gamer immediately. a couple of days later and you saw people posting about how destruction magic was useless on master difficulty at higher levels unless you chugged potions for every fight. i myself had this issue though thankfully there are mods that address this. both of those were obvious and very easy to find. so either they have complete idiots for QA testers or the testers reported the issues but nothing was done about it. either way it makes bethesda look very ameteurish.

another thing people noticed that should have been spotted were arrows/ash piles not despawning and enemies not respawning if items are left lying around. that is a nothing thing that should have been spotted by the testers.

the only thing i give developers a pass on is the occasional floating item or something like that. there is no excuse for anything else slipping by.

one thing i would like to see is Valve start up a bug testing company. they arent making any games anyways so why not. :dry: valve would never have let shoddy crap like this get through their games.
User avatar
Jason White
 
Posts: 3531
Joined: Fri Jul 27, 2007 12:54 pm

Post » Mon Jun 11, 2012 11:03 pm

Since I was told to take it here for some reason (it's a completely different topic, but I guess the moderators don't want bad press on the main boards)

Taken from http://www.reddit.com/r/skyrim/comments/o6ivz/bethesda_qa_and_respect_for_gamers/


http://www.reddit.com/r/skyrim/comments/o6ivz/bethesda_qa_and_respect_for_gamers/

I am the author of small, but almost necessary mod for Skyrim -- Weapons and Armor Fixes (or WAF, which you can find http://www.skyrimnexus.com/downloads/file.php?id=4719). I'm not writing this reddit because I want to pimp out my mod, in fact I suspect most of you won't care for it and would rather wait for the https://unofficialskyrimpatch.16bugs.com/projects/7078/bugs, from the same guys that made the Unofficial Oblivion Patch and the Unofficial Morrowind Patch (both of which literally fix thousands of bugs beyond the last official patch for these games), or are console gamers, in which case you don't have access to mods. No, what concerns me is why it's even necessary to have such a mod.

As of version 0.94 of WAF, 177 Weapons out of 2,848, and 545 armors out of 2,762 are affected. This is roughly 6.2% of all weapons, and 19.8% of all armors. While not every of those are problematic (see first bullet of section 4 on that first link above), most are. Having such a large number of weapons and armor affected speaks volume about the quality of the bug testing/QA going on at Bethesda, or how much they really care about putting out a polished product.

At the very least, 85%+ of these problems can be spotted simply by sorting the items according to various properties (weight, name, weapon type, material, ...) and play a little game of "spot the intruder". These are not the big "OMG, this game is huge and complex, so it's normal to have a few things slip through the cracks" stuff. This is basic Q&A stuff, requiring no more than a few hours' time from some minimum wage intern. I takes me more or less 5 hours to do the whole thing from scratch, with hacks, 3rd party tools (which are very early in development), and zero documentation. And I never had any modding experience prior to Skyrim.

We on the PC are very lucky that Bethesda gives us the privilege of doing their job and uncrapping their game. I can't imagine how users playing on the Xbox 360 or the PS3 must feel (http://www.youtube.com/watch?v=cUoDfO1pWto). Don't get me wrong, Skyrim is a good game, but shame on Bethesda for releasing it in such a poor state. And shame on the gaming press for awarding early beta releases >90% scores (the PS3 version has 92% FFS!), and several GOTY awards.

We as loyal gamers and fans deserve better. Bethesda should look at studios like Eidos Montreal for listening to their fans (who gave us features like the ability to turn of augmented vision in Deus Ex Human Revolution, even if they themselves felt it added to the game, because they RESPECTED their fans enough to not pretend to know what we wanted better than we did), and care for them equally regardless of platform, giving us both a true controller interface, and a true KB+M interface (not to mention proper graphics and basic options such as the ability to key binding controls out of the box for PC users). Or studios like CD Projekt Red for their beyond-phenomenal post-release support of their games like The Witcher.

It is our responsibility as gamers to let Bethesda know we deserve better. Because it's not our gaming press that will do it, for fear of being cut off the money train of early access and exclusive interviews. Nor will it be Spike TV, who's opinion Bethesda seems to consider more important than ours. Nor will it be their focus groups, because they clearly are disconnected from our needs and wants.

So tweet, write letters, blog, reddit, ... anything! But don't sit on your chair pretending it's OK for Bethesda to ask 60$ for a beta-release, or for the gaming press to give them GOTY/SOTY awards. Otherwise, it's just going to be one painful BOHICA for TES VI.

-- Headbomb
User avatar
Niisha
 
Posts: 3393
Joined: Fri Sep 15, 2006 2:54 am

Post » Mon Jun 11, 2012 10:28 am

In the case of Skyrim, I experienced about 10 CTD in total prior to the Steam LAA update and absolutely no crashes following this, over the course of playing the game 350 hours in total.

350 hours on one character or spread out over several I wonder?
On PC anyways, 350 hours on one character and still managing to maintain a playable game without permanent-CTD-areas is impressive! Save file size?
User avatar
Soph
 
Posts: 3499
Joined: Fri Oct 13, 2006 8:24 am

Post » Mon Jun 11, 2012 2:19 pm



350 hours on one character or spread out over several I wonder?
On PC anyways, 350 hours and still managing to maintain a playable game without permanent-CTD-areas is impressive! Save file size?

200 hours on the first character (level 45, savegame size around 14MB), about 150 hours on the second (level 30, save file around 11-12MB) and another couple of hours playing through Helgen with 2 other characters.

Haven't encountered any reproducible CTDs and after Patch 1.3 and the Steam LAA update, I have been playing quite a lot with zero CTD and zero purple textures. I heard that some had difficulty with the Bonechill Passage area. I haven't even found this location yet, so I haven't tried it.
User avatar
Jeff Turner
 
Posts: 3458
Joined: Tue Sep 04, 2007 5:35 pm

Post » Mon Jun 11, 2012 9:01 am

That is ridiculous, seriously one of the most ridiculous things i've read, much less about a project the size of Skyrim. There is literally probably NOTHING in existence on PC minus minesweeper that has 0 bugs...man, you just made me shake my head even more.

It amazes me that peoples level of expectation/customer satisfaction is supposed be be different (ie much lower) when it comes to computer software. Being apathethic only results in companies raising the bar higher. My experience with this particular one is that they have a really poor track record with releasing workable "out of the box" titles and providing after sales support to remedy the multitude of problems.

Here at least there is a legal obligation for any company that produces a product to conform to/implement a certain level of QUALITY ASSURANCE so that the said product is fit for use and operates as intended. Not that I've looked but I'm sure in some sort of policy statement Beth would be obliged to stipulate something along these lines. If you've ever sat through a corporate induction and listened to the company spiel then you would know what I'm talking about. This protection even applies to any product sold locally that has been manufactured overseas.
User avatar
luis dejesus
 
Posts: 3451
Joined: Sun Aug 19, 2007 7:40 am

Post » Tue Jun 12, 2012 1:02 am

I keep hearing over and over again that Skyrim is huge (it's not,) and no one should expect that all the glitches should be caught. I agree, but we're NOT talking about ALL the glitches being caught. People are asking why MOST of the glitches weren't caught. I had dozens of glitches within the first 20-40hrs of game play, dozens. It didn't stop me from playing because most of the glitches were things like weapon racks not working, and Lydia standing up and sitting down and standing up and sitting down ect. As I continued to play the glitches just keep piling up and also becoming more severe. I started a second character but the love was gone man. Why start over just to arrive back at the same place. So, now Skyrim sits on the shelf with Fallout New Vegas (yes I've heard all about how it was Obsidian's fault and not Bethesda's but I didn't buy it because Obsidian's name was on the box. I bought it because Bethesda's name was on it.) I really hope they fix it. I really do! I had fun, alot of fun. But now I'm having fun NOT playing it. I like being able to put in a game and play for a couple of hours without having to reset the xbox a half-dozen times. I'd like it better if it was Skyrim, but it's not. It's the competions games I'm playing at the moment. Are they as fun? You bet! You know why? Because I'm not getting up every 30seconds to 60minutes reseting my Xbox from the console because the game hard crashes and won't recognize the controller. Because I don't have to sift through 140+pages of glitches to figure out which quests are too chancey to begin. Because most other companies aren't releasing unfinished games because someone got a hard-on for the date of 11/11/11 (BTW: I woulda waited a couple of extra months for a finished game.)

I learnt my lesson with Fallout NV (Not their fault, uh huh, totally believe you,) I bought all the DLC thinking, "well if they're still releasing things for it, it must mean they're going to fix it." I still can't walk through an EB Games or Futureshop without spitting on every copy of that game I see. I'm not falling for it this time. If they fix it I'll eat my words, but you and I both know the ole Crybaby here is gonna go hungry. That's my prophecy, say what you want but only time will tell if I'm preachin' or whining. Hope they fix it before it's too late.
User avatar
Kelly James
 
Posts: 3266
Joined: Wed Oct 04, 2006 7:33 pm

Post » Mon Jun 11, 2012 10:03 am

Lvl. 80+, 557.+ Hrs., First Play through, 360.

After reading how many QA's " worked" on Skyrim, or any other Bethesda RPG; I haved developed a theory on their business ethic... Sell Now, Fix Later. Either the QA's are blowing off, what is instructed, or not instructed as "minor" issuses, to be fixed after the games release, OR, the QA's are reporting all the defects they find to their managers, and it is they who allow the game to be released to the public.

Obviously I've put some hours into this, and let me tell ya, I'm tripping over all sorts of bugs, glitches, and defects, and not even going out of my way to do it!!!!! I haven't even started on the PC yet.

I really like Bethesdas game lines, but come on now!!! Maybe some of their staff don't appretiate the seriousness of their jobs, or the economy as it is; and it's those workers that are bringing down the honor of the harder working staff! I'm surprised they're putting up with it.

I wonder when their last Federal Audit was? Hmmmm.

I completely agree with your theory pertaining to Bethesda's work ethic being sell now, fix later. Mostly quest- related issues and strange interactions w/ npcs. They just seem to have an unusual amount of strange glitches like dead people showing up to your wedding...
User avatar
Anna Beattie
 
Posts: 3512
Joined: Sat Nov 11, 2006 4:59 am

Post » Mon Jun 11, 2012 11:13 am

I'm curious how testing was done for PC. I do not yet own the game, but I do have intentions of buying it in the later half of this year. I've been keeping up with the complaints, praises, and problems that the community has had. Reading through many threads, I've started a list of questions for which I'd love to know the answers.

Is all the testing done in house?
Do the same people that test the Xbox/Playstation versions do the PC testing?
If so, is this a group size of 10 people? 100 people?
Are the people performing the testing familiar with how other RPG games work? i.e. has any of them ever played Morrowind? Gothic? The Witcher?
If an outside beta test group is used, what is that group size? What are their gaming backgrounds?
Do the people doing the game development do any thorough in game testing?

Note that I would be more interested in legitimate answers more than speculation, though all comments are welcome.
User avatar
Bedford White
 
Posts: 3307
Joined: Tue Jun 12, 2007 2:09 am

PreviousNext

Return to V - Skyrim