Jump to content

About recent community criticism in the direction of the QA & exp testers


KasperVld

Recommended Posts

I feel that in-house testing can only do so much. There's no substitute for genuine mass-market feedback. That's how updates work; testing, release, feedback, and a release to fix the bugs in the first release.

I'm not angry. I'm just waiting for the majority of them to be caught, while I wring the last few hours of fun out of GTA5.

Link to comment
Share on other sites

Well, I do not want to harp much, but, as someone that was already once in this kind of show, I can say that, for whatever issues KSP 1.0 has, the QA people is most likely not to blame. First , because how QA works and second, because the ultimate responsibility of any code falls in top of the developer team. It is not QA that codes or even that decides what is in game or not or what bugs are worth to lose time fixing ( or even if they are fixable at all ) ...

Link to comment
Share on other sites

I agree that blaming the testers themselves isn't fair.

One thing I'm curious about is sort of a dissection of how an individual bug makes it into the release. One I'm particularly interested in is the heatshield causing capsules to flip on reentry. How does that happen?

Link to comment
Share on other sites

I agree that the negativity towards testers isnt really called for, but then again, i do somewhat understand that a portion of teh community will go all out crazy over bugs.

I think the 1.0 rage (if you can call it that) is mostly caused by the players that insist the game wasnt ready to leave beta, and myself i am in that crowd, but im actually very pleased with what 1.0 ended up to be, i didnt quite expect as many features to be in the game. Personally most of the 1.0 features are stuff that is what i consider extra ontop of a already good game. the only features that i really cared for and wouldnt have been happy without are the aero improvements, as now we get (minus some bugs) a MUCH more plausible model, where stuff makes logical sense. The only other features were the speed ups, optimizations, and mem leak fixes that made KSP annoying and unplayeable over longer times. This was done well, and while there is still plenty that could be optimized (for example load textures on demand instead of on game start), its still a massive improvement over the old versions.

That said, there are many bugs that did get into the release, many of which are such outright exploits its not even funny, such as what is below.

Javascript is disabled. View full album

While im sure this is an oversight (and im guessing the devs never intended for people to build craft this way in the 1st place), the occlusion system for drag/heating is completely broken right now. I understand its not possible to catch every bug out there, but im guessing stuff liek this really should have been patched out before release.

That said, the issue with testing in my opinion is that its hard to imagine all the possible situations without going into a very wide variety of players. For example i tend to abuse part clipping ALOT, critisisms about my playstyle aside, this has shown me many many bugs in the new occlusion system, and many of the corners that are cut for it end up creating exploits. You gusy need to create a WAY larger experimentals team, im not saying open it up to EVERYONE, but perhaps include a few more players, and try to get a wide variety of construction styles in too. Some build rockets, some make real life replicas, some make whackjob sci-fi craft, and some just strap a booster to a pod and call it a day. While i dont know who is on the experimentals team, nor how they play the game, i feel they could use a larger subset of the community, especially a few players that push the boundaries of the game and dont make classical rocket designs. These are the people that are more likely to catch bugs, as the game seems to be designed around classical stuff, classical rocket, planes, ect, and the whackjob builders can actually catch alot of bugs (for example the one i found within 2 days of testing).

Still, while i might sound a little critical, im sure everyone has their ups and downs (i know many people are mad that craft broke, that jets nolonger go to LKO alone, ect), but overall ill say its a very good upgrade to an already awesome game. Just consider adding more testers so we can have more likelyhood of catching bugs.

Link to comment
Share on other sites

So its very easy to jump on the bandwagon for 1 or 2 issues that are the hot topics here on the forums and generate some indignant snark about how it should have been fixed. But takes those 1 or 2 issues and multiply them by a hundred and you're closer to the amount of things the testers and devs sorted through since testing was announced. There will never be a perfect release, and thats a fact.

While your post is informative and all, there's one simple problem (and that's a fact). Bugfinding and fixing was not even done by the time the 48h stream was going, see the bug scott manley ran into live on stream when he lost aerodynamics on his surfboard, when he closed the cargo bays on the launching vehicle again. That's something that i would've absolutely understood if overseen. Yet, that was actually fixed, so apparently that was more important than other issues. Now one can categorize bugs (and of course, you guys do that too) - one hand has the "i'm dicking around with stupid things, somethings not working there" bug, and the other has glaring issues with the core of the game, namely reentry/parachutes/aerodynamics. See the problem?

Apart from that, the whole time, people are trying to sell the Q&A team as an entity completely unrelated to the development, that is just plain wrong. Of course the Q&A is part of the development, one of the most important actually. I know quite a few games that got their releases delayed because of a Q&A team that said it isn't ready (one will release in a bit less than 10 days, finally TT), and that's what should've happened here too - but by the looks and sound, the "rushed feel" of things in the game might have to do with the fact that the release was indeed rushed for some reason.

Link to comment
Share on other sites

My complaints, as they have been for months, all boil down to the insanity of putting so many changes into a "final" release. There was no way this was going to work cleanly, regardless of the testing team. It's miraculous that there's so few actual game-breaking bugs (not none, see scanner-root-part, but few), but that is not an excuse. Doing a - not good - but passable job under pressure can be admirable... but not when that pressure is entirely artificial and self-imposed.

"How did the testers miss this" is answered nicely by Tiberion above.

It's also entirely possible we didn't "miss" something you think we did. Maybe we saw the issue and decided it was unworthy for now.

I find it hard to believe that they could miss the state of parachutes, farings, re-entry, and occlusion heating. I think better of our testing team than that. That those issues got pushed into being low-priority, despite being obvious giant flashing "work-in-progress" signs over the game, is a perfect illustration of what many people have been saying since 1.0 was announced. It's too much at once, too soon.

TL: DR, Kasper and Ted riding to the defense of their testing team reeks of intentional deflection of criticism.

C'mon, guys. We love your game. If we didn't, we wouldn't care about the state of it at release.

Link to comment
Share on other sites

I doubt the testers are to blame. What needs to be adjusted is the process.

  • Experimentals lasted for... two weeks? Tops? Adjust time for logistical operations and you have even less than that.
  • I don't have the numbers but I doubt there are hundreds of testers. Probably not even dozens
  • Testing can be an arduous process, especially to isolate the exact steps needed to replicate a certain problem

With all these factors together one can expect that not everything can be tested. Likely, testers will focus on "what's new" and not waste time on "what already works." Of course this is purely speculation, but it explains why we are where we are: a 1.0 version with bugs in trivial places that make everyone wonder why did this make it through testing?

Without wanting to beat the dead horse all over again: Squad has a legion of thousands of fanatical testers called "the user community." It's not the first time that a release is followed within a week with a patch based on "community feedback." Acknowledging that "the mob" can find bugs in two days the testers cannot possible find in two weeks would be a start. The decision to go straight to 1.0 and not issue an in between 0.9x version has been questioned before the release, and here we are, a 1.0 laden with bugs.

Sadly, the testers get blamed for Squad's rushed schedule.

Link to comment
Share on other sites

Don't forget that on the internet, a person might mean to be entirely polite but because they are [constructively] criticizing you it can sound harsher than it is. That's why it can be worth it to take a few extra minutes to be verbose and make it clear that you appreciate the people you're criticizing and the work they do; you just want to help.

The bugs with heatshields and parachutes in 1.0 are puzzling indeed, and it's hard to blame people for asking questions about the choices Squad made with the development and QA process. Of course that doesn't excuse rudeness towards the testers, for reasons pointed out by other posters.

Link to comment
Share on other sites

C'mon, guys. We love your game. If we didn't, we wouldn't care about the state of it at release.

Yes. Very much this. And 1.0 is on helluva update as well.

But still, someone has blundered. One cannot possibly believe the testers failed to notice the pod woes, or the chute thing. It's also hard to imagine that their Nervas didn't blow up. But either they didn't report this (why?) or their reports went unheard.

BTW, how can I donate an "a" to that poor "Nerv" engine? That thing's been called Nerva since before I even heard about, much less played KSP.

Link to comment
Share on other sites

I've played computer games for 35 years (yes we had to type the code ourself from pages of Byte back then).

And of all the 1.0 games that I bought over the years, KSP 1.0 is one of the most polished and functional ones.

I'm sure that Squad will provide a 1.0.1 (as soon as they sobered up enough), and I'm sure that it'll be better than most other games panic patches.

Link to comment
Share on other sites

I can say one thing that upon release, it is far more stable than when The Elder Scrolls: Oblivion came out. While that game was fun, I remember the first release having so many bugs that it in fact broke portions of the game. This wasn't from any indie company either.

And while The Elder Scrolls: Daggerfall was a great game; there were still bugs that were never fixed even towards the end.

Link to comment
Share on other sites

As someone who has worked in QA, I can say that it's nearly impossible to catch every bug that could happen. QA's only have so much time, and often (at least for the company I worked for; Squad may be different) they have very specific areas they are testing, and time restraints may cause some areas to be gone over less thoroughly than others. Unless they hire a thousand QA specialists, some stuff will fall through the cracks. It's up to us as the community to find what they missed and inform Sqaud in an appropriate manner.

That said, the 1.0 release is very stable for what it is. I've only encountered a few minor bugs myself, but really not to react about.

Link to comment
Share on other sites

Yes, people have been completely ridiculous with their criticism of the QA team. Constructive criticism is one thing, but some people have been downright hostile.

Some people are complaining that the reason some bugs made it through is because of the almost complete lack of a beta test. That is a fair assessment: The point of a beta test is to get the (almost) finished product out to a bunch of people so it can be thoroughly tested before calling it complete, and Squad didn't do that with 1.0.

However, the problem of an early access game is all of the hype. Squad had to find a balance between releasing a perfect game, and releasing a game with a bunch of brand new features to get people hyped. Like it or not, they have to make money off of this game, so they'll need some hype behind the release to increase sales. Now, you could easily argue that they found the wrong balance and should have been a bit more careful at the cost of a bit less hype. But that's really just a matter of opinion. In the end, all it comes down to is semantics: No matter what the version is called (beta vs. release), it'll still end up in the same situation. So Squad could've released this version as 0.99, the final beta, and then this next patch would be 1.0, but all in all, the game would end up in the same state.

Link to comment
Share on other sites

... Of course this is purely speculation, but ...

You are indeed speculating and you are wrong on a lot of it. I also object to the statement that 1.0 is "laden" with bugs - having tested MANY versions of KSP this one is not in any way abnormal; several releases have included issues far far worse than anything you're experiencing now.

I posted in the public feedback thread that I didn't think this release should be labeled 1.0 because there were a lot of changes and things to test. I always always want more time (and I would be a terrible QA manager because of that) I haven't really changed my mind, but in the end the only thing different from all the other releases is a special version number, this release is not especially bad in any way and things are being greatly overstated to fit in the narrative.

More straight talk: Something required this specific launch schedule to happen, the reasons are not known, not even to testers and may never be. It doesn't matter now, move on.

(end of part directed at Kerbart)

I'll restate it again: what people consider very important and glaringly obvious inside the echo chamber of the forums is completely different than what we're dealing with after a couple of weeks and hundreds of instances of feedback being logged. It's your OPINION that we should have changed 're-entry/parachutes/aero' instead of ****ing around, but we *DID* submit bugs and feedback on those multiple times, and they were tweaked and changed over many builds as well. So the end results may not be perfect or what you want; time to deal with it and offer proper feedback for future versions rather than assigning blame or whatever.

Also, without getting too much into how the 'sausage' is made - we tested through the weekend, but there is always a point in testing where things begin to get locked down, major changes won't happen unless critical, so that the last period of time is spent just making sure none of those critical bugs make it in. (one or two usually do anyway though)

And something I didn't mention earlier; a very important part of testing is confirming that an issue that was published in a new build was fixed and that no new issues are caused as a result. This is a criticial step and also one where a lot of "whoops, how did that happen" comes from.

I can't and won't explain individual issues (not my job) but its very likely that "unintended consequence" is the reason for most of them. Hundreds of issues, dozens of builds, lots to retest - you do the math :)

Also re: adding more testers. Probably would not have moved the needle on the amount of bugs that were addressed, the more people you add, the more feedback there is to sort through and prioritize. There's a "sweet spot" where you have the right amount of people on each specific team and I think ted has it just about right.

(Final note: Q&A means Question and Answer - We're talking about Quality Assurance - just 'QA')

Link to comment
Share on other sites

You misunderstood what i said. I didn't say "you should've changed stuff instead of dicking around", i said it's important to prioritize bugs that touch the core of the game instead of bugs that you only can reproduce by doing "weird things". I will never launch a surfboard from a flying plane, but reentry etc will happen for me quite often. And that goes for alot of players (not all, obviously, but alot).

Take that as a friendly tip, nobody is out "to get you", i said already that i don't want to offend the QA (point taken on that one) team, so there's no reason to get defensive and read stuff into things.

Link to comment
Share on other sites

You aren't hearing what I am saying...

It wasn't "Hey, re-entry is broken, but we're going to fix this surfboard thing instead"

It was "A lot of the feedback on re-entry has been addressed and tested, with multiple builds changing it and it seems to be in a good place for now. Separately, this major issue with the cargobay system comes to light and one of the devs does the work to get it fixed. Also separately, a narrow case of re-entry mechanics was at some point broken by a change and not properly found or fixed."

I promise you a lot of thought goes into bug prioritizing. The testers have guidelines on how to log their tracker issues with the right priority and the devs also change them based on their own criteria or knowledge of future development. The issue you think exists does not.

And I am not being defensive, I am trying to explain how it all works, how the work flows, and how to properly judge how important an issue is or is not, to help everyone move on.

Being defensive would be valid though because there HAVE been multiple posts where people said the testers failed or were unprepared, and there continues to be posts leaving negative feedback about the develop and testing process rather than on the actual game. Coming back later and saying "Oh but we don't want to offend you" doesn't really change that. But I am not writing any of this to save feelings, but rather because I want to help everyone transition to "a better way" of handling feedback

Feedback on the development and testing process isn't really helpful because you are largely not "in the know" about what is happening (by design) As such its really only feedback about how "it looks" or "how I think you should do things" which are valid opinions for you to have, but they don't really improve the game.

Again, focus on actual feedback about the game and specific issues and less about why issues exist and who is to blame. Everyone will be happier for it.

Edited by Tiberion
Link to comment
Share on other sites

-- Stuff about QA and Experimentals Processes and Procedures --

Tiberion I just wanted to post to again thank you for the work you and all the rest of the QA / Experimentals teams have done for KSP and all us Kerbonauts.

Secondly, I want to thank you for posting a bit of the "process" behind what you guys do in QA/Experimentals. I think a major part of the "misunderstandings" that are happening in this thread and throughout the forums are due to a complete and lack of understanding of what a QA/Experimentals tester goes through and deals with throughout the final (literal) SPRINT to the finish line. Thanks for giving us some insight into the process.

Link to comment
Share on other sites

I promise you a lot of thought goes into bug prioritizing. The testers have guidelines on how to log their tracker issues with the right priority and the devs also change them based on their own criteria or knowledge of future development. The issue you think exists does not.

Then explain to me please how certain things simply made it through. We're talking things that actually make one of the new additions (re-entry heating, feature) completely obsolete. It's not just an issue, the parachutes simply remove the need for re-entry precautions. I actually don't get it then, how can "an issue" that's basically removing one of the new big features be okay to go live? Since i feel the need yet again, it's not meant as an offense - i'm actually interested in that process.

And I am not being defensive, I am trying to explain how it all works, how the work flows, and how to properly judge how important an issue is or is not, to help everyone move on.

Being defensive would be valid though because there HAVE been multiple posts where people said the testers failed or were unprepared, and there continues to be posts leaving negative feedback about the develop and testing process rather than on the actual game. Coming back later and saying "Oh but we don't want to offend you" doesn't really change that. But I am not writing any of this to save feelings, but rather because I want to help everyone transition to "a better way" of handling feedback

No. That's simply not the case. Even if there's some idiots attacking testers, there's no reason to assume all of them are. By that logic, earlier in this thread one of the QA guys said this release he was lazy - should i imprint that on all of you guys as well? I assume we agree there that blanketstatements like that are not helping either your nor my case. The "i don't mean to offend" btw gets added by me because you come off defensive to me. I never said anything insulting or offending in the first place - apart from if you see "critique" as offense, which it seems to.

Feedback on the development and testing process isn't really helpful because you are largely not "in the know" about what is happening (by design) As such its really only feedback about how "it looks" or "how I think you should do things" which are valid opinions for you to have, but they don't really improve the game.

Again, focus on actual feedback about the game and specific issues and less about why issues exist and who is to blame. Everyone will be happier for it.

I never commented on how you "should" do things, apart from less rushing. It's actually not my job. I can only "judge" the endresult of all of that.

As a sidenote: thank you for giving me some insight into the QA(!) process, it's actually quite interesting.

Now a question that you don't need to answer (well you don't need to answer to anything anyway, you know what i mean) - as a QA guy, do you think this release was rushed for "shrouded reasons"? Or do you think (objectively) that this was enough time to "get it done"?

edit: man i'm bad at quoting apparently :(

Link to comment
Share on other sites

Watching all this back and forth, the emotion ... this is why qa/testing of a multi-million dollar project should not be saddled upon volunteers. Dedicated testers should be hired for actual money. It is amazing how different things become once money is involved.

Link to comment
Share on other sites

Maybe it was a bit optimistic to go for 1.0 release with so many new features planned. Things like the new aero, overheating and resource harvesting might have better been put through Beta first. This community is used to being Beta testers after all. Given that we were left out of the loop on these, the natural reaction is to shift blame up the design chain. And that's not fair to the QA team. I'm sure you guys did great work and there's a ton of bugs we're not experiencing to outweigh the handful that we are. Kudos :)

In the same vein, I feel that the backlash was inevitable considering the lack of a true feature-complete beta, e.g. a 0.99 release before 1.0. Releasing 0.99 and insisting that there are surely still plenty of bugs before calling it 1.0 might have diminished the criticism. Avid, loyal players' criticism is the least of Squad's concerns at this point.

Link to comment
Share on other sites

I cannot and will not get into specifics on any one issue (parachutes, heat, pod balance et al) because that is only going to provoke more fingerpoinitng which is exactly what we do not want or need.

Consider this though: there are thousands of combinations of parts and one of the greatest difficulties in testing is checking all of those combinations for "edge cases" where doing certain things in an odd way reveals a flaw. And again, we do miss things. It's easy to say something is "obvious" and "completely breaks something" after you already know about it. When you've spent a few days testing both in various states its much less obvious. That will have to satisfy your curiosity for now, sorry.

To clarify; I am not a "QA" guy - they are members of Squad and actually work for a living. I am just a veteran member of the experimentals team, which is just volunteer community members selected to take part in the final stage of closed testing. As a tester I have no extra knowledge about the release schedule (seriously, this isn't a "I know but can't tell you" situation.) As a community member just like you, I think its pretty clear there was something that was driving the release schedule for 1.0, be it outside sources, business reasons, or just unspecified development concerns. IE: 1.0 was happening and there's no stopping it. It's time to accept that.

Do I think it might have been "better" for KSP to have another beta release? Yep, I said so several times in the past in he various posts about that subject.

Do I think this build was rushed? Not really, not any more than others have been. Other than a special version number this has been a fairly typical release and the world will keep on spinning, just as it has.

Again, I think that people keep talking like this build is deeply flawed because it fits a narrative better (and we all like saying "I was right!") but it's not really so severe.

I really don't think I have much more to say about this so I probably won't. I'd just urge everyone to turn the page, leave specific feedback if you don't think something is addressed, and then go back to playing the game. It's fun!

Back to editing configs for Novapunch for me :P

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...