About recent community criticism in the direction of the QA & exp testers

[KasperVld]

It’s often easy to talk to a community when there’s good news to share, a compliment to be given or a discussion to be had. This post is none of those things: these past few days I’ve seen a lot of negativity towards people who volunteer a lot of their free time and their skill to test each and every release of KSP before it sees the light of day to make sure you get the best experience out of it, and that negativity needs to stop.

The QA & Experimental testers are a big part of what made KSP what it is today, and it’s not fair to them to criticize them for not finding each and every bug, especially considering the scope of a game like Kerbal Space Program. If you need to vent I’d rather see you direct your anger at me or even the developers, but I will take a stand when the people who volunteer their time are unfairly criticized.

It really disappoints me that a few people in the community feel the need to behave this way, instead of focussing on how to improve the situation: report that bug, gather information about it, or even help the developers fix the problem. All those things help to resolve the issues that will unavoidably pop up sometimes, and are objectively a far better investments of your time.

[Ted]

Community response to issues within release builds of KSP has always been very much at either extreme, as far as I’ve seen. There’ll be those that shrug off the issues and get on with playing as - for the most part - the bugs that make it into release builds of KSP are pretty minor compared to the ones we see in testing. And then on the other hand, we tend to see a lot of players that view an arguably minor issue as absolutely and utterly game breaking. Now this isn’t an effort to trivialise issues that make it into a release build or trivialise the frustration that a player can feel when bugs happen. It’s more to put it into perspective and make it absolutely clear that the efforts made by the Experimental and QA Teams are invaluable and that together they crushed over 200 issues during the 1.0 testing period. Please do not criticise and belittle the tireless and fantastic work they put in, they are literally only trying to help, as I’m sure you are.

As Kasper has recommended, please pop up a bug report for the issue instead and help us mop up these slippery remaining issues!

Thanks.

[DuoDex]

As a (lazy) experimentals tester, I have to agree with Kaspy and Ted. I didn't do much during the 1.0 experimentals, but I saw a lot of people who spent hours every day looking for bugs. We do our best.

Edit, because this is being taken way out of context. First of all, I am not a member of the QA team, and my membership in the Experimentals team is at best debatable. I spent about an hour or an hour and a half every day in sandbox mode, doing what I would normally do in sandbox with the 1.0 features. Some members of the team spent four to five times that amount of time, testing ALL THE FEATURES. You all know who you are . The "lazy" quote is only in relation to some of the other testers.

Edited May 7, 2015 by DuoDex
more information

[m4inbrain]

It’s often easy to talk to a community when there’s good news to share, a compliment to be given or a discussion to be had. This post is none of those things: these past few days I’ve seen a lot of negativity towards people who volunteer a lot of their free time and their skill to test each and every release of KSP before it sees the light of day to make sure you get the best experience out of it, and that negativity needs to stop.

The QA & Experimental testers are a big part of what made KSP what it is today, and it’s not fair to them to criticize them for not finding each and every bug, especially considering the scope of a game like Kerbal Space Program. If you need to vent I’d rather see you direct your anger at me or even the developers, but I will take a stand when the people who volunteer their time are unfairly criticized.

It really disappoints me that a few people in the community feel the need to behave this way, instead of focussing on how to improve the situation: report that bug, gather information about it, or even help the developers fix the problem. All those things help to resolve the issues that will unavoidably pop up sometimes, and are objectively a far better investments of your time.

In germany we say "the tone makes the music". Criticism towards the Q&A team is pretty valid, there are some glaring issues which one can not not see (parachutes, prime example - and you have to use them as a tester as well). Apart from that, while i understand your position: it's equally "not fair" to assume that a customer (consider, there's people buying into KSP right now with the release of 1.0) has to somehow point out how to improve the situation. That's the job of the Q&A. It doesn't really matter if they get paid or not, to a customer. If somehow "paid Q&A" is different (since you pointed specifically the voluntary bit out), then it's on squad to rethink the situation.

But, as i said. The tone makes the music, but if something isn't correct or was done sloppy, it has to be pointed out as well. Constructive rather than offending, obviously - but nonetheless.

edit:

Please do not criticise and belittle the tireless and fantastic work they put in

Belittleing is misplaced, of course. Maybe it's just me, but i feel that even if you do decent work, if nobody points out your mistakes, you won't improve. Obviously nothing gamebreaking was overseen, but some pretty huge issues - you can't compliment those away. It actually has nothing to do with being grateful/ungrateful. Obviously i'm happy that the game has no gamebreaking bugs, that doesn't mean i have to be happy that there's issues.

Edited April 29, 2015 by m4inbrain

[Linear]

Honestly think this has been blown out of proportion. There has been criticism towards the QA/Experimental teams - but honestly, the amount of very serious bugs that have come out of this process is a little odd.

A lot of this criticism is needed I believe, however not necessarily in the team personnel, but in the process. The bugs seem to be encountered in general gameplay (heatshields, faring mass distribution, etc) play, are the testers perhaps focusing too much on breaking systems, rather than playing them and seeing what looks odd?

The frustration you mention is generally because you tell us along the way 'how hard everyone is working' and 'how well it's going' to the extent that people are expecting extremely high-quality releases and they don't feel they're getting them. There's also the point that you can't tell anyone that the people you have on the team are any better at finding bugs than a good chunk of the people who play this game. A lot of people have 750/1000+ hours logged on this game, and this will make them feel comfortable (perhaps too much so) when voicing their opinion.

You guys are doing good work, i just question whether an analysis of the types of bugs/problems/balancing we're seeing can help make the process better.

[Fel]

In germany we say "the tone makes the music". Criticism towards the Q&A team is pretty valid, there are some glaring issues which one can not not see (parachutes, prime example - and you have to use them as a tester as well). Apart from that, while i understand your position: it's equally "not fair" to assume that a customer (consider, there's people buying into KSP right now with the release of 1.0) has to somehow point out how to improve the situation. That's the job of the Q&A. It doesn't really matter if they get paid or not, to a customer. If somehow "paid Q&A" is different (since you pointed specifically the voluntary bit out), then it's on squad to rethink the situation.

But, as i said. The tone makes the music, but if something isn't correct or was done sloppy, it has to be pointed out as well. Constructive rather than offending, obviously - but nonetheless.

Colloquialisms don't cross the culture barrier very well.

That aside; many of the criticisms are subjective or place emphasis on small problems. It also excludes the possibility that squad "just screwed up" when packaging the release, something that is dead easy to do when working with many developers and don't go all "github" on me because even using diff files can change the same line twice, one the official way and the other a "tweak testing" way.

No game has ever released without bugs; even PONG may have a bug, given that variables are likely 8-bit, in that a score over 255 resets to 0 (or, if it is signed could cause a crash [i think signed aren't suppose to wrap but may be wrong]). Modern games have bugs, customers expect this, but this COMMUNITY has made it clear that any bug in a release is considered "unacceptable" as if between the plethora of mathematical models, a single line of code that causes unwanted behavior is utterly disastrous despite a major bug crushing overhaul. Really, I am appalled at how often this community lashed out with these false statements, we utterly bludgeoned SQUAD over theoretical game breaking bugs, and now are complaining about smaller bugs; making a big deal out of them because KSP is a "finished product."

Instead of focusing on what Experimentals MISSED, look at what they caught! Look at how much work SQUAD has put into fixing the plethora of bugs in KSP!

Edited April 29, 2015 by Fel

[JedTech]

I appreciate everyone's work (paid or not). 1.0 is a great release and the patch will make it even better!

That said, I think Squad should remember that they were the ones that reminded the community over and over again of how good the QA and Experimental team were. Much of the community desperately wanted a final beta release to test features before a 1.0 release. Squad reminded us over and over again that the new features did not need to be tested by the community before the 1.0 release because of the expertise in the QA and Experimental teams. "Trust us".

So now you can see why the community was lead to have very high expectations of QA and Experimental's. But again, either way it's going to work out, and we all need to show our appreciation to everyone involved, because KSP is awesome!!

[Ted]

Honestly think this has been blown out of proportion. There has been criticism towards the QA/Experimental teams - but honestly, the amount of very serious bugs that have come out of this process is a little odd.

A lot of this criticism is needed I believe, however not necessarily in the team personnel, but in the process. The bugs seem to be encountered in general gameplay (heatshields, faring mass distribution, etc) play, are the testers perhaps focusing too much on breaking systems, rather than playing them and seeing what looks odd?

The frustration you mention is generally because you tell us along the way 'how hard everyone is working' and 'how well it's going' to the extent that people are expecting extremely high-quality releases and they don't feel they're getting them. There's also the point that you can't tell anyone that the people you have on the team are any better at finding bugs than a good chunk of the people who play this game. A lot of people have 750/1000+ hours logged on this game, and this will make them feel comfortable (perhaps too much so) when voicing their opinion.

You guys are doing good work, i just question whether an analysis of the types of bugs/problems/balancing we're seeing can help make the process better.

I should perhaps add on that by no means do I think what we do and the way we do it is perfect - there is always room for improvement! So providing feedback on the types of issues you're seeing is indeed helpful.

[Robotengineer]

I think the issue wasn't the work ethic of the testers, but rather how much time Squad gave them. I still stand by the fact that a couple thousand people can do more work in a week than 100, even if you get a lot of 'bad' reports, you still have a large amount of people discovering bugs. I still think it would have been better to have done one last beta before 1.0. It would also help if the bug tracker was more visible on the forum, or at least had a link.

[FishInferno]

When people see a problem, they usually try to find someone to blame. So when there were lots of bugs, people blamed the testers. I am not saying it is right, but it is human nature.

I personally think that 1.0 looks great, and yes there is room for improvement, but nothing is perfect and you guys did a great job!

[TMS]

I think the issue wasn't the work ethic of the testers, but rather how much time Squad gave them. I still stand by the fact that a couple thousand people can do more work in a week than 100, even if you get a lot of 'bad' reports, you still have a large amount of people discovering bugs. I still think it would have been better to have done one last beta before 1.0. It would also help if the bug tracker was more visible on the forum, or at least had a link.

This.

The critique (from my perspective) has never been about the individuals, but the conditions under which they are expected to deliver. The critique was there when the rush to release was announced... and it's still there.

I'm sure there's a few morons who enjoy putting the boot in on a personal level, but... welcome to the Internet.

[Bedwyr]

I'm sure there's a few morons who enjoy putting the boot in on a personal level, but... welcome to the Internet.

Eh, I don't quite want to throw up my hands on that. I'm not that active on here, mostly lurking till this week's release. Forum culture's hard to do, but KSP's is positive and active enough that it's worth working for. So I think it's worth calling out (gently... with some judgment and not at the person, but what they're writing) a bad attitude and trying to help redirect it. Sometimes a "dude... not cool" can really do wonders. It's not about belittling the posters being over-dramatic, but persuading that they're not helping and that there's a better way to go. It doesn't work all the time, but sometimes it helps.

[Terwin]

As a software developer myself, I think that this is a relatively low-bug release.

Especially if you take in to account many of the 'bugs' are more balance or personal preference issues than game breakers.

Considering that this is only a $30 game(less for some), and more improvements are on the way, I am quite happy with this purchase.

[Fel]

I think the issue wasn't the work ethic of the testers, but rather how much time Squad gave them. I still stand by the fact that a couple thousand people can do more work in a week than 100, even if you get a lot of 'bad' reports, you still have a large amount of people discovering bugs. I still think it would have been better to have done one last beta before 1.0. It would also help if the bug tracker was more visible on the forum, or at least had a link.

Open source development does not work in reality and we do not actually practice open source development but a form of oligarchy. The problem is that if everyone, regardless of how knowledgeable said person is, submitted "code patches" and each patch had to be treated the same, then development would grind to a halt as you would have conflicting patches and a plethora of sub par code to look through before passing along the most professional.

What REALLY happens is that the "trusted few" get a cursory glance and a check off while the "insignificant others" are pushed into a pile where someone eventually looks at them.

Squad doesn't need MORE bug reports, they need BETTER bug reports.

[Wayfare]

Maybe it was a bit optimistic to go for 1.0 release with so many new features planned. Things like the new aero, overheating and resource harvesting might have better been put through Beta first. This community is used to being Beta testers after all. Given that we were left out of the loop on these, the natural reaction is to shift blame up the design chain. And that's not fair to the QA team. I'm sure you guys did great work and there's a ton of bugs we're not experiencing to outweigh the handful that we are. Kudos

[youkofoxy]

The QA problem is simple: a lack of time.

you should not make a rigid schedule.

it never worked well in the gaming world, often the result is a Watch Dog, or AC Unity.

I do not know who was the idea, but it is a terrible idea, especially when most of the team is made of "community heroes"

and I do not have much to complain about them, will say at most "lack capacity"

and it would be true, since it needs more... QA.

not that the team is bad, but they were clearly not ready for this type of load.

obvious... one or the other bug would pass unsee, after all, the User see bugs where it's don't exis.

but the lack of time was very critical.

PS: I know that time is money, but you have to understand that the QA team is literally cheap (in terms of money, of money)

and this plus faster... you have to forget good quality.

[Robotengineer]

Open source development does not work in reality and we do not actually practice open source development but a form of oligarchy. The problem is that if everyone, regardless of how knowledgeable said person is, submitted "code patches" and each patch had to be treated the same, then development would grind to a halt as you would have conflicting patches and a plethora of sub par code to look through before passing along the most professional.

What REALLY happens is that the "trusted few" get a cursory glance and a check off while the "insignificant others" are pushed into a pile where someone eventually looks at them.

Squad doesn't need MORE bug reports, they need BETTER bug reports.

I wasn't suggesting open source development, I was suggesting that they either a) have more testers, or have a beta before 1.0. Even if the current testers took off from work and put in 16 hour days trying to find and report bugs it still wouldn't yield the results that having a beta before release would, there would be a lot of noise, but it would be pretty simple to see what the community thinks are major bugs just by glancing at the General Discussion section of the forum.

[Wayfare]

Open source development does not work in reality and we do not actually practice open source development but a form of oligarchy. The problem is that if everyone, regardless of how knowledgeable said person is, submitted "code patches" and each patch had to be treated the same, then development would grind to a halt as you would have conflicting patches and a plethora of sub par code to look through before passing along the most professional.

We used to call it "throwing it to the monkeys". It's messy and hard to quantify but damn, those monkeys will find every way to break it and then some. You can't structure it. It actually takes a lot of man hours (and a pretty thick skin) to sift through all the feedback looking for those useful nuggets and oddly emerging trends. With a gaming community as obsessive as this one though, it's just too valuable a resource to not tap into.

[tater]

I think the OP post lacks the appropriate context.

If this were update 0.91, testing would not have been mentioned at all on the forums. The fact that it is arbitrarily 1.0 sets the standards very much higher. In a normal update, the lot of us would be beta testers, and the QA people Alpha testers. By going from a successful beta test, to an expansive alpha with no beta testing… you will see exactly the reaction you have seen, and it's entirely appropriate.

It is NOT the fault of the QA testers, it is the fault of not having a wider testing of something that is "release version."

YMMV, but that's how I see it.

[Monger]

it's equally "not fair" to assume that a customer (consider, there's people buying into KSP right now with the release of 1.0) has to somehow point out how to improve the situation. That's the job of the Q&A.

What makes you think that Q&A hasn't done that?

I guess this an important misunderstanding, which almost all customers and actually many developers don't understand: Q&A is NOT responsible for quality, they are responsible for REPORTING quality. The only ones that are able to create quality are those that create the product (i.e. the developers).

If KSP 1.0 is of low quality, then the developers of SQUAD (and their product owner etc.) are to blame. They could have extended the testing period, they could have taken other measures to avoid bugs (e.g. more automated tests), they could have avoided implementing features that were too risky (in the developmental sense) in the first place.

Q&A has nothing to do with any of these.

Edited April 29, 2015 by Monger

[roguelycan]

Was the game perfect when it released 1.0? Of course not, if it was then Squad may as well clean their hands off and be done with it. Is the game better than it was in .90? I would definitely say yes. While there have been some bugs, some worse than others, none of them I would consider game breaking. Sure my first few launches everything felt kinda screwed up but that was because of how different it was than previous versions. Now that I have started getting the hang of it I am loving the new aero.

[Tiberion]

There's really nothing different about this release (other than the number) than the dozens that have come before it. I've tested a lot of them at this point.

The testing team is bigger than ever, but also more organized and provided with better tools and much more defined testing goals than we had more several years. Have no doubt that Ted transformed the QA and Experimental testing into a very capable organization. The testing team was not "overwhelmed" or "unprepared" - More time might have let us catch a few more issues or get some more feedback addressed, but that is ALWAYS true of any release, not just this one.

There are certain realities about the process that will never change.

We test a LOT. Dozens of people, 50+ builds, hundreds of bug and feedback issues logged in the tracker. There's less than 10 devs including special mod contributors. Guess what that means? All of those issues have to be prioritized. That means some of them get fixed immediately (and receive multiple builds to get it just right) while others have to wait until the end during "polish" or sadly even don't make it on the schedule during the testing period

We can still miss things. Hundreds of thousands of people play KSP, and no two players have the same exact PC or play the game in the exact same manner. So when a version gets released, someone is going to push buttons in a different order or do their special thing and find a bug we missed. This is the reality of software in general.

It's also entirely possible we didn't "miss" something you think we did. Maybe we saw the issue and decided it was unworthy for now. Or maybe we reported the issue in an earlier state and the current state is the best solution that was available (this is especially true of delicate balance issues, which we ran into often and spent hours upon hours testing changes.) So even if "the forum" thinks something is super obvious and we must have been blind.. we probably weren't.

A lot of feedback is subjective. The testers don't always agree on an issue. We'll all way in on something and then its the devs job to process the feedback and make changes if appropriate. So if you're having a discussion about a balance issue out here, we probably had the same one (multiple times)

We're never completely satisfied. Well, I can't speak for every tester, but I personally am never completely satisfied at the end of a testing session. There are always things left that we want to see changed or something we'd like more time to test but there is always a point where you need to move on. (Straight talk, we may never know why 1.0 was scheduled the way it was, I certainly have no "inside knowledge" about it either. It is what it is.) But every release has a finite testing period, and this one was no different. "We could use more time" will always be true, but life doesn't work like that.

So its very easy to jump on the bandwagon for 1 or 2 issues that are the hot topics here on the forums and generate some indignant snark about how it should have been fixed. But takes those 1 or 2 issues and multiply them by a hundred and you're closer to the amount of things the testers and devs sorted through since testing was announced. There will never be a perfect release, and thats a fact.

I can honestly say, as a veteran tester this version was probably bigger than any we've had, and it included fixes for a multitude of longstanding issues. It also changed a LOT of things, and part of what you're experiencing is dealing with those changes. it is skewing your perceptions. There is certainly more work to do for next time (as there always is) but there are good ways and bad ways to get your feedback heard.

What you can do to help going forward:

Skip the outrage; Don't proclaim the testers ineffectual, the devs sloppy or uncaring, or frankly worry about what "should have been." Don't buy into the hype that this release is somehow much worse than usual or flawed or anything like that. Plenty of actual things to be outraged about if you look around (hint: probably not KSP related)

Go read the last paragraph of Kasper's original post, it has the details on how to properly report bugs and give constructive and helpful feedback.

Keep things in perspective; Things are almost never as bad as you think they are. Have fun; It's a game after all

[Lord Aurelius]

As far as 1.0 releases go, KSP is definitely not that bad compared to other games (often from big name publishers no less), there's no critical game-breaking bugs and the game is actually quite stable now.

The big issue I take with the release is that it could have been so much better and some pretty glaring issues (heatshields, insane parachutes, broken tech tree, nonsensical contracts) made it all the way through testing and into release, even though I ran into these within the first hour of playing the game. When the community heard the dev's plan to go straight for 1.0 without an interim beta, this is what many of us feared would happen and I feel that my concern was warranted, I do not consider this to be that great of a release. I have no doubt the issues will be fixed (by modders at least if not by the devs) and I still consider this to be one of my best game purchases, but I'm still disappointed to see yet another developer succumb to rushing a release. That almost never ends well.

I do not blame QA and Experimentals, rather I commend them for doing an amazing job in spite of the circumstances. What I do blame is a lack of software design discipline on the part of the devs. In addition to the rush to release, they were adding features right up to the last minute, and I'm sure the testers had their hands full just trying to get a handle on the new content in the limited time they had before the code went gold. The relatively poor communication of the devs with the forum community throughout this whole process has only exacerbated the negativity we're seeing, and I'll admit that I'm definitely one of the forum members who is not happy with Squad's handling of the game and forum community right now. I am glad however to see this thread started by Squad so we can at least discuss these issues in a (hopefully) civil manner.

I still stand by my earlier comments that this game really needed another public beta, look at how many bugs were discovered within hours of release. If Squad had planned a final beta into their release schedule, 1.0 could have been so much better, but as it stands I'll have to wait for 1.0.1/1.0.2 to (hopefully) get the release that 1.0 should have been.

Edit: In response to Tiberion's post above mine that was posted while I was writing:

I definitely agree that there's nothing really different about this release than the ones that came before it, and that's the heart of the issue. This still feels like an alpha/beta product, but 1.0 is not just a number in the software development world, it is an indication that you've reached a milestone in product development where you're ready to release your product to the wide world, and ready to accept any criticism that comes your way due to problems users will find. The fact that the final release feels basically the same as the early access releases should be a warning sign right there.

Overall, yes I agree that we all need to step back and put things into perspective and do what we can to give constructive and helpful feedback. I still greatly enjoy playing KSP (even if I feel that the developers have made brain-dead decisions at times) and look forward to seeing what the future brings.

Edited April 29, 2015 by Lord Aurelius

[Caelib]

Bugs are to be expected, and let me congratulate you all at a successful 1.0 launch! Please keep in mind that for every 1 person complaining, there are 100+ having a good time!

[MickG]

I like the game I got. I'm having fun. There were a few surprises, a lot to relearn but that hasn't stopped me from enjoying my head off. I'm sad to see so many people unable to enjoy this game in it's current state. Is that Squads fault? The testers? I don't think so. It's impossible to make everybody happy. I'm sure a lot of mods will pop-up in the near future that will taylor this game to everybody's taste. In the mean time; let's be civil and polite. I'm sure everything will work out eventually.