About recent community criticism in the direction of the QA & exp testers

[Wooks]

Squad used to release the experimental versions to the community, and back when KSP was small with a close knit community of technically minded players who understood the game was in development and would contain bugs, this was perfectly fine.

Then KSP gained wider appeal, there were more members, the concept of early access to an unfinished game eluded them.

0.14.0 was released.

It was an absolute, unmitigated and complete disaster....

...Squad realized then that public releases of experimental builds were no longer tenable, the testing team was set up and only versions that had passed them would go to public release.

...Thanks to them you can actually play KSP.

Public access to experimental builds is never coming back, and with good reason.

I wholeheartedly agree with a small team tackling experimentals, and it certainly worked fine when the releases where rather small and controlled with a lot of wiggle room timewise, but this release was different in scope and size compared to those versions.

I believe the issue here is that SQUAD tought they could pull this v1.0 the same way and it just went over their heads, too many changes in such a compact time frame, a rather small team of testers focused on testing all the new mechanics and features, and certainly not enough testing of the finished product before releasing into the wild.

Is a little bit pointless to keep repeating that this should have been v0.95 now that the cat is out of the bag but, hey! here we are.

Nobody is saying that 1.02 didn't need to happen, it did, emergency action was needed to save the game after 1.0 but 1.0 should never have happened the way it did. Even a halfway competant testing system would have found the major issues in literally ten minutes of play. Either the testers where not give 1.0 as it released, or they where not listened to, or they testers chosen where some bizarre element of the community that uses kerbal to simulate truck driving or something. (unlikely)

I agree with this too. It might seem like v1.0 was tested properly... from the standpoint of the handfull of testers that did it in a breakneck pace over a bunch of rushed weeks. Is clear now that v1.0 needed either more testers to work out bugs in such a hurry, or more testing time with the current batch of testers.

Edited May 4, 2015 by Wooks

[Lord Aurelius]

@Aerindel

I very much agree with this. Bugs in any release are more or less inevitable, but when major gameplay-affecting bugs that show themselves within 10 minutes make it into the release (especially one as significant as 1.0), there's something wrong with the system.

I'm glad that Squad is dedicated to supporting this game and pushing out hotfixes, but as Aerindel said, they shouldn't have been necessary.

@Yemo

Very much this as well. I think a big part of the reason the community is so polarized right now is that there's basically two groups of people: those who treat 1.0 as just another release number, and those who understand that particular number to be something significant. If this had been just another early access release, there would still have been some complaints for the things the hotfixes addressed, but by and large it would have been a very positive update because the new aero and heating system especially really addressed some of the things missing from this game.

If you take 1.0 to mean that the game is "finished", then it is easy to see why so many people are frustrated. IMHO, this game still feels like it's in early access, and still lacks polish in a lot of areas. Granted, a lot of games at 1.0 have issues (even big AAA releases), but just because "everyone else is doing it" and treating 1.0 like a public beta doesn't mean that it's the best way to do things.

Edited May 4, 2015 by Lord Aurelius

[Nachtwind]

What was totally insane was to introduce entire new classes of gameplay without testing. And, yes, I am going to say WITHOUT TESTING because 1.0 had so many totally broken gameplay elements that I can't imagine that anyone with authority actually tested them even once.

...

What will 1.03 have in store for us? A few minor tweaks and bug fixes? Or will they throw more fundamental game changes at us without any testing to see if they actually work.

Damn.. you finally found out how the QA Team works.. now we have to even introduce bugs to make up for the lack of not testing stuff anymore

and honestly you disqualified from discussion right here.. right now..

[Gerandar]

Opening up the test to all users via steam or your store page would have helped round out those numbers, your QA team dropped the ball you as a dev team dropped the ball by allowing such a small number of testers there is no defense for the lack of QA that 1.0 was released with seriously you could have had a ton of users testing it and you chose to pick a handful of people instead of the ladder.

[Aerindel]

Damn.. you finally found out how the QA Team works.. now we have to even introduce bugs to make up for the lack of not testing stuff anymore

and honestly you disqualified from discussion right here.. right now..

Then explain how 1.0 happened? I ask again, did anyone actually play the game? I don't mean start it up and see if it crashes but actually try to play the game, as in do normal fun KSP activities like fly a ship to another planet or re-enter an atmosphere?

That's pretty offensive to the volunteers who did test 1.0.

I don't see how. I came up with three options that could explain what happened and only the last one could remotely be the fault of the actual testers and I even labeled that one as unlikely. Its beyond belief that the testers missed the problems 1.0 had if they where actually given 1.0 so I assume that they where either not actually given 1.0 as we saw it to test, or that their bug reports where not read/ignored.

[Renegrade]

If this had been just another early access release, there would still have been some complaints for the things the hotfixes addressed, but by and large it would have been a very positive update because the new aero and heating system especially really addressed some of the things missing from this game.

If this had been just 0.27 or 0.28, I'd have called it "outstanding" or "spectacular". There was a lot of good stuff added. For a "1.0" release, I'd just call it 'good' and give it maybe 75%, because:

IMHO, this game still feels like it's in early access, and still lacks polish in a lot of areas.

...it lacks the polish of a non-alpha, non-beta product.

(I would have been fine with continuing in the early access/alpha model)

Granted, a lot of games at 1.0 have issues (even big AAA releases), but just because "everyone else is doing it" and treating 1.0 like a public beta doesn't mean that it's the best way to do things.

I think you mean 'especially big AAA releases'~ Those guys are really lowering the bar for the rest of the industry.

[Dal]

Anyone blaming QA is rather shortsighted, as they're at the mercy of their management. Management, I'll say, which seems to be allowing hasty and occasionally sloppy work to be released with concerning frequency. The game definitely feels like a 0.65 on a 0-1 scale right now. There's still a long way to go until it really feels "whole".

[Aerindel]

Anyone blaming QA is rather shortsighted, as they're at the mercy of their management.

I think that is what everyone is saying. It seems very unlikely that the QA problem has anything to do with the individual testers, it has to be the system they operate in.

[basic.syntax]

We've just published KSP v1.0.1. This is a small revision patch to address some of the most noticeable bugs we encountered since the release of 1.0.

Some of these we knew about for a while, but couldn't fix in time for the release (as we approach publishing time, the risk of code-breaking and delaying the entire release outweighs any benefits in the potential fix), others we found out about during the weekend livestreams, and some others we found from your own feedback.

This patch isn't meant to cover every single bug, of course, just the more relevant ones. We're going to keep on with the bugfixing and tweaking as we move into development of version 1.1.

Inserting MEME: "Keep Calm and Carry On."

[longbyte1]

I don't criticize QA for anything; they aren't even paid anything and are forced to keep details of the game secret. Many of them do not even have personal ties with any of the developers.

But a wider community should also be given a proper chance to participate in bug squashing before the bugs are left in a public release. And if the version is bug-free, then those testers will be sure to keep their lips even tighter.

Also, why aren't the "stock bug fix modules" pushed up to the main game yet? It's not helping the developers if they're not accepting the help..

[hawk_za]

mr Aerindel with all due respect you not the only one who has 10000s of hours of gameplay into this game I have been with this project since about .17 and have a fairly good understanding of how these game mechanics work and how they used to work.

For goodness sakes nose cones would hinder launches.

you lay claim that this game was never tested or if the team had done Even a halfway competent testing ......so what do you think scott manly and other more capable players more so than myself or you, were doing during the period between .90 and 1.0

emergency action was needed to save the game after 1.0 I call bull

your entire argument is based an vagueness and insubstantiality you using inflammatory insults to bring across the way you are feeling by insulting the work of others

so far if you are such a hot shot why were you not on the test team could you have done better...........I think not and maybe that's why you going on like this.

then you backtrack and talk about the testers not having the actual 1.0 so (were you or were you not on the test team) as only a test team player will know that

if not stop talking about things you know nothing of

Your posts in this thread are not helping ANYONE.

Please rather than wasting peoples time, compile a bug report with actual data i.e. at this speed at this height the wings no longer work (example)

then post the bug but don't come into a forum and say other peoples work was rubbish because that only upsets people.

yours in service

the hawk

- - - Updated - - -

I don't criticize QA for anything; they aren't even paid anything and are forced to keep details of the game secret. Many of them do not even have personal ties with any of the developers.

But a wider community should also be given a proper chance to participate in bug squashing before the bugs are left in a public release. And if the version is bug-free, then those testers will be sure to keep their lips even tighter.

Also, why aren't the "stock bug fix modules" pushed up to the main game yet? It's not helping the developers if they're not accepting the help..

the problem with opening up the testing process to the wider community is that it increases exponentially the amount of useless reports you have to filter through wasting dev time

And in the end you find out that you fix less bugs than if you had kept the team relatively small.

to paraphrase it is quality over quantity of bug reports that you want as a game dev

[Wallygator]

I find it interesting that this thread continues since the boat has already left port.

I would think there is a more constructive topic at hand which is = "How might squad optimize future releases to take advantage of community input without compromising established internal testing and QA processes?"

[Aerindel]

you lay claim that this game was never tested or if the team had done Even a halfway competent testing ......so what do you think scott manly and other more capable players more so than myself or you, were doing during the period between .90 and 1.0

Good question. I don't know, chasing down bugs perhaps and not actually playing the game to see if basic gameplay works rather than just trouble shooting crashes?

emergency action was needed to save the game after 1.0 I call bull

And yet what appears to be emergency action is what was taken by squad who released two updates just a few days later and just hours apart.

your entire argument is based an vagueness and insubstantiality you using inflammatory insults to bring across the way you are feeling by insulting the work of others

Yep.

so far if you are such a hot shot why were you not on the test team could you have done better...........I think not and maybe that's why you going on like this.

Never tried to be on the test team or really had any interest in KSP except just playing it, it wasn't until 1.0 dropped that I actually felt like I needed to jump on the bandwagon and try and find out what the heck had gone wrong with my favorite game. Until 1.0 each update felt like an improvement to the game, it was pretty shocking when it went backwards in what was supposed to be the full release.

Your posts in this thread are not helping ANYONE.

I don't have the power to help anyone, I'm just adding my voice to the chorus shouting "the emperor has no clothes"

Please rather than wasting peoples time, compile a bug report with actual data i.e. at this speed at this height the wings no longer work (example)

then post the bug but don't come into a forum and say other peoples work was rubbish because that only upsets people.

People need to be upset. That is the entire point. In order for people to even start to think about change they have to be upset.

You keep trying to say I am claiming to be a hotshot or something but that I've said nothing of the sort and its entirely besides the point. The gameplay problems with 1.0 where not things you needed to be good at the game to find. They where things that a novice player would have run into on his second or third launch. The only difference is that a novice player wouldn't have known that it wasn't supposed to be that way, he would have just thought the it was a bad game.

Maybe that was the problem? all the testers where so experienced they never tried a simple rocket re-entry? There is a real problem in many fields where people become overqualified and start missing the basics. I'm an EMT and help write standard operation procedures for a fire department and see it all the time where people who are trained in advanced techniques lose sight of the actual job at hand in favor of specific procedures. I honestly could see that happening. If I hadn't started KSP over from scratch with 1.0 it could have been a long time before I ran into some of the problems.

I just don't know, I wasn't on the test team, I don't know how it works...I only know what the results where.

I would think there is a more constructive topic at hand which is = "How might squad optimize future releases to take advantage of community input without compromising established internal testing and QA processes?

Established QA

is

compromised.

But I do have a suggestion: the testing phase should be divided between teams testing for stability and performance and a team that is testing actual gameplay as those are really very different problems. There is no doubt that 1.0 had less technical bugs and had better performance, but that is like saying that a book was really good because it didn't have any spelling errors. Where the testing team failed was in looking at the actually

experience

of the game. How the new features made you feel while using them.

Instead of bug reports they should have asked for psych reports. They should ask people how re-entry felt in 1.0 rather than just asking if the game crashed or not. If you read the various slam posts on 1.0 very few of them are about technical details but rather about the emotion that the game creates, or fails to create.

The second and most obvious issue was that they moved way too fast, adding entire classes of new features in one update without a beta. New Aero, a heating system, a mining system, a heat shield system, these weren't just updates, these where entirely new fundamental systems that where just thrown out in the wild, all at once, when normally any ONE of these would have been a major new release.

Luckily, about 80% of the problems with 1.0 where fixed in 1.02. The only reason we are even talking about this still is because this type of incident is not normal for KSP and it has people rattled. Lets face it, we are in a dark age of gaming right now, the industry as a whole is floundering around, AAA is releasing broken and boring content and trying to squeeze more money out of us while doing it. The indie scene is in chaos with true gems, shovelware and vapor ware all competing for attention. FTP and PTW games are poisoning every level of the industry. Kerbal is one of the very few good things to come out of gaming in the last few years, and it almost shot itself in the head with its big official release.

Edited May 5, 2015 by DuoDex

[rockowwc]

God forbid any of these people paid the big dollars for Watch Dogs.. One of the most seriously flawed games ever released, that even the 0 day mega patch didn't fix some of the massive issues like getting stuck on rail tracks...

[sal_vager]

People need to be upset. That is the entire point. In order for people to even start to think about change they have to be upset.

No, this is where you are so wrong it's not even funny.

How to give good feedback.

How to give bad feedback.

Go have a read, you will not find any advice telling you to be angry, quite the opposite in fact, getting angry just makes others far less likely to listen to you or take your words on-board.

Edited May 5, 2015 by sal_vager

[Lord Aurelius]

I hate to say it, but Aerindel has a point. For quite some time now it's felt like the only times the devs actually interact and listen to the forum community is when they get all worked up and, well, angry. Sure we get the devnotes every week (which I'm very thankful for) but those are basically public service announcements and the devs for the most part don't seem to really interact with the community. Sure they responded to the Round-8 issue, but in the perception of many that was only because the community made such a stink about it (and Harv's post to "mod it back if you don't like it" only added fuel to the fire).

The 1.0 release has brought this issue to a head, there was a tremendous community outcry against the rushed development schedule, and now that we've received what appears to many to be an obviously rushed release, from experience it seems like the only way to get the devs attention to fix it and not make the same mistake again is to shout on the forums.

This is not the way it should work, but unfortunately in the perception of many Squad is disconnected from the community here and the only way to get them to pay attention is to raise a storm. I'm not saying this is how things actually are, but perception is huge and unfortunately my perception backs up what Aerindel is trying to say. The official bugtracker often appears to be a black hole, with many bugs in there lingering update after update and still haven't been fixed in 1.0.2. Claw has been maintaining a stock bugfix mod for quite some time (and was used in the OP as an example of how to help the devs) but it still has modules from previous update that are relevant to fixing bugs in the current release. What is the point of going through the correct feedback channels if the devs appear to just ignore them anyways? Like I said, this may not be how things are, but how things appear to many in the community.

Going forwards, something needs to be done to fix this issue. This kind of relationship between devs and a game community is flat out toxic and doesn't benefit anyone. It will likely require work from both sides (I can imagine that part of the reason the devs don't interact more with the community is that they're afraid of these kinds of blowups), but something (an honest discussion perhaps?) needs to happen here on the forums (not on Reddit or Twitter or anywhere else).

I still very much enjoy KSP and I want to extend thanks to the devs, moderators, QA/Experimental testers, modders and players who have made this game what it is, and I really hope the root cause of these issues can be addressed soon so we don't keep getting blowup after blowup over every little thing.

Edited May 5, 2015 by Lord Aurelius

[pandaman]

One possible improvement may be to keep the current QA and experimental systems and team structure pretty much as it is. But add in another 'test group', taken from experienced players of varying styles, they are then given (with relevant NDAs of course) a copy to just play with and get general real world 'in play' feedback, in essence a more controlled way of releasing a beta version out to the world so as to reduce the quantity and proportion of unhelpful responses. This could help find some of the balance issues and more obscure 'anomalies' that only seemed to show up in the pre release live streams or after we all got our hands on it.

[Wooks]

I find it interesting that this thread continues since the boat has already left port.

I would think there is a more constructive topic at hand which is = "How might squad optimize future releases to take advantage of community input without compromising established internal testing and QA processes?"

I agree, we need to adress the internal testing and QA process, what's done is done.

[BagelRabbit]

...The only times the devs interact and listen to the forum community is when they get worked up and, well, angry... What is the point of going through the correct feedback channels if the devs appear to just ignore them anyways? Like I said, this may not be how things are, but how things appear to many in the community.

Throwing my hat in the ring here: this is exactly what I think is happening.

Going forwards, something needs to be done to fix this issue. This kind of relationship between devs and a game community is flat out toxic and doesn't benefit anyone. It will likely require work from both sides (I can imagine that part of the reason the devs don't interact more with the community is that they're afraid of these kinds of blowups), but something (an honest discussion perhaps?) needs to happen here on the forums (not on Reddit or Twitter or anywhere else).

...and this is exactly what should happen.

I think that such a talk would do wonders for the morale of the community. It would prove that SQUAD really does care, and it would likely decrease the number of long, bile-filled threads such as this one, when even the moderators get caught up in ad hominem arguments. Oh, that reminds me!

Dear sal_vager,

Recently, you told Aerindel that

said something silly

That's far from polite, whether or not it's correct. If I was a member of the moderation team, I would likely edit this statement out of your post, as it's not constructive and seems downright mean.

Drink a cup of tea or step outside for a moment. It'll help.

(Actually, this advice probably applies to many of the posters on this thread too )

-Upsilon

Edited May 5, 2015 by sal_vager

[sal_vager]

Heh, you know that was a general "you" UpsilonAerospace

But yeah I'll remove it.

[Scourge013]

One possible improvement may be to keep the current QA and experimental systems and team structure pretty much as it is. But add in another 'test group', taken from experienced players of varying styles, they are then given (with relevant NDAs of course) a copy to just play with and get general real world 'in play' feedback, in essence a more controlled way of releasing a beta version out to the world so as to reduce the quantity and proportion of unhelpful responses. This could help find some of the balance issues and more obscure 'anomalies' that only seemed to show up in the pre release live streams or after we all got our hands on it.

This is basically the system we already have--and it doesn't work. What they need to do is keep the Experimentals and Q&A teams more or less as they are, but leverage the Beta option on Steam more effectively. After the Experimentals and Q&A teams have at the game, use the Betas tab in Steam to release a "Pre-Release" build based on that smaller team's feedback. This build is open to the whole community, but doesn't force anyone, at least for the moment, to use it. Anyone who is interested opts in to the Beta, plays it, and provides feedback on a public Beta Builds forum that the devs watch closely.

Minecraft, Project Zomboid, StarMade, Subnautica and other games use a system just like this and their respective communities really appreciate it. Not sure why Squad doesn't--maybe it will "delay" the process of updating? But Squad is independent. They make their own deadlines and like Blizzard can say to the community "It's done when it's done, now test this sucker for as long as you can."

That way there is an ever-increasing pool of testers from truly all segments of the KSP community critiquing the game. At least for me, things like the 1.0 release heatshield bug, while annoying and seemingly obvious, aren't the issue--it could have been an honest mistake toggling a flag that shouldn't have been toggled while addressing other Q&A feedback literally moments before pushing the update.

The issue is huge balance revisions that increase the drag for every part across the board that can or have the potential to render whole fleets of rockets useless across the entire player-base with no warning. The Steam Beta tab feature is a way to work around that while experimenting with the balancing of features.

Hot-fixes and bug fixes are good...across the board gameplay balances implemented with no discussion (even if they are marketed as "fixes" or provide a "better" game) are not so much, regardless of whether or not it takes the game in a favorable dimension. If Mojang for example wanted to "fix" building in Minecraft by incorporating a physics based structural integrity simulation, while making people craft a "cement" item to make placed blocks stick together if they don't have direct contact with the ground (i.e. no more floating houses in the sky), the game might be more fun because it involves management of another resource and also make the game more "realistic". However, whole structures across the entire player-base would come tumbling down.

Mojang, however, before they implemented such a game-changing feature would release it during their opt-in Pre-Release build phase the whole Minecraft community can take part in. For some reason Squad makes similar game changing, uh...changes, but does not involve the community in this manner, using instead a very small group of players relative to the ever-expanding player-base.

Edited May 5, 2015 by Scourge013

[HafCoJoe]

I gotta say that all you devs and testers are amazing. The time you all put in to testing and bug spotting is an immeasurable use, and you guys do a great job of it. I'd wish to be a tester/media man if I had the time you guys had! My signature says what it says for a reason. You guys rock!

[DMSP]

if nobody points out your mistakes' date=' you won't improve. [/quote']

There is a huge diffrence between pointing out mistakes and being all out rude.

Example:

Pointing out mistake:

Oh, by the way, there is something wrong with the Drilling System. I will post more details and try to help you fix it!

What some people actually do:

Darnit, my base is ruined! You suck, why can't you catch bugs? You're so bad at your job!

Well, DMSP out.

I actually never saw anyone doing something like that bad example, and also, it's just an example

- - - Updated - - -

People need to be upset. That is the entire point. In order for people to even start to think about change they have to be upset.

When you think about it, that is true.

Kinda sad though...

I wish the world was not like that, but that'll only be in my dreams.

[sarbian]

I agree, we need to adress the internal testing and QA process, what's done is done.

You assume that the process does not work when you know nothing about said process.

Edited May 5, 2015 by sarbian

[monstah]

I hate to say it, but Aerindel has a point. For quite some time now it's felt like the only times the devs actually interact and listen to the forum community is when they get all worked up and, well, angry. Sure we get the devnotes every week (which I'm very thankful for) but those are basically public service announcements and the devs for the most part don't seem to really interact with the community. Sure they responded to the Round-8 issue, but in the perception of many that was only because the community made such a stink about it (and Harv's post to "mod it back if you don't like it" only added fuel to the fire).

http://tvtropes.org/pmwiki/pmwiki.php/Main/ViolenceReallyIsTheAnswer

This is basically the system we already have--and it doesn't work. What they need to do is keep the Experimentals and Q&A teams more or less as they are, but leverage the Beta option on Steam more effectively. After the Experimentals and Q&A teams have at the game, use the Betas tab in Steam to release a "Pre-Release" build based on that smaller team's feedback. This build is open to the whole community, but doesn't force anyone, at least for the moment, to use it. Anyone who is interested opts in to the Beta, plays it, and provides feedback on a public Beta Builds forum that the devs watch closely.

In my opinion, one of the sanest opinions said in this topic. Oh, and as a bonus, it's not a complaint, but useful criticism to be applied in a future situation. The feature is called "Beta" on Steam, but it's more like a dev version, and could come with any version, say, 1.0.3.