About recent community criticism in the direction of the QA & exp testers

[Chezburgar7300]

*Read the front and last page, not the whole 15 pages.*

There's heaps of criticism? Jeez. I've noticed a few bugs here and there but nothing game-breaking or serious. Isn't that the norm with games?

Is there a need to bag the QA/Experimentals team? They're volunteers. As Kasper said, release the kraken on him not the volunteers! They were helping out, and without them we would have waited a much longer time for 1.0 and any updates prior.

[stibbons]

There's heaps of criticism?

This was how I knew you hadn't read the whole 15 pages.

[brienne]

Some fans may overreact. I guess it s a tiny minority.

KSP is great. KSP 1.0 is a big step ahead.

imho, KSP 1.0.2 came a bit too far and put the mess.

Months of work before release, and 2 patches in a week. Weird.

Stuff happens.

Edited May 4, 2015 by Red Iron Crown
Profanity is not OK, even if obfuscated.

[AbhChallenger]

Squad used to release the experimental versions to the community, and back when KSP was small with a close knit community of technically minded players who understood the game was in development and would contain bugs, this was perfectly fine.

Then KSP gained wider appeal, there were more members, the concept of early access to an unfinished game eluded them.

0.14.0 was released.

It was an absolute, unmitigated and complete disaster.

The forums were absolutely hammered with endless screaming and abuse, many players expected, demanded, fully working and playable versions from a game in the early days of development.

What we've seen the last few days has been nothing in comparison.

Squad realized then that public releases of experimental builds were no longer tenable, the testing team was set up and only versions that had passed them would go to public release.

0.15 was playable, so was 0.16 and 0.17 and so on, every update has been playable.

If you were given the experimentals, or worse, the QA builds, and saw everything break and all your saves and craft destroyed you'd flip your lid, the testers know they can't expect things to work which is why they are on the team.

Thanks to them you can actually play KSP.

Public access to experimental builds is never coming back, and with good reason.

That was ages ago. And unless I am mistaken. It was not using the steam beta. Steam beta you have to deliberately enable the thing that says beta. The thing that says "Testing only" or "UNSTABLE!!!"

We do not need the QA builds or very early Experimental. Just toss a week on the planned date and let people poke around with an RC version. It does not have to be the higher quality of testing at that stage. It needs quantity.

[basic.syntax]

That was ages ago. And unless I am mistaken. It was not using the steam beta. Steam beta you have to deliberately enable the thing that says beta. The thing that says "Testing only" or "UNSTABLE!!!"

We do not need the QA builds or very early Experimental. Just toss a week on the planned date and let people poke around with an RC version. It does not have to be the higher quality of testing at that stage. It needs quantity.

This assumes that additional reports will be more effective at getting bugs fixed; lack of testing isn't always the bottleneck in release quality. I wrote about this conundrum a few pages back. Time definitely factors in; I don't think any tester would object to extending a schedule, nearly indefinitely

At some point down the road, there may be an new open call for applications to join the Experimental test team. That is Squad's process thus far, and something to watch out for.

[AbhChallenger]

Personally I do not sign NDAs so I would never be allowed to join the experimental team.

[sarbian]

It needs quantity.

Quantity of what ? People complaining that their rocket do loops because they don't bother trying to understand the change in the new aero ? Testing require a basic understanding of the underlying mechanics to provide worthy reports. Releasing it to the masses would just generate an endless flow of duplicate bugs, poorly explained bugs and "bugs" that are actually point of view on the change. Look at how badly any thread about realism goes and try to imagine that for a public testing.

Not so long ago the public bug tracker was full of useless reports and without the work of some of the QA/exp team it would still be like that. Now if the flood gate are opened the tracker will become useless again.

Yes, the current process is not perfect. I have my opinions about how it could work better but adding testers is definitively not on that list.

[Nachtwind]

We do not need the QA builds or very early Experimental. Just toss a week on the planned date and let people poke around with an RC version. It does not have to be the higher quality of testing at that stage. It needs quantity.

Ahem... by playing Alpha and Beta versions you basically were playing Alpha and Beta Versions of KSP.

[AbhChallenger]

sarbian, Flood gates open for a week of RC testing on Steam? Sounds a bit dramatic to me. And frankly is borderline elitism.

- - - Updated - - -

Ahem... by playing Alpha and Beta versions you basically were playing Alpha and Beta Versions of KSP.

That were tested by QA and Experimental teams before we got to use them. Which to your credit worked well. What I am worried about is what the switch to Unity 5 has in store.

[Nachtwind]

That were tested by QA and Experimental teams before we got to use them. Which to your credit worked well. What I am worried about is what the switch to Unity 5 has in store.

Well... so what difference does it make to wait for the final release and get a fix afterwards? The point just is that any RC wouldnt really change anything. People, and this is yet another fine example of eliterism, would just use the beta as an excuse to get a bleeding edge version of KSP and complain about the bugs instead of filing bug reports. Just see what happened here when 1.0 was released.. Nothing else would happen if a 1.1RC would come out.

The change U5 will be heavy but it is not the first change of Unity Versions we had so far and till now we handled that stuff quite well, dont you think?

[Superfluous J]

sarbian, Flood gates open for a week of RC testing on Steam? Sounds a bit dramatic to me. And frankly is borderline elitism.

I personally prefer it that only elite QA testers are testing while the game is in QA.

Is it elitist to say that people who have proven they can't drive cars should not be allowed to drive? This is the same thing, only less bloody.

[Ignath]

That was ages ago. And unless I am mistaken. It was not using the steam beta. Steam beta you have to deliberately enable the thing that says beta. The thing that says "Testing only" or "UNSTABLE!!!"

We do not need the QA builds or very early Experimental. Just toss a week on the planned date and let people poke around with an RC version. It does not have to be the higher quality of testing at that stage. It needs quantity.

You're right...that was ages ago. Now, there are FAR more people who play KSP, far more people who would "participate" (note this word is in quotes because their participation would be nothing more than "Hey, I'm going to play KSP 1.X before everyone else...Oh balls, it's broken. I must go rage on KSP forums") in the "testing" of the RC version and we'd end up with the same debacle we saw back in 0.14. No matter how many times it is said by QA/Experimentals team members, for some reason the key point isn't getting through:

It's not about QUANTITY of testers, it's about QUALITY of testers.

Right now, after the years of testing that some of these QA/Experimentals members have gone through, I don't think dumping thousands upon thousands of "green" testers is going to help anything...just bring about a bunch of bug reports and issues (some of which will already have been solved) that won't go anywhere toward improving the end-result. In fact, it's likely to cause the same issues that are showing up now, with people complaining about bugs that are missed and wondering why (now since there are so many testers) these "obvious" bugs were allowed to slip through.

Personally I do not sign NDAs so I would never be allowed to join the experimental team.

Correct, which is a choice of your own and not really SQUAD's fault. You can't expect that they'd want you (or anyone else) divulging all the information about their Experimentals testing can you?

[Silanda]

My $0.02: While it makes sense to keep details of future releases under NDA, it seems somewhat counterproductive from a community building point of view to completely bar testers from talking about versions that have already been released. It makes even less sense if, as it seems from the replies here, testers aren't even allowed to discuss whether a bug in a publicly released build was logged by them or not. It might be beneficial for isolating Squad from criticism but it seems to place the experimentals and QA teams in the firing line when there's public ire over bugs.

The testers have been taking flack, probably unfairly, because of the presence of obvious bugs that seem to have made it to what was supposed to be the "ready for primetime" release. As has already been said though, this could be because they missed the bugs, or it could be that they were reported but left unfixed by Squad in favour of a quicker release. If it is the latter then, from an outside perspective, the testers seem to be being thrown under a bus. Squad don't seem to be in a hurry to comment, and, by barring them being able to confirm or deny whether the bugs were known, the testers are made to look poor. I understand that this will happen in a corporate environment but it seems pretty crappy when the people on the receiving end are mainly unpaid volunteers.

[Red Iron Crown]

Squad don't seem to be in a hurry to comment

See the OP of this thread.

[Silanda]

See the OP of this thread.

I did. AFAIK Squad have not commented on why the supposedly feature complete game then required large aero changes almost immediately (is the aero model not a feature?). Changes that have created yet more issues. They also haven't commented on whether the bugs were missed or whether there simply wasn't enough time to fix them before launch. That's what I'm talking about.

It's fair for them to say that it's unfair to criticise the testers, because it is, but if the testers are receiving unfair criticism as a direct consequence of decisions that Squad made, then Squad need to put their hands up and accept responsibility. That would be quite refreshing for the game industry and really would quell criticism of the game's testing.

[Red Iron Crown]

I read the OP as "Don't blame the testers", but I suppose that's up to interpretation.

[Silanda]

I read the OP as "Don't blame the testers", but I suppose that's up to interpretation.

I do see where you're coming from, it does tell people to blame the developers if they have to blame someone. Personally I'd have just liked a bit more openness though (well, personally I'd have liked 1.0 to have been 0.95 and to have not left early access until the balance had been nailed but I see I'm not alone).

[hawk_za]

just my 2 cents

I feel that not only was 1.0 a huge improvement over .90 but I also feel that 1.0.1 and 1.0.2 went a ways to fix a few of the more glaring errors in the matrix

and this i.m.h.o could not have been achieved without the help of the test team I for on would like to take a moment to stand up and say thank you

thank you for not only helping the dev team with things I would like to have helped with but just could not due to work and lack of fundamental understanding of coding

but doing it for FREE over days and days and nights and ....well you get the gist

I myself am a volunteer fireman and trust me when I say do a kindness and be scared no you cant please everyone but I recon you guys, dev team and test team did a phenomenal job

it is indeed a pity that our previously kind community has degenerated into this abysmal cesspool of name calling and blame throwing and fault finding

bug reports are one thing but come on guys we are so much better than this

lets all try reign it in and see how we can repair this schism we are creating

yours in service

the hawk

[rkman]

It is ok if i generally blame Squad for creating the impression that 1.0.x would not be yet another beta version?

[m4v]

It does not have to be the higher quality of testing at that stage. It needs quantity.

Open source projects will like to have a word with you, big projects need a triage team dedicated exclusively to clean and fix the utter crap that users like to fill bugtrackers with.

[Scourge013]

CC @Scourge013...

Squad is already aware of a lack of a certain amount of help for new players, and have stated they are working on a full tutorial and scenario overhaul for the game. Yes, it wasn't done in time to be released with 1.0, but I hardly call that a reasonable excuse to be up in arms over that. There is still a plethora of videos and guides online to help, made by slave laborers(since that term seems to be popular in this thread, I use it only in jest, of course) out of their own time with no call from Squad to do such...blah blah blah

I don't think you have quite understood my point. It's not that we literally don't know how to play the game. I for one, have at least 1,000 hours playing it. 700 on Steam and at least several hundred undocumented hours in "offline" play and earlier versions. Most of this time was broadcast on Twitch as well. I'm quite familiar with the game and its mechanics and have been an active, if somewhat irregular part of this community for years--just not on the forums.

My criticism isn't leveled at "Oh, woe is me, I have to redesign my fleet of rockets because of the new Aero or Ore Tank changes." If I may put on my monocle: That feat is done easily enough for someone at my skill level.

My criticism is that I have to. The game is no longer Early Access. Release builds of the game should no longer be treated as mass testing or gameplay balancing versions. The game should be stable both in terms of software bugs and gameplay practices. For the Record: No software is going to be bug free...but at least for me, that's not the issue in my criticism of the process since 1.0 hit.

I'm going to go back to my card game analogy for these changes. KSP has always been a game about flying rockets to space. Poker has always been a game about having a stronger hand. However, when one changes the aerodynamics model on a week to week basis, which is well before many of us can complete a career play through, it's like going from Poker to Texas Holdem while still playing the same round. They are both undeniably similar games, but also undeniably different. How are we supposed to play when the rules keep changing so drastically?

This is a direct result of Squad failing to use their chief advantage of having a game in Early Access: the sheer amount of people willing to test the game. It's why I bought the game in Early Access. I understand the need to have a smaller team test the game to make sure the product pushed to Steam isn't all regression, no progression. But once a build gets to that point, it should be provided to the Early Access community.

I seem to recall a poll on this very forum asking this community if KSP should go from .90 to 1.0. And the poll was decidedly against that. By several hundred votes. The issues we are seeing now with the game aren't the result of Q&A or Testers, but of a failure by the developers to leverage the advantage of having an Early Access community. They should have released components of what eventually became 1.0 to the wider Early Access community gradually over several builds.

We could have critiqued the game over the past several months constructively and had a 1.0 version that doesn't change rules or core mechanics from week to week. Early Access builds of any product are supposed to change rules and core mechanics--and frequently. If people were to complain, that's too bad for them, because they don't understand Early Access. We have now had more updates in a single week than we did in the past four months. These updates change the way you have to play the game. These are not bug fix releases--they are core changes.

The game is now in a different phase of its life. It's supposed to be "done." And its gameplay and underlying systems should not be revised as drastically as they have in the past week. These revisions are necessary...but they are the result of not doing Early Access properly. It makes it difficult to play the game, not because the game gets harder or easier...but because we don't know if we are doing Poker rules or Texas Holdem rules.

At this juncture, Squad needs to pick one and stick with it--and also tell the community not to expect an update to the mechanics in question for a significant amount of time so that we know it's "clear" to start a serious long-term play through of the game.

Edited May 4, 2015 by Scourge013
Added the "For the Record.." bit

[hawk_za]

At this juncture, Squad needs to pick one and stick with it--and also tell the community not to expect an update to the mechanics in question for a significant amount of time so that we know it's "clear" to start a serious long-term play through of the game.

that is insane if a game mechanic is not working correctly it should be fixed REGRDLESS of version number

to that point we all know squad is working on multiplayer for post 1.0 that's a core aspect of the gameplay but he our software has a 1.0 tag so in the words of tatu you aint goanna get it (heheh)

I defiantly feel there has just been an self justified overreaction here over what a version no

this is getting insane now

[Aerindel]

this is getting insane now

No, its getting sane now.

What was totally insane was to introduce entire new classes of gameplay without testing. And, yes, I am going to say WITHOUT TESTING because 1.0 had so many totally broken gameplay elements that I can't imagine that anyone with authority actually tested them even once. I found major problems with just my second 1.0 launch. And no I'm not talking about all the people complaining because their .90 ships don't work, those people should be ignored, I am talking about all the people like myself who have hundreds or thousands of hours into Kerbal who where waiting for 1.0 to start their "real" game and who found a totally broken package.

I mean, just look at how many people with low post counts but early forum join dates are talking about this all of a sudden. You have people who joined the forum years ago who only now have felt the need to start posting. 1.0 was not just the normal "kerbal" 1.0 was a special kind of mistake.

Nobody is saying that 1.02 didn't need to happen, it did, emergency action was needed to save the game after 1.0 but 1.0 should never have happened the way it did. Even a halfway competant testing system would have found the major issues in literally ten minutes of play. Either the testers where not give 1.0 as it released, or they where not listened to, or they testers chosen where some bizarre element of the community that uses kerbal to simulate truck driving or something. (unlikely)

The reason we are so up in arms right now is because when you see a fundamentally broken system in charge of something you really like it causes a lot of fear that we will lose kerbal as a enjoyable game and whole house of cards will come down. 1.02 had more changes from 1.0 than 1.0 had from .90.

What will 1.03 have in store for us? A few minor tweaks and bug fixes? Or will they throw more fundamental game changes at us without any testing to see if they actually work.

Edited May 4, 2015 by Aerindel

[Red Iron Crown]

What was totally insane was to introduce entire new classes of gameplay without testing. And, yes, I am going to say WITHOUT TESTING because 1.0 had so many totally broken gameplay elements that I can't imagine that anyone with authority actually tested them even once.

That's pretty offensive to the volunteers who did test 1.0. You have no idea what it was like before testing, or what things more critical to the game functioning properly did get fixed.

[Yemo]

[...]

this is getting insane now

We went from a very long alpha, to a beta (0.90) that was not a beta (not "feature complete") to a release (1.0) that was actually an open beta (lots of new untested features and 2 hotfixes within a week).

If you want to tarnish your reputation without any need or incentive, that is how you do it.

No need to scream at the closed beta testers, scream at the people mislabeling the state of the game, as that is the only issue here.

And to the closed beta testers, you are left out in the public "rain" by those that mislabeled the game state, complaining to the rain because you get wet is not going to help either.

Edited May 4, 2015 by Yemo