About recent community criticism in the direction of the QA & exp testers

[eddiew]

Let me preface my comments

...

Nothing is wrong with Poker or Texas Holdem. But gosh darn it, there is something wrong when the average person doesn't know which one they should be playing. And that's what it feels like since 1.0 hit. It's not that the changes themselves are bad. It's that I don't know how to play the game because the rules might be different next week. This kind of instability is appropriate for Early Access. Unacceptable for a 1.0 version.

That was both eloquent and very accurate, and I think an excellent summation of why a few tempers got frayed over the last week. I can only agree that the choice to not involve a community of early access players, who had already paid for the game, who were actively asking to be involved in the 1.0 testing, was a very odd decision.

I do appreciate that many fixes and patches appeared pretty quickly after launch - which is not as uncommon as it should be - but my personal worry is that the 1.0.2 changes to aero are now going to stick, despite being an obvious emergency-tweak and not really being for the better in any other scenario than capsule re-entry. We now have to face that we've either got a sub-optimal aero model forever, or that it will change again in the future and potentially invalidate existing ships for a second time. This is not the fault of the testers, since they were limited in number and presumably weren't given scripts to ensure they covered many scenarios; rather it was a lack of engaging the wider community.

Based on this experience, I would like to politely urge Squad to consider using Steam's beta-opt-in to enable interested folks to continue to help with play testing for future adjustments of this ilk before they become mainstream.

[Lord Aurelius]

Based on this experience, I would like to politely urge Squad to consider using Steam's beta-opt-in to enable interested folks to continue to help with play testing for future adjustments of this ilk before they become mainstream.

I very much agree with this. Such an approach would address many of the concerns the community currently has with the development process (i.e. ignoring the presence of the early access community to help beta test the single most important release in the game's life cycle) and help vet any future changes while still providing a stable game for new players or players who don't want to beta test. Obviously the community should be final tier of testing before release, the current system merely just needs to be extended to allow the community to get their hands on the release so the devs can see what breaks in time to fix things properly without having to rush knee-jerk fixes like 1.0.2. The QA and Experimentals teams did an amazing job given the circumstances, but there still isn't really a substitute for a large-scale community beta test to ensure that the official release is as stable, balanced and bug-free as possible.

[basic.syntax]

The discussion circle comes around again, asking for more testers, in hopes more bugs would be caught. There's close to 100 in the "experimental" test group, (I'm not one) drawn from users of this forum. That seems a fairly large number to me, a good cross-section of players who care about KSP.

Hypothetically, if you (anyone not now a properly signed-up KSP tester) had private build access and knew something was broken, did due diligence and filed the bug... I think we would still have release deadlines that can't be moved out any more, and patch notes worded like this:

We've just published KSP v1.0.1. This is a small revision patch to address some of the most noticeable bugs we encountered since the release of 1.0.

Some of these we knew about for a while, but couldn't fix in time for the release (as we approach publishing time, the risk of code-breaking and delaying the entire release outweighs any benefits in the potential fix), others we found out about during the weekend livestreams, and some others we found from your own feedback.

This patch isn't meant to cover every single bug, of course, just the more relevant ones. We're going to keep on with the bugfixing and tweaking as we move into development of version 1.1.

After seeing and filing a bug based on your hypothetical access to development builds, and seeing it appear in the 'final' product for reasons outside your control, I think you might even be more frustrated, than if you didn't know it was coming

One of the problems that interrupts the steady process of building, reporting, [deadline hits, have to ship] new build with fix [oops, missed the deadline - patch time] - is the word "deadline." Someone has to decide when to stop the internal iteration of builds, and publish. Why does it have to be.. someone?

Why not throw that decision open to the community?

-But-

What percentage must agree that a build is "release quality?" 75%? 90%?

--we need to have a poll to discuss this.

How long is the poll kept open, so that a good majority have a chance to vote? Two days? Five?

--we need to have a poll to discuss this.

Is development work stopped while the vote is taken?

--Of course not, new changes, and bug reports are still going back and forth.

So, a new vote is called, on a new release candidate build.

I'm sorry that I don't have a good answer that will fix the problems that lead to bugs and patches.

We all want the best for KSP, everyone wants updates that are consistently stable & as bug-free as possible.

[sarbian]

As it was said more than once the fact that a bug is present in the final product has nothing to do with the fact that it was spotted.

What is the point of the thread now ? Half the posters don't read the answers given by the dev or the member of the test team. And the testers can't explain most of the process since we are bound by the NDA.

But hey, why bother replying since you just want to vent and explain how you would do so much better. But even if the hate was expected I am quite astonished that some of you seems to think QA is a beach party with new toys.

[Tiberion]

As it was said more than once the fact that a bug is present in the final product has nothing to do with the fact that it was spotted.

What is the point of the thread now ? Half the posters don't read the answers given by the dev or the member of the test team. And the testers can't explain most of the process since we are bound by the NDA.

But hey, why bother replying since you just want to vent and explain how you would do so much better. But even if the hate was expected I am quite astonished that some of you seems to think QA is a beach party with new toys.

I expected a beach party and all I got was sand in my shorts.

[sarbian]

Sarbian at MechJeb is pulling out his hair dealing with this.

Sarbian is doing fine, thank you. Don't use me as tool to hit the exp team (even more so since I am part of the team).

And paying us would not change anything. We don't lack motivation to test so money won't change a thing.

Also can you please delete all my mods ? They are the product of slave work from you definition and you clearly don't support that.

[Vanamonde]

... we're entitled to a bug-free release.

There has never been such a thing, and likely never will be. If you are holding out for this ideal, you are in for a lifetime of disappointment.

In less than a week Squad had to pinch out two hotfixes...

"Had" to or cared enough to? How many other game makers do this?

... outsourcing QA to what basically amounts to slave laborers...

A "volunteer" is pretty much the opposite of "slave."

Anyway, it needs to be said again: nobody is saying 1.0 didn't have bugs, nor that you are required to be happy about the bugs it does have. The point of this thread, the only point KasperVld was trying to make, is that the testers do a lot of hard work, without pay, for the sake of making the game better for all of us. You are not required to believe they are perfect. But they are deserving of respect and gratitude, and there is no call for insulting them.

[adsii1970]

I'm going to throw my comments out for condemnation and ridicule...

we're entitled to a bug-free release.

What planet are you on? (<--- Not sure if this would count as a personal attack ) As someone who has been playing computer games since 1987, let me be the first to tell you that compared to what was once shoved out the door as a completed product is NOTHING like what has happened here. In fact, let me be the first to say that in the 80s, the latest computer game was either passed onto you by a friend on a 5 1/2 floppy OR you entered each line of code in DOS, saved the file, and hoped to God that some editor or programmer at PCWorld Magazine copied the code PRECISELY as it was given, otherwise, you could spend days debugging nearly 2000 lines of code to find that one command was wrong and it was why you could not slay the cube that represented a dragon or other target... or the program would give you the awful beep and the dreaded "syntax error" message... At least KSP has been fairly stable since I began playing it under whatever version it was when I first joined.

I have been a big fan of Sid Meyer's Civilization franchise since I first bought the original version' Civ V had so many bugs that when I installed it, first ran the game, it immediately gave me a warning box and began to download 43 patches - and that was with the game only being out for two weeks and me spending $45 US on it! In fact, SimCity 4, SimCity 3k, Simcity 2000 were all similar - had many fixes before the game could even be played halfway decently. X-Wing vs. Tie Fighter had similar issues with dll files that made playing with a joystick very unjoyful. To play any of those games, you simply did what you had to do - go to the website of the software company and downloaded the patches. At least with KSP, it does it fairly automatically - and I want to thank the team for that.

There has never been a "bug free release" of any software package much less one that is a game. I doubt there ever will.

I would love to volunteer my services to be a "slave tester" as someone put it. KSP is probably my favorite game since Civ III or Civ IV; even with some of the issues I currently experience, I am glad to have had the opportunity to see this game evolve. Just thinking about how far this game has progressed in 2 years is just truly astounding; I stand in awe of some of the graphics mods, the amount of detail and attention given by Squad and some of the modders, as well as some of the really great people I have met in this forum.

Edited May 3, 2015 by adsii1970

[Wooks]

Let me preface my comments by saying ...

...It's not that the changes themselves are bad. It's that I don't know how to play the game because the rules might be different next week. This kind of instability is appropriate for Early Access. Unacceptable for a 1.0 version.

This. Squad read this.

[Hyratel]

I've been on the Experimentals team since version 0.18.2 (that's late Jan '13) so I've been doing this for well over two years, simply because I love the game and want it to be the best game it can be. no tester is perfect, no programmer or dev is perfect. things slip under the radar simply because it never occurred to somebody that that might even be an issue. EEven now, i'm finding and logging little things that I spot in the release 1.0.2

[AlamoVampire]

I am just gonna say this much: a game, any game, has hundreds of thousands of lines of code, if not MILLIONS of lines of code by time you reach a product that is acceptable enough and close enough to the initial design ideal to call a 1.0 product. Look, if you change 1 thing in one line of code, what does that single change do for line 120343 or line 343043? do you even SEE the effect right away? is the effect good or bad? is it subtle enough to be not seen until someone not on the design team or QA team does something that was unexpected and made it pop up?

It is delusional in all sincerity to think let alone expect or demand a 100% flawless product. No product is perfect. There is always a flaw somewhere. Perhaps its so minor that it doesnt matter, like say a small bubble in the glass that makes up a glass coffee mug, or perhaps its severe, like say something that causes that glass to shatter at the first cup of coffee. Look, coding isnt easy. if it was, we ALL would do it. Squad is a SMALL design house that is infact NOT a design house by original design, it is an advertisement firm iirc. Heck, even a MAJOR design house like say UBISOFT has launched products that have had multimillion dollar design budgets with odd glitches. Doubt me? Look on google for the funny stuff that goes wrong in a number of the AC games. Just chill, wait, relax.

[Nachtwind]

As yet another senior (3 years and counting!) member of the slave workers caste i would like to give a few words on this matter as well.

I think Hyratel brought up the most important point:

We test and help improving the game for the simple reason that we love it. None of us is perfect and i am glad that SQUAD offers us the opportunety to work on KSP the way we do it. Especially 1.0 was a greta example of the dialogue between testers and devs in order to create a great final game.

As for regarding people who say they are entitled to a bug free version... just ask yourself if you have ever submitted any bug report to the forum or if you have just keptplaying/restarted the game. There is a lot each and every user can do to help improving the game - but flaming is not a valid option.

[SpenSpaceCorp]

I just want to say, I (along with others) have recognized Squad (and the experimental team, plus everyone involved) having gone above and beyond in the 1.0 update! While there are a few minor bugs and changes that may not be welcome (ex: I LOVE the 1.0.0 aerodynamics but do not entirely like the 1.0.2 multiplier changes), the overall product is shining of the excellent work put forth by all those involved. while there is still room for improvement, the hard work and effort put forth really shows and IS recognized.

So amidst some of the negativity and not-so-constructive criticisms, I would just like to thank and congratulate Squad as well as all those who have tested, experimented, and contributed in any way to the full release of the game.

Great job, and keep up the excellent work!

[samstarman5]

This. Squad read this.

CC @Scourge013

Squad is already aware of a lack of a certain amount of help for new players, and have stated they are working on a full tutorial and scenario overhaul for the game. Yes, it wasn't done in time to be released with 1.0, but I hardly call that a reasonable excuse to be up in arms over that. There is still a plethora of videos and guides online to help, made by slave laborers(since that term seems to be popular in this thread, I use it only in jest, of course) out of their own time with no call from Squad to do such.

If you are honestly expecting a manual, then I have to ask...what game in the past 20 years has actually released a manual with their game (that at least was actually expected to be read)?

The closest thing KSP has to a manual is its wiki, which has admittedly fallen into neglect. One thing to remember is the wiki is not run by Squad, but by regular people who contribute what they learn. You are more than welcome to do your part to help bring it up to date, as is anyone else out there who have rode KSP through the Early Access ride and can contribute relevant information to help the next guy.

There is a sad explosion in the number of self-entitled players who feel the need to point out problems, bugs, or things they don't like (even if those things actually work), but will do nothing to actually help remedy the issues. Criticism is fine, as it tends to mean that problems are being pointed out with details laid out to help find the problem. Complaints are just pointing out problems or disliked features with zero submission of details, ideas, or even civil response. If we actually want to see KSP become a functional and fun game for at least most players, then we need to see a lot fewer base complaints and more actual contribution to finding the bugs.

Just because KSP is out of Early Access, that doesn't mean we have to stop chipping in to help fix the issues. It sure as heck doesn't stop players doing so with games that never see Early Access. If you can't contribute constructively to issues with the game, bugs, glitches, and exploits, then keep it to yourself. You are only exacerbating the problem by making it harder for the devs, QA testers, and Experimental testers to find issues to track down so they can be fixed.

[brienne]

Squad is ok. They released KSP 1.0 and made several fixes with 1.0.1. Ok.

But why did they release 1.0.2 so quickly ?

Looks like a panic patch, and Squad missed.

Can't have any Testers advices after 2 days^^

Edited May 4, 2015 by brienne

[r4pt0r]

Squad is ok. They release KSP 1.0 and made several fixes with 1.0.1.

But why did they release 1.0.2 so quickly ?

Looks like a panic patch, and Squad missed.

Here is the change log for 1.0.2:

Bug Fixes and Tweaks:

Thermal:

* Fixed ships potentially overheating when splashed down.

Parts

* Small tweak to Mk16 parachute drag.

[DunaRocketeer]

OK Squad, you think our attitude sucks? Frankly I think you're completely wrong, and it's you who have been offensive, not us. There were several very obvious bugs/bad implementation with 1.0. The QA testers should have spotted them in 5 minutes, because the community sure as hell did.

Next you rush out a quick series of 'hotfix' patches of that are causing memory leaks.

Are you feeling burned by the fan community's negative reaction? I suggest you look in a mirror to find the root of the issue.

[impyre]

Admittedly, I didn't read 15 pages of responses... but here's my opinion for what it's worth.

I saw a few bugs... but it was largely a vast improvement over .90.

I don't believe that crying, whining, or being aggressive helps anything.

I seriously doubt moderators or Squad reps can come into our community, one that's built up around this game... sure... but still made up mostly of fans, and with a few words cause people to stop being the way they are. You cannot control others, and I doubt that being whiny or angry is against forum rules (nor should it be in my opinion) it's just people sharing how they feel. If someone gets nasty and posts things that violate the rules, that's one thing... but it seems silly to come in here and essentially say "Stop being so mean!". If what you mean to say is "Hey, we have feelings too... we don't get paid for this... we do it because we think you guys are awesome... and we're doing the best we can, so stop being so mean"... well, that's a different story... but you have to accept the fact that people will still be who they are (for better *and* for worse).

Lastly, there are probably some people that it's best to just ignore... I'll leave it to you to figure out who they are. I haven't had any problems with what squad has done so far, but trying to control free discussion seems a bit... much. It'll blow over, just do the best you can... I think that's all most of us ask.

Oh, and the percentage of people who have complaints that post them is far higher than the percentage of people who don't have complaints post just to say "no problems here".

[AbhChallenger]

We don't know IF the two teams spotted the issue before 1.0 was out. Because of Squad's (Rather silly in my opinion) NDA.

Personally I feel a good middle ground for 1.1 is simply allow steam beta say a week or so before planned launch. Anyone complaining about changes that happen between steam beta and steam public will be branded a fool because the beta will clearly say that it is for testing ONLY.

Yes that does mean absolutely crap bug reports (Oh noes my rocket flipz!! HALp!!!1) But it will allow a little heads up if an issue somehow made it past the two teams that are under NDA.

IF 1.1 is indeed the great move to Unity 5. I feel that all the testing that is possible should be allowed.

[Nachtwind]

There were several very obvious bugs/bad implementation with 1.0. The QA testers should have spotted them in 5 minutes, because the community sure as hell did.

Just between the two of us... Ted once wrote a nice article on how QA works, therefore i wont break the NDA here right now.. but dont think we have access to the same product you receive when it goes public in QA. We test "branches" - more or less one feature at a time.. only in Experimentals the game is "nearly" comparable to the final product and even then only a few dozen testers can test it and first have to run down a lengthy list of bugs to test against.

Sometimes tracing a bug, even severe ones, takes hours - and we are not speaking of fixing it at this point.

So with limited time and testers only a limited amount of bugs can be found, especially since we test on multiple platforms and hardware sets all the time. Some bugs happen on AMD GPUs, others only on german Linuxes and yet others happen to everyone.

Gladly we can say that we find a lot of bugs during QA and Exp but of course some can slip and if yuo crunch some numbers its obvious why:

Say 1% of all windows 7 x64 users have a certain bug - that makes maybe 1 person in Experimental experiencing it.. yet when the game is released there can be hundreds who have that bug and think it went unnoticed. But no. If we only have like 50 people on that configuration the chance is quite low womeone experiences it.. but if thousands of people run said configuration a lot of users will see it.. And how many exactly report it the way it would be right? Not all too many..

Therefore i would like to ask of you some understanding of how things work and how we do our stuff.. for the past three years it worked very good and fpr 1.0(.x) it worked great as well, especially in 1.0.ish since i never saw Devs and Testers work that closely before.

unfortunately NDA prohibits handing out old testing versions or even changelogs/buglists.. but regarding this ....storm it would be nice to show some evolution from first QA versions down the road to 1.0 final.. and many people would understand and accept the work that was put into this seeminly "rushed" release.

But even that wouldnt satisfy some noisy people i guess.. so we wont put our efforts into this than rather keep improving he game beyond 1.0...

[Temeter]

OK Squad, you think our attitude sucks? Frankly I think you're completely wrong, and it's you who have been offensive, not us. There were several very obvious bugs/bad implementation with 1.0. The QA testers should have spotted them in 5 minutes, because the community sure as hell did.

Next you rush out a quick series of 'hotfix' patches of that are causing memory leaks.

Are you feeling burned by the fan community's negative reaction? I suggest you look in a mirror to find the root of the issue.

It's not the fan community's negative reaction, but a bunch of drama queens. Just to keep things in perspective. On other forums the voices are like 99% positive.

[AbhChallenger]

............

Gladly we can say that we find a lot of bugs during QA and Exp but of course some can slip and if yuo crunch some numbers its obvious why:

Say 1% of all windows 7 x64 users have a certain bug - that makes maybe 1 person in Experimental experiencing it.. yet when the game is released there can be hundreds who have that bug and think it went unnoticed. But no. If we only have like 50 people on that configuration the chance is quite low womeone experiences it.. but if thousands of people run said configuration a lot of users will see it.. And how many e.................

This shows in my opinion that a steam beta as a "third stage" of testing would be beneficial. Let folks not under NDA test for a week or so before it comes out. They may find nothing. They may find the showstopper that slipped past the two other teams.

[Nachtwind]

This shows in my opinion that a steam beta as a "third stage" of testing would be beneficial. Let folks not under NDA test for a week or so before it comes out. They may find nothing. They may find the showstopper that slipped past the two other teams.

The question just is: How useful could that be in the end?

Lets face it - the general community is pretty lazy when it comes to submitting bugs, are they not? Also versions can change a few times per day on qa/experimentals so how to keep track of users and their bugs? People may submit already solved bugs and keep people busy finding this stuff just because they were no longer up to date. It may sound trivial, but that would be a major problem i guess.

Its just the old manpower problem. What 10 people do in 1 hour cannot be done by 600 in one minute.. There is a certain threshhold beyond which the amount of testers would become ineffective if not a burden on the (streamlined) process that has been established a few ages back.

Of course people could find gamebreaking bugs, but they'd so so on the final release... and then? Well, if you create a hotfix one day after release or during the "third stage of testing" would come to the same point: You have a fix for a problem, dont you?

I dont want to be rude or arrogant here, but in general the whole testing business is quite streamlined and people work like a clockwork during testing periods and just putting a ton of unexpierenced testers could be a burden on our back rather than help.

[RAINCRAFTER]

Admittedly, I didn't read 15 pages of responses... but here's my opinion for what it's worth.

I saw a few bugs... but it was largely a vast improvement over .90.

I don't believe that crying, whining, or being aggressive helps anything.

I seriously doubt moderators or Squad reps can come into our community, one that's built up around this game... sure... but still made up mostly of fans, and with a few words cause people to stop being the way they are. You cannot control others, and I doubt that being whiny or angry is against forum rules (nor should it be in my opinion) it's just people sharing how they feel. If someone gets nasty and posts things that violate the rules, that's one thing... but it seems silly to come in here and essentially say "Stop being so mean!". If what you mean to say is "Hey, we have feelings too... we don't get paid for this... we do it because we think you guys are awesome... and we're doing the best we can, so stop being so mean"... well, that's a different story... but you have to accept the fact that people will still be who they are (for better *and* for worse).

Lastly, there are probably some people that it's best to just ignore... I'll leave it to you to figure out who they are. I haven't had any problems with what squad has done so far, but trying to control free discussion seems a bit... much. It'll blow over, just do the best you can... I think that's all most of us ask.

Oh, and the percentage of people who have complaints that post them is far higher than the percentage of people who don't have complaints post just to say "no problems here".

*Claps* I also have no problem with the QA team and the Exp team because they are volunteers. They expect hard work out of washing this release from it's bugs, and they do miss a few so they don't have their whole day consumed by just one thing. Also, I'm guessing that 90 is the new number of workers working on bug fixing a inde game suffering from not enough bug fixers, at least according to the offensive post everyone is quoting from.

[sal_vager]

Squad used to release the experimental versions to the community, and back when KSP was small with a close knit community of technically minded players who understood the game was in development and would contain bugs, this was perfectly fine.

Then KSP gained wider appeal, there were more members, the concept of early access to an unfinished game eluded them.

0.14.0 was released.

It was an absolute, unmitigated and complete disaster.

The forums were absolutely hammered with endless screaming and abuse, many players expected, demanded, fully working and playable versions from a game in the early days of development.

What we've seen the last few days has been nothing in comparison.

Squad realized then that public releases of experimental builds were no longer tenable, the testing team was set up and only versions that had passed them would go to public release.

0.15 was playable, so was 0.16 and 0.17 and so on, every update has been playable.

If you were given the experimentals, or worse, the QA builds, and saw everything break and all your saves and craft destroyed you'd flip your lid, the testers know they can't expect things to work which is why they are on the team.

Thanks to them you can actually play KSP.

Public access to experimental builds is never coming back, and with good reason.