About recent community criticism in the direction of the QA & exp testers

[pandaman]

Having previously done some bug reporting as part of my of job role (expensive commercial sofware, not games) I can say that it's not as easy as it may appear. Spotting a bug, or something that doesn't look right, is the easy part. To be of any real value to the developers you need to analyse what you were doing when it occurred, then taking time to try and recreate the issue, and then document it as fully as possible so that the developers themselves have as much detail as possible so that they can recreate it to analyse and find and fix the causes (not an easy task for volunteers to achieve in free time under time presdure to get it done).

That is why a relatively small group of testers, with the time and ability to report effectively, is of far more value than a community wide 'play and tell us what doesn't work'. Don't get me wrong 'throwing it to the monkeys' is a good way of pointing out lots of issues quickly (especially obscure ones), but unless they are at least reported sensibly they can be of little value, and sometimes a distraction, to the fixing process.

Overall a good job on v1.0, thanks to the whole team that made this awesome game.

[Wallygator]

I've tried to be sensibly and constructively critical of any output and or manner where decisions and processes are perceived to be faulty - this has never been about people from my perspective and comments. I'm sure everyone in the QA and Exp teams is doing their jobs to the best of their ability - and likely with little or no pay! I would buy them all an espresso, but I can't afford it or the air fair at the moment

That said, I'm guessing here (but probably close to being accurate) that the core KSP team has been and is continuing to deliver a complex product with an expert small group of individuals. There is likely little time to spend documenting processes or improving delivery quality, measures, tracking and reporting, etc. All the while knowing that delivering quality software is a complex endeavour that boggles even large global organisations (Microsoft anyone?). Doing it successfully as an Idie team is close to freaking miraculous. (I help improve programs and organisations that deliver tech and SW as a profession - nobody does everything perfect - Nobody!)

As a community we all should consider re-thanking our Squadies for this incredible road they have traveled, recommit ourselves to being a bit more focused in our criticism of product and process, and staple notes to our foreheads that remind us to absolutely refrain from personal attacks.

As for Squad, you all might take learnings from this overall 1.0 event as well. How you communicate (when and where) matters a lot. Squandering trust is easier than building it. Thankfully, you still have a strong committed core of supporters (AKA critics) that can still contribute more than just comments - they might even feel that have a stake in the final design from years of input and support. Further opening up your design decisioning processes for comment prior to releasing features might be something to consider - but as always, it's your decision to make. (For Example: creating a carefully selected cross sectional group of key users to act as an external design review board - confidential NDA, non-binding decisions of course - maybe you already have that - would be nice to know about it).

So from me... Thank you! I love 1.0 (Please post that on your Internal Jumbotron if it exists, and then get back to something more interesting like fixing those pesky bugs... and revamp career mode, and those fairings, and... and.... and...)

Edited April 30, 2015 by Wallygator

[Merinsan]

I want to thank the testers, the devs, and the modders brought on board for delivering a pretty damn good product for such a strangely rushed release. Certainly things were missed but that happens all the time, and I can especially see it happening with a sandbox game as complex as KSP.

This update feels like one for the long-term customers, the people who have been here a long time tinkering with this game. I'm really happy to have it in my hands as it brings some much-needed corrections and systems to the game, making it feel very much fresh again. There's still a lot of work to be done and I'm not entirely certain I'd call this one 1.0 (maybe the next, after a bit of polish), but good lord, is KSP fun again.

Completely agree with Regex here.

I've been on a fairly long break from KSP, because I'd mostly mastered it. This update makes KSP feel like a brand new game, which is a good thing, but at the same time it comes with issues. Yes, it's not as polished as it should be, but it's a lot of fun!

[ShadowZone]

@Squad: I know your plight. I am working as Product Owner for a software development company. No matter how hard you test your system, there can always be bugs. It's not easy to unite required scope and targeted deadline.

And personally: I think 1.0 is a tremendous update! There are so many new additions to it and the performance and stability improvements alone would have made me pay again for the game if that would have been necessary.

That being said I do have some issues with how certain things in the game turned out. And yes, there are bugs. But on the other hand, I only today, through KasperVld's opening post of this thread, found out that there is a dedicated bug reporting tool. And I am using the forums for months now.

So yeah, I think there is room for improvement. And maybe there are even some bugs the QA team found but couldn't be fixed until release. Who knows? I am waiting for 1.01 and I am definitely not hating on the devs or the QA team for anything. KSP 1.0 is a great update und I love playing it. And I know you will get around to fixing all those bugs

[smart013]

I want to thank the testers, the devs, and the modders brought on board for delivering a pretty damn good product for such a strangely rushed release. Certainly things were missed but that happens all the time, and I can especially see it happening with a sandbox game as complex as KSP.

This update feels like one for the long-term customers, the people who have been here a long time tinkering with this game. I'm really happy to have it in my hands as it brings some much-needed corrections and systems to the game, making it feel very much fresh again. There's still a lot of work to be done and I'm not entirely certain I'd call this one 1.0 (maybe the next, after a bit of polish), but good lord, is KSP fun again.

100% agree. My day to day playing experience (note the word "play") has massively improved to .90. The update to end all updates? No. A great and necessary step forward? Yes. In my opinion we would have had the exact same amount of more or less fair criticism when 1.0 hits - mostly from the same people i suspect - even after .91/.92/.95 or whatever. It lies in the nature of the community and their view of themself that every release of 1.0 would maximise drama.

The now possible quatum leaps for nuFAR, deadly reentry and the complete works of roverdude, let alone the imho massive performance gains of 1.0 is more than enough for me to keep smiling when thinking about the future of ksp.

3c from me.

Edited April 30, 2015 by smart013

[jarmund]

I for one like 1.0 for what it is. Tremendous amounts of content/features added, with the most serious bugs fixed.

To the critics: Yes, there are still bugs in the game, but how many 1.0 releases in general have you seen that have been completely bug free?

The more complex the code base, the more likely it is that bugs creep in. I do not know how many lines of code, objects, or function calls the KSP source tree involves, but judging from the product, the outstanding issues are minor.

In other words: It's 1.0. It's not expected to be perfect, and there will be updates and fixes along the way. If 1.0 is so horrible as some of the claims I've seen, I'm sure it's possible to stick to 0.90.

[Cdr_Zeta]

Hello,

Testers is a great idea for those who spend endless hours lookin for bugs and improvements; they dont get to play the game like we do; I am a good beta tester and even asked to be on the list once but I was hesitant; this is the one game I want to play; not fix.

If I missed something in any negativity I am glad.

Zetadude

[Watoh]

I feel for the testers etc.

A few relevant facts I've learnt over the years:

Software is rarely released without bugs.. no matter the budget or size of the DEV team

No 'simulator' game has EVER been released without bugs.

In a large forum community there are always prima donnas who make a big deal over issues and bark at the top of their virtual voices, often drowning out the more rational majority.

Game testers generally put in long hours with little reward and do it because they are passionate about the product.

Game developers always like constructive criticism and hate rants where users act like the world has fallen in.

Laters x

[SelectHalfling0]

I really could care less about bugs. In an update this large I was expecting them, and there were very few (I've only encountered 2). I can only imagine what Q'A ran into.

[John FX]

Just a thought, why not have a hotfix pre-planned in the release schedule for every release?

Keep all the current QA and experimentals but just after it is released, "throw it to the monkeys and see what breaks". Then address the worst bugs and easiest to fix bugs in a hotfix?

If it is not possible to catch all the bugs and it is known that some easily fixed bugs which would ruin most peoples gameplay (like for example pods with heatshields flipping around and exploding, myself I can't understand how that one was not caught by 200 testers but still...) will make it through due to the fault of nobody then why not explicitly state (as was done retroactively with 1.0) that a hotfix will follow every release and people should submit any bugs they find and SQUAD would commit to addressing them *before* the next major update?

I feel that a lot of the frustration for some people comes from their view that the bugs are staying until the next release, which could be many months away. An expected hotfix would allay a lot of their worries IMHO.

This does not excuse being rude of course, people should be nice and submit a proper bug report instead of just moaning.

TL;DR have a minor update (hotfix) follow every major update to address things not caught in experimentals.

[michaelsteele3]

There is room for improvement, and the QA testers did there best. Overall 1.0 is amazing, and there would be more bugs if the QA testers didn't do anything. So let's all be glad that we actually have a QA team.

[StainX]

People blame QA team, thats hilarious!

Its always people taking critism on stuff although they don't know whats happening behind the curtains.

I think its a pretty solid build, when considering the mass of improvements and features added to this update.

Although i understand that people are complaining about stuff like the pod re-enter issue, but i'm sure you people working on that already.

Cheers

[Grunnt]

If you look at this from a meta perspective, you could also consider the emotional reactions a (big) compliment: apparently some / many of the players have become deeply emotionally attached to this game. If KSP were really a bad game nobody would care enough to post heated comments about how terrible it is that <insert bug here> is in the game.

Overall I think the 1.0 relase was a great success. Good reviews, 10/10 on Steam and for me personally I'm having a blast with it. The balancing, aero and heat changes bring so much life to the game, much more than with a modded 0.90.

Also, for players bothered by some of the bugs in 1.0, there's a good bugfix mod which helps. I have no doubt that these issues will be fixed soon enough.

[Kaezan]

Article? Link please? My google-fu has failed me.

This article was an explanation of what and why.

http://forum.kerbalspaceprogram.com/content/326-Beyond-Beta

[John FX]

[Trimmed to keep the discussion on-topic.]

As the bit I wanted to say has gone, I'll just leave this here.

Edited April 30, 2015 by John FX

[TriggerAu]

I'm always amazed by the depth of feeling that is generated by the KSP Community, and also by the Devs as they work on KSP. As a member of the QA test team I feel pretty privileged to be able to see the work that goes on in both QA and Exp testing and did want to comment on this thread, hopefully in a way that doesnt breach the NDA I'm under and get me in trouble.

The members of the test teams we have are all volunteers, all working whatever hours they can spare from their day job to help make KSP the best game it can be. And when I say whatever hours they can spare I reckon I'd be scared if they were paid employees and HR came a knocking about work hours/conditions, some of these guys put in a lot of time. They do this for the same reason that people raise concerns about 1.0 naming and about release quality, they really love KSP and want it to be the best game it can be. Re: the size of the teams, you can easily see that QA team size in the credits (15 names) and the Exp team is considerably larger (not sure if I can give that number away). The number that always blows me away is 6. Why? because there are only 6 devs working on this game, and not just on the game itself. The amount of content in the features that is worked on, along with the volume of information that the testers create, that is triaged and handled by such a small team is to my mind pretty impressive. Not every bug may have been caught, but some pretty impressive adjustments/fixes are made in each release. Increasing the size of the test teams would help to bring in more information, but could it be of a high enough quality so as not to drown the dev's, and thats one of the balancing acts that needs to be handled.

In every test period I have been involved in I think I have thought at one point or another - wow, thats a lot of stuff to work through - and this one was no exception. Tiberions maths about how many moving parts there are is an interesting thought as I have seen a lot of the feedback, issues, testing, resolution, discussions, etc that have been covered during testing, and making assumptions about how/why specific things make it to a release build (there were 125 builds this time round by my count) isnt going to help anyone. This includes processes to check the quality of an item is up to scratch for inclusion, or the evaluation of criticality for a bug to be remediated now or later and other processes. Rest assured that Ted, the Devs and all the testers are constantly thinking about better ways to improve the testing process, provide better feedback, and help identify things that are bugs, could be tweaked, could be improved. For those people who do want to understand more about the way Ted transformed the testing experience he wrote this article a while ago: A Peek Behind the Kerosene Curtain: All About QA and Experimentals . And of course if you do want to be involved read up on Kaspers links and keep an eye out for any tester application invites.

Personally I wasn't happy with the "1.0" badging, and feel that if versioning simply used a number and version 27 was the one that was not Early Access then it would be less of a discussion both before and after the fact. That said, if calling it 1.0 gets KSP more noticed, more sales and therefore more "fuel" for Squad to develop it further maybe we are the ones hung up on the 1.0 name. We, the people involved in the forum, should be telling people about how many times Squad have quickly fixed things from feedback, listened to the community and report things as well as we can to help out.

Lastly I'd say this - we have a community that is pretty darn fine IMO - I dont know anywhere else that brings together people of such disparity who are happy to talk/help each other with topics ranging across - How does orbital mechanics work? What is DeltaV? Why does this fly upside down? and my personal favorite "How can I make this explosion more insane?" - the strong attachment people have to this game is awesome, and remember that the people we are talking about/to also have that strong attachment, so before you hit that post button take a second and think "How will the person I am talking to/about take this?"

[Jonnyp]

QA team are the mutts nuts!

Squad does not communicate well regarding bugs.

They have a public tracker, which the Devs, QA team and Exp team are active on, as well as an active support forum, What more do you want?

[Vanamonde]

A fair number of off-topic posts have been removed from this thread. Please, do not use this thread to list the bugs that are bugging you, nor to repeat the argument about whether or not this release should have been numbered 1.0. The topic here is the work the volunteer QA testers have done to make the game better.

[henryrasia]

See if I understood correctly:

The QA team look for and report bugs that have to do with textures, broken mechanics, crashes, glitches, missing assets, things quite bluntly not working, etc, while following a guideline for substantial notes on the issue (to avoid "KSP crash SQUAD pls fix ASAP k thx bye" syndrome)

Therefore, they fail to catch mistakes on the big picture of the gameplay, independent of version, OS, etc, that the community could catch easily and wouldn't need any specific training.

I propose an approach close to what Arma 3 does, which has a "dev branch" wherein you can get the new features (at the expense of lots of bugs) and you can file issues (bugs or otherwise) using the issue tracker. It's like an automatized version of what the "feedback and suggestions" subforum used to be back in the day. I think that way we can improve the game faster and better and make everyone happier.

[basic.syntax]

See if I understood correctly:

The QA team look for and report bugs that have to do with textures, broken mechanics, crashes, glitches, missing assets, things quite bluntly not working, etc, while following a guideline for substantial notes on the issue (to avoid "KSP crash SQUAD pls fix ASAP k thx bye" syndrome).

They have two teams: I think your concerns are generally addressed. The QA team is a small group internal to Squad with high knowledge of the game's inner workings. The close-to-100 "Experimentals" team is drawn from player volunteers, with less expectation that they understand the game's inner workings. Take a look at the "A Peek Behind the Kerosene Curtain" linked by TriggerAu for a more detailed explanation. Some players wish for the "Experimental" stage builds to be thrown open to anyone: a third level of testing. That is where your comment about weakly documented bug reports would apply. Players do have access (link in first post) to a bug reporting site.

[DrD]

Well said KasperVid. An important part of leadership is taking more blame than credit.

I've been in gaming since the late 1980's and almost every release has bugs, sometimes more than others. As long as it's not too bad and fixed quickly it's not a big deal.

I've learned to deal with it simply by rarely buying/playing a new release until the first patch.

Early access is different because so many have been playing the game already for so long. If you complainers had never played KSP before Monday I bet you'd be singing a different tune.

Of course I agree legitimate feedback is ok and beneficial but let's all remember how great this game is and how responsive the developers are. Take a deep breath and wait for the patch. Remember how much time with family and friends the developers gave up to bring you KSP and be appreciative. Yes I know it's their job but everyone deserves respect and recognition for a job well done, even if (or especially if) said respect and recognition is accompanied by constructive feedback.

Meanwhile the British Empire is under attack by the HRE in 1402 so that needs my attention for awhile yet! (CK2).

[RW-1]

I've never had a bad word for Squad, nor the people who were given the opportunity to test out the software pre-release.

I believe in that Squad, for whatever reason, fell victim to their own hypetrain for a 1.0 release. don't know the exact reasoning, and don't care:

I got an update to a terrific program, and I didn't have to pay for it. Plain and simple. I certainly have received my payments worth a couple times over. already.

There is so much put into this, that I don't see any release that comes out that will "fix" everything without imparting other bugs, nuisances or annoyances. I consider them challenges until fixes are in place.

Humans are involved, mistakes can be made, the re-entry heatshield thing seems to just have been an omission - flip a 1 to a 0 and fixed (unless that was the temp fix and the actual one is more involved.)

Using that one as an analogy here, yes, it slipped past, but throwing out curses and being negative on the BB doesn't do squad squat (see what I said there? )

Make the report, send it in and play. Hell there are so many here that are fairly expert with the configuration and code of the software that someone already had a temp fix in place within a day.

Yep DrD, even "releases" .....

Lastly, IMHO, if you have had the game for a long time and are complaining, my thought is "you've played how many iterations of this without paying extra for it? hmm?" Squad could have just said that 1.0 would not be covered for early adopters, and with all the clambering (see what I said there? ) going on about this release, who says the next one needs to be free to all of us?

I look forward to playing with 1.0 shortly (haven't been able to for a couple of days) and look forward to 1.0.1 or whatever you guys decide to call it.

[DoToH]

I only have a big THANKS for QA and experimetals testers. I can imagine the work they have done.

Unfortunately, I can't talk about the underlying cause of this criticism (IMO) because...

Please, do not use this thread to list the bugs that are bugging you, nor to repeat the argument about whether or not this release should have been numbered 1.0. The topic here is the work the volunteer QA testers have done to make the game better.

^^ this gives very little margin to some opinions (and I'm a good boy who doesn't want to be banned)...

[TriggerAu]

...The QA team look for and report bugs that have to do with textures, broken mechanics, crashes, glitches, missing assets, things quite bluntly not working, etc, while following a guideline for substantial notes on the issue (to avoid "KSP crash SQUAD pls fix ASAP k thx bye" syndrome)...

Basic Syntax is correct in that there are two teams, and the focus of the Exp team is much more around the whole gameplay experience and trying to test from Go to Woah, and both teams contain peopel on different OS/Architecture/setups - is part of the application process so ted can try and balance out the process. Having that variety and team differences is one of the things we try to balance in the whole process, QA Shouldn't pass a dodgy feature to Exp, so Exp can test the whole picture.

Your idea of a Dev branch is something I have heard floated before, and it could work, the challenge then becomes, quality bug reports and assigning dev resources - so you need more admin staff. Is definitely an idea that has been batted around before too, but I assume it didnt get legs previously or thats what we would see.

...The QA team is a small group internal to Squad with high knowledge of the game's inner workings. ...

Just to be specific here, we are volunteers too, not internal to Squad. As for the high knowledge of the inner workings, thats all commitment and time

[Paul Kingtiger]

Maybe some of the frustration is not with the testers, but with the review / publicity need for the release version to be as bug free as possible.

I've encountered a few bugs without even leaving the pad and have watched various twitch streams also find bugs relatively quickly. Bugs that might have been found and fixed with either a longer experimentals phase or even better a release between .90 and 1.0.

None of us want the full release of KSP to be reviewed as bugged and I suspect that is where the frustration is coming from.

Personally I think the new functionality of 1.0 is excellent, but that Squad missed the opportunity for the large and loyal KSP community the help bug hunt a 0.95 release in preparation for highly polished 1.0.