How will bug reporting be managed for version 1.0

[katateochi]

Version 1 is going to be totally bug free right? Well, that would be a nice dream, but no software is bug free and KSP 1.0 is certainly going to have bugs and we will need to be able to report them. So...

- Is the bug reporting process going to remain the same? (kind of confusing to users and a horrible mess for developers).

- Will the bug tracker have the slate wiped clean (archived) to make it easier for those reporting bugs to search for existing reports that cover the issue they want to report?

- Will there be a new approach to how bugs are reported in order to deal with the increased user-base and to prevent the bug tracker being overrun with poor quality and duplicate reports?

In the discussion Do you feel KSP is ready for 1.0? Maxmaps said something that bothers me, but I can also understand where he's coming from;

We've had terrible experiences with the signal to noise ratio of any broad-facing testing environment. Sorting through reports becomes a mammoth task when the overwhelming majority of them are of extremely low quality. The amount of extra overhead this causes ends up having a worse effect that just staying with our core group of really experienced, super efficient QA Crew. Branch testing for QA started as soon as it was possible, and has gone on since. Add to that the fact that we're effectively scheduling the longest experimentals period in the history of our game, and we can be reasonably certain that most major things will be caught by then.

What I read from this is they are fed up with dealing with our (generally poorly written) bug reports and it sounds like the solution is a "hear no evil" approach - stop taking bug reports from the wider community and focus on just what the QA team comes up with. This is partly why we never saw another beta release, because several beta releases where done internally and tested by the QA team and not the unruly rabble (us). But does this still apply as version 1.0 comes out?

I can understand that the dev team has been overwhelmed by the existing bug reports. Many are poor quality "itsh na workin'" type reports, many are duplicates, there's a whole bunch of feature requests mixed in, I've seen several that are just people dumping crash logs without any explanation, etc. These all add overhead to sort through before the poor devs can actually find one they can do something about.

So who's fault is it that bug reports are of a poor quality? Well, it's our fault! But it's also Squads fault for expecting a wide internet community to do anything different.

What is the solution? To urge users to make better bug reports, do thorough searches before reporting and perform their own tests to make sure the bug is actually a repeatable bug etc. Is that going to happen? Well in this community is has a higher chance of happening, but generally no, there is still going to be a big mess.

The solution needs to be in the form of a better better approach to bug tracking. I think Squad's mistake was to open up the bug tracker for all of us to do what we did all over it! They should have kept that as an internal tool and had a process for taking user reports in the forum and adding them to the tracker in an ordered fashion and then marking those forum threads as tracked. That at least would keep duplicates and poor reports out of the tracker while still enabling users to report their problems in the forum.

So my suggestion for bug tracking in the future is;

- we no longer have write access to the bug tracker (maybe we still have read-only access).

- we report bugs in the support section of the forum (as we already do)

- A select group (most likely the current moderators) are able to write to the bug tracker and they take bugs that have been well reported in the forum (ie with several users saying they have that issue, example cases where the bug occurs etc) and make an entry in the bug tracker. then update the forum thread to mark it as tracked.

- It is still up to us to do our best to find existing reports in the forums and try to limit the number of duplicate reports, but at least the developers bug-tracker is protected from the mess.

Problem with this approach is an increased burden on the moderators. People like Claw work very hard to maintain the support section and reduce duplicate threads and direct people to existing threads, this would also require them taking a further action of converting threads into bug-tracker entries and having a process that enables them to know what threads have already been added to the tracker.

Taking this further; I don't think the forum is really a good tool for bug reporting or for Q&A. It takes a fair bit of hunting before you can be sure that what you're reporting/asking hasn't already been posted and when you do find an existing post finding the best answer can take a while too. I think a lot of users can't be bothered with reading through all the replies so it's easier to just create a new thread.

What would be better than the forum?

Something that automatically searches existing threads for similar content when you start to create a new thread and then directs you to existing threads rather than creating a new one. A system with ranking that brings the best answer to a particular question to the top so you don't have to read a whole thread to find the solution, where popular questions are ranked up so it's easier to find the common issues and equally where "bad" questions are ranked down. Where users gain rank based on them asking good questions or writing good replies and their increased rank gives them different privileges to do things like edit other posts, flag posts, create new tags etc.

Does such a system exist? Yes, it's called StackExchange, originally setup for programming questions (StackOverflow) it now has StackExchange sites for many topics; cooking, car-repair, general gaming, lego and tons more. As a community we could setup a new StackExchange dedicated to KSP and it's totally free to do so. The only thing is a new StackExchange needs to gain a certain following before it becomes accepted. Someone did already try to set one up for KSP but no one followed it and it got closed. There is a gaming.stackexchange that has about 200 questions tagged as kerbal-space-program. Here's an example of a commonly asked question http://gaming.stackexchange.com/questions/97324/how-do-i-clean-up-this-space-town note the best answer is right there under the question.

With a StackExchange site my suggestion for the bug reporting process is slightly different and should have less of a burden on the moderators:

- Use a StackExchange site to replace the current support section of the forum to ask "how do I" type questions and to report issues. The task of weeding out duplicate questions/reports will be automated and this will free up moderator time.

- we report issues to the StackExchange site and tag them as a bug. StackExchange will automatically suggest similar existing reports of the same issue and we can vote up existing reports rather than creating duplicates.

- moderators can then monitor the unanswered questions that are tagged as a bug and they can use the voting system to see which reports are "popular". When a report has gained a certain rank they can add an entry into the bug tracker in a structured format and tag the report in StackExchange as tracked.

If a move to StackExchange is too much of a leap I still think that the process of reporting bugs needs refining. The goal of that must be to continue to enable us to report issues, but without drowning the dev team in poorly made reports and duplicates.

If there is any interest in taking the StackExchange option then I'll set one up. It will then be up to us to use it so it gains enough following to become an accepted StackExchange. New StackExchange sites start their life in Area51! Here's a faq about how a new site becomes accepted http://area51.stackexchange.com/faq

And here's a link to KSP tagged questions on the existing "Arqade" site so you can see how it works - http://gaming.stackexchange.com/questions/tagged/kerbal-space-program

Discuss!

TLDR;

- in light of max's comment; is community bug reporting still being monitored?

- is the bug reporting process still the same as we move to V1.0?

- will Squad provide better guidelines for bug reporting or a different approach to how bugs are added to the tracker?

- the forum is not really adequate for Q&A or issue reporting and I'd suggest using a StackExchange site instead.

[Vanamonde]

Long ago, the volume of bug reports grew so large that Squad mostly stopped responding to them on the forum. This was not a policy decision, nor does it indicate that Squad stopped caring. There was just too much for individual attention. They are aware of the bugs players experience, though. You might not see that until they get around to fixing the one you care about, but that doesn't mean they don't know about it or aren't working on it. Also, remember that the forum moderators participate in testing and help players with bugs, and we read just about every thread on the forum. So Squad gets feedback from us as well. We can and do confirm the existence of bugs, state our opinions about their severity, and so on. Like you, we rarely get specific replies. That's just the nature of the situation.

I don't believe there are any plans to change the system due to a version number, even if it's a version number with a great deal of symbolic significance, but that is not to say that the system couldn't be improved, so thank you for these suggestions, and feel free to offer more.

[Xavven]

I'm pretty happy with Vanamonde's explanation, and I have nothing to add to your direct question about how bug reports should work. But I will say that I noticed that when Squad solicited applications for people to join the experimentals team that it sort of seemed like they created another layer of "early access". It's as if this game became so popular that they needed beta testers (experimentals) before it's released to the beta testers (us). And perhaps Squad starting viewing us as real consumers, instead of beta testers per se. Anyway... just an observation.

[Crzyrndm]

And perhaps Squad starting viewing us as real consumers, instead of beta testers per se. Anyway... just an observation.

More likely Squad learned that general joe public consumers just can't double as testers.

Pubic bug reports are about 70% useless, 25% duplicates of the same 5-10 issues (mostly without enough information), 3% less obvious/new bugs but without enough information, and maybe 2% less obvious/new bugs that are reported in enough detail to be useful (source: I used to do QA work for a different company and had to filter public reports on occasion. Each internal team member would often submit more bug reports per day (of significantly higher quality) than all of the useful public reports over the same period.)

Public reports are useful for catching the edge cases. For getting it to the point where edge cases are actually a thing, a structured team will do many, many times better.

Edited May 6, 2015 by Crzyrndm