A proposed solution to identify KSP bugs before they come out

[Guest]

This update and many before it seem to be full with bugs, and I know bugs happen a lot (from my experience making games.) But it seems that there are ways to reduce bugs before release that seems to work. KSP would probably benefit from having a kind of system where the public can test some of the features before release. I like to reference to Minecraft's tactic to getting identified of bugs before release; snapshots.

The Pros:

-would help to identify bugs ahead of time instead of at launch

-get's more people on board on giving feedback before launch

-gives us a way of understanding the content in the update ahead of time

The Cons:

-get's rid of the secrecy of some parts of the update (I don't see this as a problem but squad might)

-would complicate things on KSP launcher and steam

-might not work on the scale KSP is played at

So what do you guys think? And remember, this is my opinion, and my opinion isn't really that popular right now...

[The Aziz]

Afaik there are some people in the community who are beta testers. So more like closed beta rather than open beta.

I might be completely wrong though.

[Gargamel]

There are a number of people who Beta test already for SQUAD.  There are also a number of paid QA people on SQUAD staff.  But even with the best testing methods, some things are going to squeak through.   Things that they just didn't think of.

Imagine you're writing software for an automated coffee shop. 

You have thought of practically everything.

Your QA testers come in and try to break the system.   They order Small Venti Waters.   They order 3 gazillion black coffees with milk.   Everything you can throw at it you do.   Passes with flying colors.

As soon as you go live, the first customer walks into the store and asks to use the bathroom.    Entire building burns to the ground.  

What might be obvious in hindsight was not very obvious at design time.  

And that's why KSP uses the testing group known as the community.   I have never seen a more deranged and limit breaking group of gamers.   If we don't require a bug patch with every update, then I become disappointed in the community.   It just means we're not pushing the game hard enough. 

And yeah, maybe somethings should have been caught earlier, but the devs just didn't see them coming until it was too late.  

So yeah, we're 11 versions into the full release of KSP1, (and I'm almost sure at this point the difference between 1.0 and 1.11 is more than .23 and 1.0, so hats off to the devs for keeping the game alive) and each and every time we've need a couple patches to work out the usually minor bugs that have crept up.   I just wait until the 1.x.1 update drops before I bother updating.   That way the bugs that happen at initial release aren't a problem for me.   I still have my 1.3.1 save I like to break out from time to time.    Getting the newest and shiniest version isn't on my priority list.

So, my point is, if the existence of the bugs upon update release bothers you, then wait a bit before updating.  The Squad guys aren't going anywhere, they're going to fix the update, and usually rather quickly.     If you like the cutting edge, and don't mind encountering a few quirks along the way, then by all means up date as soon as you can.  

 

And this Development suggestion has been moved to the Suggestions Forum. 

[Superfluous J]

What is the effective difference between the community getting a x.0 release with bugs, and the community getting a beta version with bugs?

[Guest]

4 hours ago, Gargamel said:

And that's why KSP uses the testing group known as the community.   I have never seen a more deranged and limit breaking group of gamers.   If we don't require a bug patch with every update, then I become disappointed in the community.   It just means we're not pushing the game hard enough.  

true... that's a good thing too that I agree with.

3 hours ago, Superfluous J said:

What is the effective difference between the community getting a x.0 release with bugs, and the community getting a beta version with bugs?

probably right... 

[jimmymcgoochie]

I once took part in a beta-like group which tested new features prior to their release for an MMORTS game- the group were supposed to keep it quiet, but it leaked like a sieve and the supposedly “trusted testers” didn’t last long before the developers justifiably pulled the plug. While I’d be happy to join a KSP beta (software testing is my full time job), I don’t think it will happen because people would keep blabbing about new stuff.

[Snark]

21 hours ago, The Doodling Astronaut said:

KSP would probably benefit from having a kind of system where the public can test some of the features before release.

...

So what do you guys think?

It's not a dumb idea.  And, surprising though this might be, you're not the first to think of this. 

In fact, they used to do this.  Back in the day, there was a while where Squad actually did have what amounted to an open beta, and pretty much anyone who wanted to could join in.

However... it turned out not to work as well as you might think, and eventually they scrapped it and moved to closed, private betas.  This is one of those ideas that sounds great "on paper" but turns out to be more problematic than you might think, in practice.  Lengthy explanation below, but the TL;DR is that such a program generates huge amounts of low-quality, non-actionable noise for the developers to wade through, to the point that the costs may outweigh the benefits.

The details:

The problem is that when you throw the doors wide open like that, the quantity of bug reports goes way up... but the quality goes through the floor.  The average player, no matter how well-intentioned, is simply not a very good bug reporter.  QA is a skill, and part of that skill lies not just in what you test and how much testing you do, but also how you report and document it.

Things that a good bug report should entail, which the average player isn't necessarily aware of, and/or diligent enough to be willing to do, and/or know how to do effectively:

• Makes a reasonable effort to check in the bug database whether it's already reported (duplicate bug reports don't help and waste people's time)
• Does the footwork beforehand to document what are the reproduction steps
• Is conscientious about the installation, e.g. you don't have random mods and what-not cluttering up your execution environment
• Is meticulous about documenting the environment, e.g. which build you're on, what OS, etc.
• ...And so forth.

What happens with an open beta is that you get tons of people who contribute quickie "Hey it broke when I did <thing>" bug reports, often without enough information to be useful, often duplicating existing bug reports.  The signal-to-noise ratio is really low.  Yes, you'll get a few users out there who contribute good, actionable bug reports... but their useful signal gets drowned out amidst thousands of fluff reports.

Every such bug report is a time sink for the people making the game.  There are a finite number of professional QA people and a finite number of developers, and each such bug report is going to have an impact.  Some QA person is going to have to spend time reading it, and scratching their head, and trying to figure out what the person meant, and trying to confirm it themselves, before passing it along to a developer to fix.  That's time that they can't spend working on high-quality, actionable bug reports themselves.

Therefore, what can work better is a limited-scope, closed beta, where there will be a lot fewer people testing than if you threw the door open to thousands... but the quality will be much higher, and the reports they produce will be much more likely to be actionable and therefore not waste people's time.

 

Aside from all that, taking your above points in order:

21 hours ago, The Doodling Astronaut said:

-would help to identify bugs ahead of time instead of at launch

Except that "identifying" bugs doesn't help unless they're actionable and have repro steps that allow a developer to confirm what's going on.

21 hours ago, The Doodling Astronaut said:

-get's more people on board on giving feedback before launch

In practice, this turns out to be a con, not a pro, because then you have to wade through all the stuff they churn out, much of which isn't very helpful.  It turns out that it's more valuable to get the right people on board, not more people.

21 hours ago, The Doodling Astronaut said:

-gives us a way of understanding the content in the update ahead of time

How is this a pro?  You'll "understand" it when it comes out, yes?  And they can always offer sneak peeks ahead of time, as they've done in the various "KSP Loading..." posts.

It can also generate confusion, because in the process of testing, sometimes they need to change how things work based on feedback and so forth.  If you open it all up to the public before everything has gelled pretty hard, it generates a lot of noise for the players.

21 hours ago, The Doodling Astronaut said:

-get's rid of the secrecy of some parts of the update (I don't see this as a problem but squad might)

This is very much an issue for a lot of software developers.  Also, aside from "secrecy" per se, there's also the "generate confusion and noise for the players if details get out that get changed later" that I mention above.  With a closed beta, you can have the testers sign NDAs so you're protected from this.

21 hours ago, The Doodling Astronaut said:

-would complicate things on KSP launcher and steam

This is actually not really a con.  Steam has built-in functionality to let game developers do exactly this, it has "betas" built in as a feature in the platform and includes the ability for the publisher to make it open or require a key to get in.  So this turns out not really to be an issue.

21 hours ago, The Doodling Astronaut said:

-might not work on the scale KSP is played at

This is the big one-- the "drowning in too much well-meaning but non-actionable input" problem that I describe above.

[Kerbart]

1 hour ago, Snark said:

This is actually not really a con.  Steam has built-in functionality to let game developers do exactly this, it has "betas" built in as a feature in the platform and includes the ability for the publisher to make it open or require a key to get in.  So this turns out not really to be an issue.

Actually, it was an issue but not for the right reasons. When Squad was doing the pre-releases they used Steam only (and not GOG or even the KSP store), I suspect for logistical reasons.  That raised a ruckus because many non-Steam players (in some cases anti-Steam who felt they were doing Squad a favor by not using Steam) felt that they were unjustly barred from the privilege of a "preview".

That, in itself, highlights one of the problems; many of the "testers" don't really intent to test but rather see the early release as a sneak preview.

Throw in the questionable quality of the bug report ("my rocket explodes when I launch it". maybe accompanied by a screenshot of the gamedata folder containing 118 mods and another gamedata folder) and I agree that those bug reports create more work than the time they would save.

I did always enjoy the sneak peeks people like Das Valdez, EJSA and Scott Manley would give us, and they consistently would find bugs (and show how to create a proper bug report by making it perfectly reproducible)