QA Request

[Kerbart]

I'll admit that I'm clueless on QA in general, and particular about how it's done at Intercept Games.

However, as my day job requires me to create financial reports covering large amounts of money (let's just say that I need to adjust the column width in Excel to fit the numbers in the output) I do have some experience in quality checking, and from my laymen, Monday Morning Coach, Captain on the Quay viewpoint it seems some checks are being skipped.

See, I know what to look for.. Discrepancies by week, month, product, and a couple of other dimensions. So my quality checks look at those. But there are always the things you don't look for. So, I pull out the detail reports for our top-5 customers, and for another random 5. And that's the step that's missing with KSP2 QA.

Run a handful of extended play-through scenarios:

• Build a ship from scratch in the VAB, launch it, put it in orbit
• Build another ship from scratch, launch it, dock with the first one
• Undock both ships and return to Kerbin
• Build a ship, launch, and land it on  Mun, plant a flag,  and return to Kerbing
• Build a ship, launch it, and land it on Minmus, plant a flag,  and return to Kerbin
• Build a ship, launch it, land it on Duna, plant a flag, and return to Kerbin

Between those 5 builds you can cover a wide variety of parts. And granted while not covering all situations and scenarios, it will cover  a lot of ground.

Current testing seems to focus on prescribed scenarios bound to known errors, and for time saving purposes those are either pre-saved or set up using a "cheat" dialog. And that's ok. You want to know if bug #684 has been addressed in all 12 known scenarios, and there's hundreds of cases you have to go through.

But clearly, clearly, the vital act of "spot-checking" — in this case just playing the game — is not performed right now. Or if it is, some people need to get fired. There are just too many bugs in the game to believe a serious effort is made in that direction. If there is play testing, then why do we now have a hotfix for the fairing UI bug in the VAB. HOW WAS THAT MISSED? Fairings respawning on engine plates every. single. time. How could someone not see this and report this?

Please. Add a handful of actually playing the game to the test book.

Edited June 30, 2023 by Kerbart
Added "plant a flag" to the landing missions

[RalphKerman]

I support this point of view.

For me it was really baffling that the camera bug that is currently in the game was not spotted (camera is facing in a fixed direction after returning from map screen and not where it faced when entering map screen).

That is such a normal thing to do while playing the game (do something, check map, go back and do more stuff) that I can't really understand why that was not found during testing. It supports the theory of testing isloated cases with special setups (debug menu to get there) to test a specific issue and not look besides that point.

I am working in tech support and often clash with my devs as they develop and test with "blinders over their eyes" (developing something or testing it without looking 5 millimeters left or right, just straight ahead and therefore miss something very important to the customer that could have easily been detected if they looked a little bit around).

The 6 steps that Kerbart has mentioned in his post are really good and cover a lot of what is needed to enjoy KSP and covers (I would bet) 95% of all missions players want to do right now. If they work reasonably well the acceptance of missing features and delays will increase as players can enjoy a good amount of core gameplay.  But from reading the bug report forum my understanding is that none of the above 6 scenarios can be completed right now without encountering at least one serious issue or RUD. Even veteran content creators who do the most crazy things in KSP 1 are struggling with the basics in KSP 2 right now, so please think about the "normal" player base who may even be new to the game and wants to learn about rockets and space flight. Give them and all of us a "basic stable game" (the 6 steps above with no kraken attacks) that we can enjoy and players will come back and/or convince new players to join.

[dr.phees]

Add some driving around on the surface and (at some point) on the water, using 3-, 4- and 6-wheelers.

[Superfluous J]

On 6/30/2023 at 9:50 AM, Kerbart said:

• Build a ship from scratch in the VAB, launch it, put it in orbit
• Build another ship from scratch, launch it, dock with the first one
• Undock both ships and return to Kerbin
• Build a ship, launch, and land it on  Mun, plant a flag,  and return to Kerbing
• Build a ship, launch it, and land it on Minmus, plant a flag,  and return to Kerbin
• Build a ship, launch it, land it on Duna, plant a flag, and return to Kerbin

If they do this, they won't be doing their job.

It should be more like

• Build a ship from scratch with the mk1 pod, launch it, put it into orbit.
• Build a ship from scratch with the mk2 pod, launch it, put it into orbit.
• Build a ship from scratch with the lander can, launch it, put it into orbit.
• ...
• Build a ship with the octagonal strut, launch it, put it into orbit.
• Build a ship with the cubic octagonal strut, launch it, put it into orbit.
• ...
• Build a ship with the you-get-the-idea, launch it, put it into orbit.

I think this old joke is appropriate here.

Quote

A QA checker goes into a bar. Orders a beer. Orders 2 beers. Orders pi beers. Orders 1^-1 beers. Orders "; delete table employees; beers. Approves the bar for customers.

The first customer comes in, asks where the bathroom is. The bar explodes.

 

[Kerbart]

9 hours ago, Superfluous J said:

If they do this, they won't be doing their job.

It should be more like

(...)

This is a case where perfect is the enemy of good. Testing all parts that way won't be done because it'll take way too much time. The point is to run a few "full game" scenarios and uncover bugs anyone playing the game encounters but testers (because they test a scenario and don't play the game won't.

Testing every single part full sequence is definitely desirable but will add like three weeks to a release and be deemed impractical. A more incomplete test (albeit better described perhaps but I'm not versed in game QA) might lack many things but the one thing it has going for it is huge: it's hard to find an argument not to do it, as it can be done by one tester in half a day or less.

[WelshSteW]

 

Yeah, this is along the lines of the post I made in the hotfix topic.