Preparing for the pre-release; guidance please on bug-hunting.

[Papa BullDust]

As we know, KSP is soon™ to enter the pre-release stages for 1.1 which will hopefully mark the beginning of the end for the ‘search and destroy mission’ on bugs before the release.

 

As an avid fan and noob programmer (someone who has a good grounding in programming ideas and processes etc, but no mods under my belt yet) I’d really like to help out and do some testing, but since I have no real-world experience at QAing big software, I’d like to gather suggestions on how best to go about looking for bugs.

 

My first port of call was to read the page on the bug-tracker (http://bugs.kerbalspaceprogram.com/projects/ksp/wiki) which I found to be a great reference dealing with the practical aspects of reporting bugs such as the documentation required and keeping logs, screenshots, craft files etc. However, I’d like some community guidance on where best to look when in-game. The practical side of me says “well, just attach parts together and see if they explode in a variety of craft conditions” but having thought further, I’m guessing this kind of more basic, systematic testing (and considerably more!) is likely to have been covered extensively in the Experimentals by a team of specialists. I’ve tried to layout my thought processes in the form of questions:

 

“Where should we be looking?” - I appreciate the irony of asking where to look for hidden and often elusive problems, but is the testing meant to be on a SOI basis (test as much as is practically feasible in one physical SOI) or on a biome specific basis (the same but for an individual biome) or should we actually be doing the ‘test every part with every other part’ type testing? A reasonable assumption may be 'all of the above', but any thoughts would be helpful.

 

“Where would you start?” - Is it simply a matter of going about choosing a career type to focus on and going about daily Kerbal business until something wacky happens, or should it be more directed than that?

 

“Will there be more information made available?” - As a last question mainly to Squad staff, is there going to be further guidance posted on this nearer the time, or is it intended that anyone who is actively participating with the bug-hunt would have prior knowledge of testing?

 

I’m sure you guys will come up with answers, and probably better questions too, but this will hopefully be a jumping off point. I would like to add though, I don’t mean this to post to add to the steam/non-steam debate, those with AND without access to the pre-release have a wealth of knowledge which would prove invaluable in guiding the bug-hunt.

 

Much love y'all,

D.

[GeneralVeers]

As somebody who used to be a software developer and knows the ins and outs of testing software, here's a few essentials:

#1: Begin at the beginning. Pull up the list of known previous bugs and start out by testing for the bugs Squad knew about and intended to squash. Because testing for bugs you don't know about is very dicey.

#2: Unit testing. Instead of testing "every part with every other part" (a HUGE workload!) start by testing each individual part on its own. Make sure each one works.

#3: Variance testing. Build some rockets and spaceplanes identically to previous models (which you flew before, so that you know they work) and see if they work any differently in the new version. The goal is to make sure the same stuff that worked, still works. If, for example, a spaceplane is suddenly acting differently when Squad specifically said they didn't change the code for the aerodynamics, something is probably wrong.

#4: Clustered testing (a name I made up just now). Instead of testing "every part with every other part" (again) build test ships that use, say, 20-30 different parts. Overengineer, add useless parts, doesn't matter. Rather than build 227x226 different ships to test every part with every other part, test a bunch of parts at once. If the entire test ship works, you know 20-30 parts all work with each other, bug-free, with one test. If something explodes, you know the problem is somewhere in that group of 20-30 parts.

#5: Watch out for intermittent bugs. While doing all your other testing, try to record everything weird that happens, even if it seems minor. If a glitch happens once and then mysteriously disappears on a retest, don't discount it right away--a bug that only happens some of the time is extremely difficult to catch. Be extra-vigilant for those.

[HebaruSan]

Keep in mind they've already done a traditional QA pass and Experimentals, which should cover focused, "test plan" style testing. If you do that, you'll be re-treading the same paths that are most likely to be cleared of bugs already. The pre-release has two main advantages as I see it:

1. With such a large audience, many more things will be tried than a smaller team could do, especially things that are less obvious
2. We can post findings on the forums and quickly develop a consensus about what the worst problems are

With that in mind, my advice would be:

• Pay very close attention, including using tools like the aero arrows and heating overlays
• Try as many different kinds of missions, different kinds of crafts, styles of staging, editor actions as you can, right click everywhere, etc.
• If you have any of your own mods (I know you said you don't, but I'm thinking of testers in general now), update them and see if that exposes any new problems
• Take good notes and post them on the pre-release forum once it's announced
• Read what other people are reporting on the pre-release forum to get more ideas of things to try, and help pinpoint test cases of intermittent problems

1 hour ago, drewthedrewman said:

“Will there be more information made available?” - As a last question mainly to Squad staff, is there going to be further guidance posted on this nearer the time

The latest dev note says they're working on documentation for the pre-release, so hopefully the answer to this is yes. They at least need to tell us how to access it (even though many of us are pretty sure we can guess), and they'll probably answer questions in that thread.

[Papa BullDust]

Cheers very much for the replies guys, this is exactly the kind of thing I was looking for. Since I usually play with mods, I'mma get to the 1.0.5 drawing board tomorrow and create some crafts in my stock version and have a cheeky testing session. That should give me somewhere to start so I know how these vessels behave.

I also take the point about staying in communication with the forums during the process, I was/am interested in how this is to be co-ordinated, but that gives me a much clearer picture. Much appreciated

[GeneralVeers]

4 hours ago, HebaruSan said:

Keep in mind they've already done a traditional QA pass and Experimentals, which should cover focused, "test plan" style testing. If you do that, you'll be re-treading the same paths that are most likely to be cleared of bugs already.

Oh, hey--that reminds me of the.....well, I don't recall if there's an official name for this, so just call it the two-heads-are-better-than-one rule. Short version: the people doing the testing should usually NOT be the same people who wrote the code. In my personal experience, one tends to have biases about the code they wrote and it can be very easy to miss things.

So even if "they" already did the QA, duplicate at least some of it anyway. If you catch something Squad missed, it'll make you feel like a million of whatever the Kerbals call the money you earn in Career mode.

[Papa BullDust]

9 hours ago, GeneralVeers said:

...the people doing the testing should usually NOT be the same people who wrote the code.

I can see this; one can have preconceptions about how the code is supposed to work. I used to work with large amounts of healthcare data and we always had a policy of having a colleague look over what we had done before it was sent out. Not quite the same, but a similar principal.

[Basto]

I used to do software testing and I always liked to approach things from the do something weird or unexpected approach, try and break the game. Try using parts in ways they weren't "intended". Build weird ships and see what happens (using the same ship on 1.05 as a control of course) We know that wheels had some of the biggest changes so personally I would focus allot of time on rovers. 

Like others have have said they have already done a ton of QA so testing to obvious stuff at this point may be less useful. 

[KerbonautInTraining]

If you have a compatible nVidia graphics card, use shadowplay!!

Set it to shadowplay & manual mode, maybe 2 minutes shadow time. If something weird starts happening, pause the game, save the shadow recording, and turn on manual recording. That way, you get the entire incident recorded. 

(If you can't get it to work with KSP, go to the preferences and enable desktop recording)

[Kerbart]

15 hours ago, HebaruSan said:

Keep in mind they've already done a traditional QA pass and Experimentals, which should cover focused, "test plan" style testing. If you do that, you'll be re-treading the same paths that are most likely to be cleared of bugs already. The pre-release has two main advantages as I see it (...)

This. Remember the 1.0 release with its flurry of patches for all kinds of bugs that made us think “did they actually play the game before releasing it?”

The systematic stuff has already been done in QA. This is about playtesting. Communicate about bad interface design, weird things that shouldn't happen (remember Scott Manley's airflow bug). That doesn't mean don't do stuff to see if it breaks—by all means do weird stuff — but do it literally in a playful manner. That's where the benefit of this stage comes from, to discover things you’d be doing while playing the game. The QA testing has already been done.

[Nich]

As an aerospace engineer I have seen 3 basic types of errors. 

System errors. A and b talk a talks in float 32 b listens in float 64

divide by zero errors 

overrun errors 361 degrees float overrun, int overrun 

you don't really have the tools the management or code access to really do proper testsing I would just play the game and keep logs. Knowledge of other bugs will help keep duplicate reporting to a minimum.