Question on testing

[Leopard]

Question related to have the game is tested, brought about by the debacle on the consoles with the new edition, but being asked more generally.

How is this software tested?

It feels like this is largely being handled automatically - i.e. does this code function do what its meant to do, unit testing etc, even daisy chasing functions and testing the code that way.

Because what it doesn't feel is happening is someone actually sitting down and trying to play the thing, certainly not testing like this with inexperienced users, users who won't "work around" a problem, but will be floored by it.

e.g. on screen buttons that don't respond, assigning "more info" and "close window" to the same control - both functions may work perfectly well as code, but you cannot actually use them from the interface.

 

Speaking as someone who develops software I'm aware GUI design is hard, and even harder to do well, and then trying to do it with the very limited input functionality on a games console even more so - but I'm also aware you test things and make sure they work, not release and let your users find out that they don't

[FrostedShoe]

Where I work some software was released to the end user without proper testing (and no instructions but that's a different matter) because it's cheaper to fix problems as they are discovered instead of spending lots of money testing. It might just be a cultural shift, KSP is not the only game to be released with bugs.

[Leopard]

3 hours ago, FrostedShoe said:

Where I work some software was released to the end user without proper testing (and no instructions but that's a different matter) because it's cheaper to fix problems as they are discovered instead of spending lots of money testing. It might just be a cultural shift, KSP is not the only game to be released with bugs.

A sad state of play really

seriously leaves a bad taste in the mouth, this should be a wonderful game, seriously addictive, but when it has control bugs you find in the first few minutes you wonder if anyone bothered to actually try playing it?

[Rocket In My Pocket]

The console port was handled by a different company; like most ports these days.

So...it's not exactly Squads fault. However I'll agree that given the first port was so bad it needed a re-release; It's a bit alarming that the "new and improved" release is already having issues. Despite being a pretty hardcore KSP fan; I don't think I could ever recommend the console version to anyone after all this, it's been one travesty after another and obviously, someone, somewhere doesn't really give a flying toss about the quality of the console version. Which is sad; because it hurts KSP's reputation as a whole.

I certainly don't envy the folks who had to squeeze all those controls onto a handful of buttons though.

Edited January 20, 2018 by Rocket In My Pocket

[Mako]

2 hours ago, Rocket In My Pocket said:

The console port was handled by a different company; like most ports these days.

So...it's not exactly Squads fault.

I would like to say to this that in many of the Weeklys Squad did say they were playing new builds from the port team as BlitWorks went along. So, apart from a third-party testing team and BlitWorks own testing (whatever that may or may not be), Squad admitted having eyes on new builds as work progressed. I'm sure that didn't always, or possibly even ever, involve rigorous testing by Squad, but Squad must have at least been aware of the controls.

Now here's the thing regarding that: Squad had the port devs who could walk them through the controls as they were finalized, so Squad had a distinct advantage over the end user when it came to understanding how to interact with the game. From a lot of the initial feedback I've seen, it seems like there was a major lack of information provided in-game for end users to best familiarize themselves with the controls. It seems like there really should have been a skippable, repeatable control tutorial that pops up when you first run the game and remains available from the main menu if you need to refresh your memory.

Anyway, I'm hopeful that many of the issues get resolved. BlitWorks has a significantly better reputation than FTE, and for all of the ugliness that crowd-testing creates, it certainly is effective at finding lots of bugs. I hope they eventually get this right. I thought I might finally be able to recommend KSP to a friend who only has consoles, but it seems like it still isn't the right time.

[Geonovast]

3 minutes ago, Mako said:

I would like to say to this that in many of the Weeklys Squad did say they were playing new builds from the port team as BlitWorks went along.

As someone who's been a part of a beta test for a piece of hardware, It's possible that a lot of feedback goes straight into the trash.  We reported a lot of device-breaking problems with that thing that were still present when the product was released.

Also, was there time to fix the problems?  They had two issues they were dealing with - the quality of the port, and the barrage of users who'd been screwed by the first port who wanted the new one NOW and PERFECTLY.

[Mako]

@Geonovast

While of course anything is possible and we have no way of knowing what happened, I was simply relaying the information that Squad themselves claimed in their Weeklys. They said for many weeks that they were playing new builds and they were coming together nicely. If BlitWorks were discarding Squad's feedback I would think it would become apparent to Squad rather quickly.

As for time, well, they had to balance getting it out as quickly as possible with also getting it out right this time. You obviously can't have both quick and right, but you can chose which is more important. I'm not claiming I know what's best here, and either way it's much easier to evaluate something after it happens than it is to anticipate exactly what will happen.

[Geonovast]

@Mako

Agreed, no way of knowing.  I was just throwing it out there as a possibility that should be considered.

[Leopard]

5 hours ago, Geonovast said:

As someone who's been a part of a beta test for a piece of hardware, It's possible that a lot of feedback goes straight into the trash.  We reported a lot of device-breaking problems with that thing that were still present when the product was released.

Also, was there time to fix the problems?  They had two issues they were dealing with - the quality of the port, and the barrage of users who'd been screwed by the first port who wanted the new one NOW and PERFECTLY.

Its interesting there appears to be a range of "known issues" at launch, this is actually to be expected and acceptable - if they are not mission critical, or in this case game breaking.

There also appear to be a lot of issues that should have been picked up during testing, basic interface problems for example that anyone actually trying to play the game will find and find quickly, these are much less acceptable.

e.g. you buy a new TV, say one time in five the "on" button doesn't work so you have to unplug it and plug it back in, is that "acceptable"?

 

if there were a few bits of the GUI that don't work, and are known not to work, the simple solution is to disable and hide them until they are fixed - e.g. the option to purchase parts in R&D is broken - so turn it off and make the control to turn it back on invisible - now you have a broken feature the end user will not even be aware of as broken so it can be fixed and re-integrated later.

 

Didn't want the new port to be perfect, well Want and Reasonably Expect are not the same, what I think was not unreasonable is for the new port, after 6 months of testing, to be no worse than the previous one in terms of functionality - i.e. if FTE could make something work and work reliably, then the new port should also function in that regard - then an existing play will only see improvements.

 

Sticking out the release with a note that there are a range of known issues, with a list of them, or the ones most likely to be discovered allows a "don't worry, we are still working on this but wanted to get something to you now" approach which would likely have gone down a lot better.

 

we have seen many months of reports from the "Extensive" testing, noting all the little GUI glitches and bugs that have been found and fixed, then upon release we discover some pretty basic features don't work

[invision]

this game shouldnt have been ported. it cripples decent pc's i can only imagine how bad it runs on console and watching all the dev notes and posts for month its clear they bit off more than they expected. honestly they should just fully refund all sales if they cannot give users a game that isnt broken.

hasnt it been a year now since its release? and its still broken? 

 

[ExtremeSquared]

The PC version was pretty broken for at least a year after its "release" also. Not excusing anything, but I feel like it was worth the wait at least.