Crashes upon reloading VAB/SPH/Launch excessively

[Kowgan]

KSP: 1.4.5 x64

Problem: KSP crashes when either reloading the VAB/SPH or Launching a vessel repeatedly in a short amount of time. Or just generally reloading those scenes too many times, nor necessarily  in a short time.

Mods installed:

Spoiler

Toolbar 1.7.17.11
USITools 0.12
ABCORS 0.5.0.1
ADEPT 1.0*
Procedural Wings 0.50
Better Burn Time 1.6.1
Bluedog's LEM*
Brake Toggle (recompiled it myself for KSP 1.4.5)
Community Category Kit 3.1
Community Resource Pack 0.10
Community Trait Icons 1.0
Contract Configurator 1.25
Contract Packs:
-- Field Research 1.2.1
-- GAP 1.4
-- Kerbal Academy 1.3.0.1
-- Tourism 1.5.2
Crowd Sourced Science 4.1.2
DMagic's Orbital Science 1.3.0.12
Firespitter Plugin 7.9
Interstellar Fuel Switch 3.6.4.1
KAS 2.6
KIS 1.14
MechJeb 2.7.3
Mouse Aim Flight 1.1.1 (recompiled it myself for KSP 1.4.5)
NavBall Docking Alignmen tIndicator CE 1.0.4.2
Patch Manager 0.0.16
Kerbal Planetary Base Inc 1.6.4
Procedural Fairings 1.4.3.4
RealChute 1.4.6
SCANsat 1.1.8.7
Science Full Reward 3.0
Station Science 2.4.2
Surface Experiment Pack 2.6
Survey Transponder 0.5.2a*
USI Akita rover 0.5*
USI Life Support 0.9
USI Malemute 0.6*
USI Sounding Rockets 0.9*
Variable Thrust Limiter 0.1.5 (recompiled it myself for KSP 1.4.5)
Waypoint Manager 2.7.2

*Simple parts mod, not used or loaded into the editor or crafts used during crashes.

Reproduction steps:

Load a save, re-enter VAB/SPH/Launch vessel and Revert to building multiple times.

Logs:

output_log.txt

error.log

Notes:

I was unable to reproduce the issue on Stock KSP, so I might try to narrow it down to a specific mod. But it might be difficult due to the inconsistency of the issue. Sometimes I am able to crash the game within a couple of reloads - Other times, it takes me several attempts unitl it crashes.

After a quick research, I've found that usually 0xc0000005 errors are related to the lack of available RAM.
In my case, I have 16GB of RAM,  running the 64bit version of KSP and having around 50% memory free at the crash moment. KSP was using around 4GB of RAM at that moment.

Any help is much appreciated.

Edited August 22, 2018 by Kowgan

[JoE Smash]

Did you get a different error when you did the xinput1_3.dll swap?

Like did that disappear from the error log at least?

I've never seen "Bonjour" on the list before...

What's that do? Can you disable/close it when running the game?

https://apple.stackexchange.com/questions/45765/do-i-really-need-bonjour-on-windows

I'd do the xinput1_3.dll swap AND ditch bonjour and see if the game is stable....

[Kowgan]

No, \system32\xinput1_3.dll was still there when I swapped these DLL files.

Bonjour is a service related to Apple iTunes. I have no clue why it shows up on this error log, though. I should be able to disable it, although I'm not sure how it would affect the game. Then again, nothing in this has made much sense to me so far.

Edited August 22, 2018 by Kowgan

[JoE Smash]

I'm saying I've looked at quite a few error logs at this point and I've not yet seen one with bonjour on it....

Xinput1_3.dll is on pretty much EVERY error log I have seen....

I'm just going by the fact that it is unusual to see it listed. It may be a hint....lol.

[Kowgan]

I've uninstalled Bonjour (it's useless to me, and I should have done it a long time ago anyway) and will try to reproduce the crash again. I'll post an updated log file if here if it crashes again. Thanks for the support, it's much appreciated

[JoE Smash]

I hope it was the culprit....so you can enjoy playing without annoying crashing....

[Kowgan]

Alright, I managed to crash it again. Same methods, just constantly loading SPH and launching on the runway.

output_log.txt
error.log

I'm considering switching to OpenGL to see if things get better.

[4x4cheesecake]

Have you tried to remove the mods you recompiled on your own? Also, Surface Experiment Pack creates a few exceptions...I don't think they are related but it should not be a big deal to remove it as well (at least for testing)

[Kowgan]

Thanks, @4x4cheesecake. I haven't tried this yet, but according to Lisias:

On 8/22/2018 at 10:09 PM, Lisias said:

I run KSP on a Mac machine, and it's usually eats up 10 to 12G of RAM.

If your KSPx64 is consistently dying at 4GB of memory allocated by the process, I definitively would bet on some 32 bits DLL somewhere in the System being used by KSP, and getting killed when KSP reaches that memory limit and the get shoot on the feet when it calls that DLL using a memory pointer above that.

If the XInput trick didn't helped you, there's a good change that something else on your system is getting into this.

Another thing is VRAM memory. Each texture you add to KSP via mods eats VRAM from the GPU Card. When you exhaust your VRAM, Unity also crashes without any clue about the reason.

I don't know if the GPU reaching its max memory should be the case since it's not VRAM, but maybe some 32bit DLL being called while the game is consuming over 4GB of RAM could be the case. Not sure.
The inconsistency of the crashes is also something to keep in mind. Lately, I've had some 3 or 4 hours sessions with no crashes. And I didn't keep reentering the SPH/Runway every 30 seconds or so.

For now, I guess I can just live with it. If I can play 3 hours with no crashes, I'll just boot it up again whenever it happens.

[Lisias]

57 minutes ago, Kowgan said:

And I didn't keep reentering the SPH/Runway every 30 seconds or so.

I'm a dumb SAS. Until now, I completely missed about this important piece of information besides being advertised ad nauseaum.  

It's still a guess, but now a more informed one.

Joysticks are not needed except on Flight Scenes. So, it's reasonable to assume that the XInput are "opened" when you enter the Flight Scene, and "closed" when you exit it/enter VAB, SPH, et all.

It's also well known that our "friend" Unity just don't releases memory no matter what. And since it's well known its memory leakings, it's plain clear that memory footprint is aways on an ascent - what varies is the slop.

So:

1. User builds a plane/rocket.
2. User enters Flight Scene
   1. XInput is "opened", and a memory buffer is allocated for it.
3. User flies this toy
   1. Unity globs some more memory, raising the footprint a bit
4. User goes to a non-flight scene (Mission Control, Tracking Station, whatever)
   1. XInput is "closed", and the current used buffer is released
   2. This memory is not under Unity's control
5. User goes back  to flying
   1. XInput is "opened again".
   2. Since the previously used buffer was released and probably already reused for something else, a new buffer is allocated

As the memory footprint grows, eventually this new buffer is allocated beyond the 4Gb boundary and then things goes KABOOM on 32 bits DLL.

@Just Jim, I think we nailed down a deterministic Modus Operandi in order to reproduce the issue. A small custom part that "eats memory while flying" would speed up the process. Perhaps MemGraph could do the trick too.

Edited August 24, 2018 by Lisias
typos, what else? =P

[Kowgan]

Interesting. That would explain a lot. Also, I wonder if my self-recompiled version of Mouse Aim Flight could have something to do with it. FWIW, I know it already has a glitch where it doesn't correctly remove the stockbar icon, so you can still see that mod's icon when you unload back to the main menu. I wouldn't be surprised if the root of this problem is directly related to the ever-increasing memory comsumption as well.

I'm tempted to replace the system32\xinput dll file with the one provided by KSP and see what explodes.

Edited August 24, 2018 by Kowgan

[Lisias]

9 minutes ago, Kowgan said:

Interesting. That would explain a lot. Also, I wonder if my self-recompiled version of Mouse Aim Flight could have something to do with it. 

I think it would be another "victim", I gave a peek on the source and I only saw calls to UnityEngine, not "direct calls" to any native DLL.

 

12 minutes ago, Kowgan said:

I'm tempted to replace the system32\xinput dll file with the one provided by KSP and see what explodes.

Most games doesn't eats memory as candy as KSP. I don't expect too much fireworks.  

[JoE Smash]

2 hours ago, Kowgan said:

I'm tempted to replace the system32\xinput dll file with the one provided by KSP and see what explodes.

I actually did that first....that ALSO works...

My concern was any OTHER game that wanted a 64bit version of xinput1_3.dll would then get the wrong thing and crash....because the KSP one is 32 bit....

That worked equally well as putting the windows/system32 on in the games 64bit plugin folder.....

[steve_v]

17 hours ago, Lisias said:

It's also well known that our "friend" Unity just don't releases memory no matter what. And since it's well known its memory leakings, it's plain clear that memory footprint is aways on an ascent - what varies is the slop.

KSP leaks like a sieve, always has. Here's a quick & dirty graph from my modded 1.4.5, those spikes are scene changes:

I can produce something similar with a stock install if anyone wants more proof of this, it looks almost exactly the same, though the increase in memory consumption is slightly slower.
This machine has 32GB of RAM, and I'm getting close to 40% here. Properly ridiculous, innit?
I have no idea how a game can possibly justify using so much memory, except through lousy design decisions, and lousier engineering.
I can think of no justification at all for the ever-increasing memory footprint, especially in a managed runtime... Unless of course that runtime is also excrementse.

 

Edited August 25, 2018 by steve_v

[JoE Smash]

7 hours ago, steve_v said:

KSP leaks like a sieve, always has. Here's a quick & dirty graph from my modded 1.4.5, those spikes are scene changes:

I can produce something similar with a stock install if anyone wants more proof of this, it looks almost exactly the same, though the increase in memory consumption is slightly slower.
This machine has 32GB of RAM, and I'm getting close to 40% here. Properly ridiculous, innit?
I have no idea how a game can possibly justify using so much memory, except through lousy design decisions, and lousier engineering.
I can think of no justification at all for the ever-increasing memory footprint, especially in a managed runtime... Unless of course that runtime is also excrementse.

 

I have noticed the same issue.

It would be great if @SQUAD would fix this at some point...

How many patches and updates have there been since 2014?

Still the game eats more and more RAM the longer you play and the more launches you do....until eventually it crashes or you are forced to reload because of lag....

[Lisias]

10 hours ago, steve_v said:

I can think of no justification at all for the ever-increasing memory footprint, especially in a managed runtime... Unless of course that runtime is also excrementse.

The runtime is way....  More adequate to be used as fertilizer than as memory manager. :-)

Unity did some "optimizations" on the Mono Garbage Collector. Given the way Unity does things, this alone is already a concern. However, Unity has also a policy to never give memory back to the system. They alleged that giving memory back and then asking again for it is expensive, and once a memory was needed once, it will probably be needed again.

What's is not wrong, but what they don't get is that swapping memory to disk is a dozen orders of magnitude more expensive and it's exactly what the Operating System will do as soon someone else ask for the memory Unity is compulsively hoarding.

Now add the memory leaks to this near sociopathic behaviour... 

They manage to get away with this stunt on small to medium games (1 to 2 G ram). But KSP is a monster, almost an Operating System itself - and Unity is consistently borking on it.

 

Edited August 25, 2018 by Lisias
Better grammars

[steve_v]

56 minutes ago, JoE Smash said:

How many patches and updates have there been since 2014?

Not sure off the top of my head, but I am pretty sure this and most of the other game engine problems have been around since I first bought the game, at version 0.19, in 2013.
KSP has been plagued by game engine bugs and appalling memory requirements since day one. Hell, there was a long period where there was no usable 64bit Windows build at all, so running part mods on that platform was almost impossible.

Squad can't or won't fix the lousy game engine they committed to so long ago, and they can't or won't port the game to something else. So we get ridiculous memory requirements, ever-present memory leaks, infuriating garbage collection freezes and generally awful mostly single-threaded performance.
The problem is Unity3D, as always, and Squad doesn't touch unity problems.

Performance Improvements have been made, for which I am grateful... But most of those revolved around a game engine update and brought a raft of new bugs and regressions. I'm still waiting for joystick support on Linux, which they completely broke in the last major update, to be fixed.
Since this is another game engine problem, it probably never will be fixed, just like the random crashing that flushed the entire 1.1 series down the toilet. If there's a 1.5, I expect it will break something else important. Probably wheels, as usual.

Don't get me wrong, I love the game. I'm just getting a little impatient waiting for the finished, stable and performant game I anticipated when I backed the alpha. The same old performance problems and the same old bugs 5 years later is pretty disappointing... to say nothing of the new ones.

I'd love to report a bug against the memory leak, but since no source code or profiling tools are available, I can't submit anything useful.

 

12 minutes ago, Lisias said:

The runtime is way....  More adequate to be used as fertilizer then memory manager.

I've said it before, and I'll say it again: Mono was not designed for performance-critical real-time applications. Unity used it because it's free, that's all. Cheap game engine, cheap components, crap performance.

 

12 minutes ago, Lisias said:

Unity did some "optimizations" on the Mono Garbage Collector. Given the way Unity do things, this alone is already a concern.

I know, I've poked about in the source pretty extensively. It' a mess. "s/optimization/butchery/g"
 

12 minutes ago, Lisias said:

They alleged that giving memory back and then asking again for it is expensive.

On a modern OS, this is pretty ridiculous. The Linux memory subsystem at any rate is certainly no slouch. I can't speak to Windows, but I'd expect similar performance.
And as you say, not letting the OS manage memory can be a extremely expensive under some fairly common conditions. Having the OOM killer nail your app every few hours is expensive too.

 

Edited August 25, 2018 by steve_v

[Kowgan]

The question has been pretty much answered, now. Thanks everyone. And at this point, we pretty much can assume SQUAD won't fix the memory leaks that have been around forever, despite the countless requests we had over time. The stock game doesn't crash often, and they technically don't have any obligation to make the game run smoothly while modded.

What I can't figure out is how streamers and many other players can run a heavily modded game on Windows for 12+ hours without heavy stuttering and/or crashes.
Anyhow, In my case, I have been careful-ish and avoided too many relaunches in a short time, so I have dodged any further crashes so far.

[steve_v]

8 minutes ago, Kowgan said:

The question has been pretty much answered, now. Thanks everyone.

Yeah, it was probably answered long before I put my oar in too.
Coming back from dinner to find my system swapping and KSP using 27GB of RAM just put me in a complaining mood. It tends to do that. How it can leak even while the game is paused is beyond me.

Edited August 25, 2018 by steve_v

[Kowgan]

@steve_v

I figured.
 

2 minutes ago, steve_v said:

How it can leak even while the game is paused is beyond me.

Since I learned a copy of Windows will constantly eat disk space over time even when it stays fresh clean, I believe anything, heh.

Edited August 25, 2018 by Kowgan

[steve_v]

4 minutes ago, Kowgan said:

Since I learned a copy of Windows will constantly eat disk space over time even when it stays fresh clean, I believe anything, heh.

Almost certainly the abomination that is windows update. Some things never change.
Real glad I don't have to deal with that anymore, I traded it for hours of GCC grinding away (Gentoo).

[JoE Smash]

Sorry, I'm new, I stroaked the fire because I'm already quite aggravated with all of the above and I just got here....

Only been playing a couple months now and I've already had to delete everything and start over four times now because something has gone tragically wrong with my game due to some major error with something.

I already lost all ambition of playing mostly due to a lack of any support or feedback from Squad.

I mistakenly thought the support forums were to get support from the game devs. I didn't realize they actually ignore the forums and only other players or modders offer support. The only staff I see is moderators moving threads because they are in the wrong place, and then leaving the answering of the actual questions to whomever is bored enough to look at someone's log....

I suppose it's time for me to play something else....

Thanks for the chat!

[Lisias]

2 hours ago, steve_v said:

I've said it before, and I'll say it again: Mono was not designed for performance-critical real-time applications. Unity used it because it's free, that's all. Cheap game engine, cheap components, crap performance.

Not only Mono, but Java and any other Garbage Collector based runtime - including C++ ones.

To make GC work on Real Time systems, you need to satisfy a lot of constraints and demandings that, usually, renders manual memory management easier to implement.  

Only now, after what… 20, 25 years? Java has a somewhat usable Real Time kind of friendly GC (G1GC). But frankly… Who wants to do business with Oracle?  

Now take all this investment and try to figure out if isn't cheaper to just teach the !#$##!@# developer to handle memory and apply best practices and static tests when writing Real Time applications. 

 

2 hours ago, steve_v said:

On a modern OS, this is pretty ridiculous. The Linux memory subsystem at any rate is certainly no slouch. I can't speak to Windows, but I'd expect similar performance. And as you say, not letting the OS manage memory can be a extremely expensive under some fairly common conditions. Having the OOM killer nail your app every few hours is expensive too.

A call to such OS function will be, always, more expensive than just allocating something from your heap. It will incurs on a context change to the privileged ring, a search on the memory tables for a suitable memory block (and sometimes repacking the free memory to build such block by "force"), mapping it into the Process Descriptor tables, zeroing it for security reasons and some more details I don't remember right now. It is a very expensive call.

But yes... It's way more expensive not to do it. It's plain appalling that someone that calls itself "Professional Developer" just ignore this simple fact.

"We don't aim to serve games with such long lifespan" it would be an acceptable answer. Ok, it's a low quality and cheap Engine for casual, 15 to 30 minutes session, small games. It would be a honest answer. But this "The O.S. does it wrong, we know better" approach? Really?

 

2 hours ago, Kowgan said:

Since I learned a copy of Windows will constantly eat disk space over time even when it stays fresh clean, I believe anything, heh.

It's due an abomination called WinSXS. Every time a program is installed, a "snapshot" of every DLL it uses is recorded by its installer. So, if another program or system update happens, this program still uses its own well known "copy" of the DLLs.

With time and updates, you end up with almost a private copy of almost entire O.S. for every program you have.  

As an emergencial hack to keep things working while a proper solution is worked out, it's acceptable. Unfortunately, Microsoft have a notion of "temporary" that appears to be incompatible with Reason.

 

2 hours ago, steve_v said:

Almost certainly the abomination that is windows update. Some things never change.
Real glad I don't have to deal with that anymore, I traded it for hours of GCC grinding away (Gentoo).

You too? Another Follower of the Sacred Way of a SystemD-less System?  

 

2 hours ago, Kowgan said:

What I can't figure out is how streamers and many other players can run a heavily modded game on Windows for 12+ hours without heavy stuttering and/or crashes. Anyhow, In my case, I have been careful-ish and avoided too many relaunches in a short time, so I have dodged any further crashes so far.

Stuttering can be delayed by using MemGraph. You "pre-allocated" some memory ahead before it's really needed, so it would take sometime before the GC is needed - that stuttering is essentially the GarbageCollector freezing everything in order to reclaim unused memory.

However, you do that too much, when finally the GC kicks in there're so many garbage to be collected that the freeze takes too much, and something else on the Engine blows up due the freeze.

The GC must halt every other thread in order to do its thing. It's a limitation of the technology - threaded GC's are some of the harder problems to be cracked on this trade.

About the 12+ hours gaming sessions - there's a unspoken true: some of us (me included) don't only mod the game, but we mod the mods themselves. We profile, detect and fix the performance/memory leaking problems for ourselves. It's the reason some old school players are still using 1.3.1 or even 1.2.2 - the effort to redo all that work is too much for the benefits they will get in exchange.

You can count about 1.000 to 5.000 lines of code rewritten and months and months of work. Not for the faint of heart.

Edited August 25, 2018 by Lisias
grammars. #sigh

[RedKraken]

https://www.reddit.com/r/KerbalSpaceProgram/comments/56117i/we_should_explore_the_idea_of_an_open_source/

https://www.reddit.com/r/OpenSpaceProgram/

[steve_v]

11 hours ago, Lisias said:

A call to such OS function will be, always, more expensive than just allocating something from your heap.

Sure, if your heap is a sensible size...
Hoarding so much memory that you cause undesirable side-effects on the rest of the system, not to mention shooting yourself in the foot by forcing the OS to swap your pages to disk, that's both stupid and obnoxious.
Honestly, I'm not sure Unity is entirely to blame here, it was a conscious decision early in KSP development to have "no loading screens", and not design a dynamic asset loader.... which really just means everything in RAM, all the time.
 

11 hours ago, Lisias said:

It's due an abomination called WinSXS. Every time a program is installed, a "snapshot" of every DLL it uses is recorded by its installer.

I figured something along those lines was (still) happening. I haven't been that deep in winblows for a long while, but I do vaguely remember the private copies of libs thing. A bit nuts if you ask me.

 

11 hours ago, Lisias said:

Another Follower of the Sacred Way of a SystemD-less System?

Another Follower of the Sacred Way of a Choice-more System
I have no big problem with SystemD itself, I have a problem with it being mandatory on many distros. But yeah, I'm not real excited about svchost for Linux, my grubby shell scripts work just fine. Unix philosophy and all that.