• 0
Sign in to follow this  
Kowgan

Crashes upon reloading VAB/SPH/Launch excessively

Question

KSP: 1.4.5 x64

Problem: KSP crashes when either reloading the VAB/SPH or Launching a vessel repeatedly in a short amount of time. Or just generally reloading those scenes too many times, nor necessarily  in a short time.

Mods installed:

Spoiler

Toolbar 1.7.17.11
USITools 0.12
ABCORS 0.5.0.1
ADEPT 1.0*
Procedural Wings 0.50
Better Burn Time 1.6.1
Bluedog's LEM*
Brake Toggle (recompiled it myself for KSP 1.4.5)
Community Category Kit 3.1
Community Resource Pack 0.10
Community Trait Icons 1.0
Contract Configurator 1.25
Contract Packs:
-- Field Research 1.2.1
-- GAP 1.4
-- Kerbal Academy 1.3.0.1
-- Tourism 1.5.2
Crowd Sourced Science 4.1.2
DMagic's Orbital Science 1.3.0.12
Firespitter Plugin 7.9
Interstellar Fuel Switch 3.6.4.1
KAS 2.6
KIS 1.14
MechJeb 2.7.3
Mouse Aim Flight 1.1.1 (recompiled it myself for KSP 1.4.5)
NavBall Docking Alignmen tIndicator CE 1.0.4.2
Patch Manager 0.0.16
Kerbal Planetary Base Inc 1.6.4
Procedural Fairings 1.4.3.4
RealChute 1.4.6
SCANsat 1.1.8.7
Science Full Reward 3.0
Station Science 2.4.2
Surface Experiment Pack 2.6
Survey Transponder 0.5.2a*
USI Akita rover 0.5*
USI Life Support 0.9
USI Malemute 0.6*
USI Sounding Rockets 0.9*
Variable Thrust Limiter 0.1.5 (recompiled it myself for KSP 1.4.5)
Waypoint Manager 2.7.2

*Simple parts mod, not used or loaded into the editor or crafts used during crashes.

Reproduction steps:

Load a save, re-enter VAB/SPH/Launch vessel and Revert to building multiple times.

Logs:

output_log.txt

error.log

Notes:

I was unable to reproduce the issue on Stock KSP, so I might try to narrow it down to a specific mod. But it might be difficult due to the inconsistency of the issue. Sometimes I am able to crash the game within a couple of reloads - Other times, it takes me several attempts unitl it crashes.

After a quick research, I've found that usually 0xc0000005 errors are related to the lack of available RAM.
In my case, I have 16GB of RAM,  running the 64bit version of KSP and having around 50% memory free at the crash moment. KSP was using around 4GB of RAM at that moment.

Any help is much appreciated.

Edited by Kowgan

Share this post


Link to post
Share on other sites

Recommended Posts

  • 1
57 minutes ago, Kowgan said:

And I didn't keep reentering the SPH/Runway every 30 seconds or so.

I'm a dumb SAS. Until now, I completely missed about this important piece of information besides being advertised ad nauseaum. :D 

It's still a guess, but now a more informed one.

Joysticks are not needed except on Flight Scenes. So, it's reasonable to assume that the XInput are "opened" when you enter the Flight Scene, and "closed" when you exit it/enter VAB, SPHet all.

It's also well known that our "friend" Unity just don't releases memory no matter what. And since it's well known its memory leakings, it's plain clear that memory footprint is aways on an ascent - what varies is the slop.

So:

  1. User builds a plane/rocket.
  2. User enters Flight Scene
    1. XInput is "opened", and a memory buffer is allocated for it.
  3. User flies this toy
    1. Unity globs some more memory, raising the footprint a bit
  4. User goes to a non-flight scene (Mission Control, Tracking Station, whatever)
    1. XInput is "closed", and the current used buffer is released
    2. This memory is not under Unity's control
  5. User goes back  to flying
    1. XInput is "opened again".
    2. Since the previously used buffer was released and probably already reused for something else, a new buffer is allocated

As the memory footprint grows, eventually this new buffer is allocated beyond the 4Gb boundary and then things goes KABOOM on 32 bits DLL.

@Just Jim, I think we nailed down a deterministic Modus Operandi in order to reproduce the issue. A small custom part that "eats memory while flying" would speed up the process. Perhaps MemGraph could do the trick too.

Edited by Lisias
typos, what else? =P

Share this post


Link to post
Share on other sites
  • 0

Did you get a different error when you did the xinput1_3.dll swap?

Like did that disappear from the error log at least?

I've never seen "Bonjour" on the list before...

What's that do? Can you disable/close it when running the game?

https://apple.stackexchange.com/questions/45765/do-i-really-need-bonjour-on-windows

I'd do the xinput1_3.dll swap AND ditch bonjour and see if the game is stable....

Share this post


Link to post
Share on other sites
  • 0

No, \system32\xinput1_3.dll was still there when I swapped these DLL files.

Bonjour is a service related to Apple iTunes. I have no clue why it shows up on this error log, though. I should be able to disable it, although I'm not sure how it would affect the game. Then again, nothing in this has made much sense to me so far.

Edited by Kowgan

Share this post


Link to post
Share on other sites
  • 0

I'm saying I've looked at quite a few error logs at this point and I've not yet seen one with bonjour on it....

Xinput1_3.dll is on pretty much EVERY error log I have seen....

I'm just going by the fact that it is unusual to see it listed. It may be a hint....lol.

Share this post


Link to post
Share on other sites
  • 0

I've uninstalled Bonjour (it's useless to me, and I should have done it a long time ago anyway) and will try to reproduce the crash again. I'll post an updated log file if here if it crashes again. Thanks for the support, it's much appreciated :)

Share this post


Link to post
Share on other sites
  • 0

Alright, I managed to crash it again. Same methods, just constantly loading SPH and launching on the runway.

output_log.txt
error.log

I'm considering switching to OpenGL to see if things get better.

Share this post


Link to post
Share on other sites
  • 0

Have you tried to remove the mods you recompiled on your own? Also, Surface Experiment Pack creates a few exceptions...I don't think they are related but it should not be a big deal to remove it as well (at least for testing)

Share this post


Link to post
Share on other sites
  • 0

Thanks, @4x4cheesecake. I haven't tried this yet, but according to Lisias:

On 8/22/2018 at 10:09 PM, Lisias said:

I run KSP on a Mac machine, and it's usually eats up 10 to 12G of RAM.

If your KSPx64 is consistently dying at 4GB of memory allocated by the process, I definitively would bet on some 32 bits DLL somewhere in the System being used by KSP, and getting killed when KSP reaches that memory limit and the get shoot on the feet when it calls that DLL using a memory pointer above that.

If the XInput trick didn't helped you, there's a good change that something else on your system is getting into this.

Another thing is VRAM memory. Each texture you add to KSP via mods eats VRAM from the GPU Card. When you exhaust your VRAM, Unity also crashes without any clue about the reason.

I don't know if the GPU reaching its max memory should be the case since it's not VRAM, but maybe some 32bit DLL being called while the game is consuming over 4GB of RAM could be the case. Not sure.
The inconsistency of the crashes is also something to keep in mind. Lately, I've had some 3 or 4 hours sessions with no crashes. And I didn't keep reentering the SPH/Runway every 30 seconds or so.

For now, I guess I can just live with it. If I can play 3 hours with no crashes, I'll just boot it up again whenever it happens.

Share this post


Link to post
Share on other sites
  • 0

Interesting. That would explain a lot. Also, I wonder if my self-recompiled version of Mouse Aim Flight could have something to do with it. FWIW, I know it already has a glitch where it doesn't correctly remove the stockbar icon, so you can still see that mod's icon when you unload back to the main menu. I wouldn't be surprised if the root of this problem is directly related to the ever-increasing memory comsumption as well.

I'm tempted to replace the system32\xinput dll file with the one provided by KSP and see what explodes.

Edited by Kowgan

Share this post


Link to post
Share on other sites
  • 0
9 minutes ago, Kowgan said:

Interesting. That would explain a lot. Also, I wonder if my self-recompiled version of Mouse Aim Flight could have something to do with it. 

I think it would be another "victim", I gave a peek on the source and I only saw calls to UnityEngine, not "direct calls" to any native DLL.

 

12 minutes ago, Kowgan said:

I'm tempted to replace the system32\xinput dll file with the one provided by KSP and see what explodes.

Most games doesn't eats memory as candy as KSP. I don't expect too much fireworks. :) 

Share this post


Link to post
Share on other sites
  • 0
2 hours ago, Kowgan said:



I'm tempted to replace the system32\xinput dll file with the one provided by KSP and see what explodes.

I actually did that first....that ALSO works...

My concern was any OTHER game that wanted a 64bit version of xinput1_3.dll would then get the wrong thing and crash....because the KSP one is 32 bit....

That worked equally well as putting the windows/system32 on in the games 64bit plugin folder.....

Share this post


Link to post
Share on other sites
  • 0
17 hours ago, Lisias said:

It's also well known that our "friend" Unity just don't releases memory no matter what. And since it's well known its memory leakings, it's plain clear that memory footprint is aways on an ascent - what varies is the slop.

KSP leaks like a sieve, always has. Here's a quick & dirty graph from my modded 1.4.5, those spikes are scene changes:
mem-graph.png
I can produce something similar with a stock install if anyone wants more proof of this, it looks almost exactly the same, though the increase in memory consumption is slightly slower.
This machine has 32GB of RAM, and I'm getting close to 40% here. Properly ridiculous, innit?
I have no idea how a game can possibly justify using so much memory, except through lousy design decisions, and lousier engineering.
I can think of no justification at all for the ever-increasing memory footprint, especially in a managed runtime... Unless of course that runtime is also excrementse.

 

Edited by steve_v

Share this post


Link to post
Share on other sites
  • 0
7 hours ago, steve_v said:

KSP leaks like a sieve, always has. Here's a quick & dirty graph from my modded 1.4.5, those spikes are scene changes:
mem-graph.png
I can produce something similar with a stock install if anyone wants more proof of this, it looks almost exactly the same, though the increase in memory consumption is slightly slower.
This machine has 32GB of RAM, and I'm getting close to 40% here. Properly ridiculous, innit?
I have no idea how a game can possibly justify using so much memory, except through lousy design decisions, and lousier engineering.
I can think of no justification at all for the ever-increasing memory footprint, especially in a managed runtime... Unless of course that runtime is also excrementse.

 

I have noticed the same issue.

It would be great if @SQUAD would fix this at some point...

How many patches and updates have there been since 2014?

Still the game eats more and more RAM the longer you play and the more launches you do....until eventually it crashes or you are forced to reload because of lag....

Share this post


Link to post
Share on other sites
  • 0
10 hours ago, steve_v said:

I can think of no justification at all for the ever-increasing memory footprint, especially in a managed runtime... Unless of course that runtime is also excrementse.

The runtime is way....  More adequate to be used as fertilizer than as memory manager. :-)

Unity did some "optimizations" on the Mono Garbage Collector. Given the way Unity does things, this alone is already a concern. However, Unity has also a policy to never give memory back to the system. They alleged that giving memory back and then asking again for it is expensive, and once a memory was needed once, it will probably be needed again.

What's is not wrong, but what they don't get is that swapping memory to disk is a dozen orders of magnitude more expensive and it's exactly what the Operating System will do as soon someone else ask for the memory Unity is compulsively hoarding.

Now add the memory leaks to this near sociopathic behaviour... 

They manage to get away with this stunt on small to medium games (1 to 2 G ram). But KSP is a monster, almost an Operating System itself - and Unity is consistently borking on it.

 

Edited by Lisias
Better grammars

Share this post


Link to post
Share on other sites
  • 0
56 minutes ago, JoE Smash said:

How many patches and updates have there been since 2014?

Not sure off the top of my head, but I am pretty sure this and most of the other game engine problems have been around since I first bought the game, at version 0.19, in 2013.
KSP has been plagued by game engine bugs and appalling memory requirements since day one. Hell, there was a long period where there was no usable 64bit Windows build at all, so running part mods on that platform was almost impossible.

Squad can't or won't fix the lousy game engine they committed to so long ago, and they can't or won't port the game to something else. So we get ridiculous memory requirements, ever-present memory leaks, infuriating garbage collection freezes and generally awful mostly single-threaded performance.
The problem is Unity3D, as always, and Squad doesn't touch unity problems.

Performance Improvements have been made, for which I am grateful... But most of those revolved around a game engine update and brought a raft of new bugs and regressions. I'm still waiting for joystick support on Linux, which they completely broke in the last major update, to be fixed.
Since this is another game engine problem, it probably never will be fixed, just like the random crashing that flushed the entire 1.1 series down the toilet. If there's a 1.5, I expect it will break something else important. Probably wheels, as usual.

Don't get me wrong, I love the game. I'm just getting a little impatient waiting for the finished, stable and performant game I anticipated when I backed the alpha. The same old performance problems and the same old bugs 5 years later is pretty disappointing... to say nothing of the new ones.

I'd love to report a bug against the memory leak, but since no source code or profiling tools are available, I can't submit anything useful.

 

12 minutes ago, Lisias said:

The runtime is way....  More adequate to be used as fertilizer then memory manager.

I've said it before, and I'll say it again: Mono was not designed for performance-critical real-time applications. Unity used it because it's free, that's all. Cheap game engine, cheap components, crap performance.

 

12 minutes ago, Lisias said:

Unity did some "optimizations" on the Mono Garbage Collector. Given the way Unity do things, this alone is already a concern.

I know, I've poked about in the source pretty extensively. It' a mess. "s/optimization/butchery/g"
 

12 minutes ago, Lisias said:

They alleged that giving memory back and then asking again for it is expensive.

On a modern OS, this is pretty ridiculous. The Linux memory subsystem at any rate is certainly no slouch. I can't speak to Windows, but I'd expect similar performance.
And as you say, not letting the OS manage memory can be a extremely expensive under some fairly common conditions. Having the OOM killer nail your app every few hours is expensive too.

 

Edited by steve_v

Share this post


Link to post
Share on other sites
  • 0

The question has been pretty much answered, now. Thanks everyone. And at this point, we pretty much can assume SQUAD won't fix the memory leaks that have been around forever, despite the countless requests we had over time. The stock game doesn't crash often, and they technically don't have any obligation to make the game run smoothly while modded.

What I can't figure out is how streamers and many other players can run a heavily modded game on Windows for 12+ hours without heavy stuttering and/or crashes.
Anyhow, In my case, I have been careful-ish and avoided too many relaunches in a short time, so I have dodged any further crashes so far.

Share this post


Link to post
Share on other sites
  • 0
8 minutes ago, Kowgan said:

The question has been pretty much answered, now. Thanks everyone.

Yeah, it was probably answered long before I put my oar in too. :P
Coming back from dinner to find my system swapping and KSP using 27GB of RAM just put me in a
complaining mood. It tends to do that. How it can leak even while the game is paused is beyond me. :confused:

Edited by steve_v

Share this post


Link to post
Share on other sites
  • 0

@steve_v

047028a214.png

I figured. :P
 

2 minutes ago, steve_v said:

How it can leak even while the game is paused is beyond me. :confused:

Since I learned a copy of Windows will constantly eat disk space over time even when it stays fresh clean, I believe anything, heh.

Edited by Kowgan

Share this post


Link to post
Share on other sites
  • 0
4 minutes ago, Kowgan said:

Since I learned a copy of Windows will constantly eat disk space over time even when it stays fresh clean, I believe anything, heh.

Almost certainly the abomination that is windows update. Some things never change.
Real glad I don't have to deal with that anymore, I traded it for hours of GCC grinding away (Gentoo).

Share this post


Link to post
Share on other sites
  • 0

Sorry, I'm new, I stroaked the fire because I'm already quite aggravated with all of the above and I just got here....

Only been playing a couple months now and I've already had to delete everything and start over four times now because something has gone tragically wrong with my game due to some major error with something.

I already lost all ambition of playing mostly due to a lack of any support or feedback from Squad.

I mistakenly thought the support forums were to get support from the game devs. I didn't realize they actually ignore the forums and only other players or modders offer support. The only staff I see is moderators moving threads because they are in the wrong place, and then leaving the answering of the actual questions to whomever is bored enough to look at someone's log....

I suppose it's time for me to play something else....

Thanks for the chat!

Share this post


Link to post
Share on other sites
  • 0
2 hours ago, steve_v said:

I've said it before, and I'll say it again: Mono was not designed for performance-critical real-time applications. Unity used it because it's free, that's all. Cheap game engine, cheap components, crap performance.

Not only Mono, but Java and any other Garbage Collector based runtime - including C++ ones.

To make GC work on Real Time systems, you need to satisfy a lot of constraints and demandings that, usually, renders manual memory management easier to implement. :P 

Only now, after what… 20, 25 years? Java has a somewhat usable Real Time kind of friendly GC (G1GC). But frankly… Who wants to do business with Oracle? :/ 

Now take all this investment and try to figure out if isn't cheaper to just teach the !#$##!@# developer to handle memory and apply best practices and static tests when writing Real Time applications. :huh:

 

2 hours ago, steve_v said:

On a modern OS, this is pretty ridiculous. The Linux memory subsystem at any rate is certainly no slouch. I can't speak to Windows, but I'd expect similar performance. And as you say, not letting the OS manage memory can be a extremely expensive under some fairly common conditions. Having the OOM killer nail your app every few hours is expensive too.

A call to such OS function will be, always, more expensive than just allocating something from your heap. It will incurs on a context change to the privileged ring, a search on the memory tables for a suitable memory block (and sometimes repacking the free memory to build such block by "force"), mapping it into the Process Descriptor tables, zeroing it for security reasons and some more details I don't remember right now. It is a very expensive call.

But yes... It's way more expensive not to do it. It's plain appalling that someone that calls itself "Professional Developer" just ignore this simple fact.

"We don't aim to serve games with such long lifespan" it would be an acceptable answer. Ok, it's a low quality and cheap Engine for casual, 15 to 30 minutes session, small games. It would be a honest answer. But this "The O.S. does it wrong, we know better" approach? Really?

 

2 hours ago, Kowgan said:

Since I learned a copy of Windows will constantly eat disk space over time even when it stays fresh clean, I believe anything, heh.

It's due an abomination called WinSXS. Every time a program is installed, a "snapshot" of every DLL it uses is recorded by its installer. So, if another program or system update happens, this program still uses its own well known "copy" of the DLLs.

With time and updates, you end up with almost a private copy of almost entire O.S. for every program you have. :/ 

As an emergencial hack to keep things working while a proper solution is worked out, it's acceptable. Unfortunately, Microsoft have a notion of "temporary" that appears to be incompatible with Reason.

 

2 hours ago, steve_v said:

Almost certainly the abomination that is windows update. Some things never change.
Real glad I don't have to deal with that anymore, I traded it for hours of GCC grinding away (Gentoo).

You too? Another Follower of the Sacred Way of a SystemD-less System? :D 

 

2 hours ago, Kowgan said:

What I can't figure out is how streamers and many other players can run a heavily modded game on Windows for 12+ hours without heavy stuttering and/or crashes. Anyhow, In my case, I have been careful-ish and avoided too many relaunches in a short time, so I have dodged any further crashes so far.

Stuttering can be delayed by using MemGraph. You "pre-allocated" some memory ahead before it's really needed, so it would take sometime before the GC is needed - that stuttering is essentially the GarbageCollector freezing everything in order to reclaim unused memory.

However, you do that too much, when finally the GC kicks in there're so many garbage to be collected that the freeze takes too much, and something else on the Engine blows up due the freeze.

The GC must halt every other thread in order to do its thing. It's a limitation of the technology - threaded GC's are some of the harder problems to be cracked on this trade.

About the 12+ hours gaming sessions - there's a unspoken true: some of us (me included) don't only mod the game, but we mod the mods themselves. We profile, detect and fix the performance/memory leaking problems for ourselves. It's the reason some old school players are still using 1.3.1 or even 1.2.2 - the effort to redo all that work is too much for the benefits they will get in exchange.

You can count about 1.000 to 5.000 lines of code rewritten and months and months of work. Not for the faint of heart.

Edited by Lisias
grammars. #sigh

Share this post


Link to post
Share on other sites
  • 0
11 hours ago, Lisias said:

A call to such OS function will be, always, more expensive than just allocating something from your heap.

Sure, if your heap is a sensible size...
Hoarding so much memory that you cause undesirable side-effects on the rest of the system, not to mention shooting yourself in the foot by forcing the OS to swap your pages to disk, that's both stupid and obnoxious. :confused:
Honestly, I'm not sure Unity is entirely to blame here, it was a conscious decision early in KSP development to have "no loading screens", and not design a dynamic asset loader.... which really just means everything in RAM, all the time.
 

11 hours ago, Lisias said:

It's due an abomination called WinSXS. Every time a program is installed, a "snapshot" of every DLL it uses is recorded by its installer.

I figured something along those lines was (still) happening. I haven't been that deep in winblows for a long while, but I do vaguely remember the private copies of libs thing. A bit nuts if you ask me.

 

11 hours ago, Lisias said:

Another Follower of the Sacred Way of a SystemD-less System?

Another Follower of the Sacred Way of a Choice-more System :P
I have no big problem with SystemD itself, I have a problem with it being mandatory on many distros. But yeah, I'm not real excited about svchost for Linux, my grubby shell scripts work just fine. Unix philosophy and all that.

Share this post


Link to post
Share on other sites
  • 0
2 hours ago, steve_v said:

Honestly, I'm not sure Unity is entirely to blame here, it was a conscious decision early in KSP development to have "no loading screens", and not design a dynamic asset loader.... which really just means everything in RAM, all the time.

Frankly? Knowing what I know from Unity, this would not be a (new) issue.

Dynamic Loading Assets would reduce the initial memory footprint (and the initial loading time) by a mile , but would also make the memory problems worse. Try to imagine the pathetic Unity's GC handling, also, many megabytes of binary assets being loaded and unloaded on demand on playing time...

 

2 hours ago, steve_v said:

Sure, if your heap is a sensible size...

I have no big problem with SystemD itself, I have a problem with it being mandatory on many distros. But yeah, I'm not real excited about svchost for Linux, my grubby shell scripts work just fine. Unix philosophy and all that.

SystemD had bitten me on the SAS more than once. This thing manages to be worse than Unity's GC, and you are a happy man if you didn't had to figure it out by yourself. :) 

Share this post


Link to post
Share on other sites
This thread is quite old. Please consider starting a new thread rather than reviving this one.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this