Game crashes and takes other programs down with it

[Infinite_Maelstrom]

Hello,

I have been having this issue with my most-modded install of KSP for about 20 months, but it has been getting worse and now KSP will not even fully load before it crashes - where previously I could play for at least 30 mins or more before a crash occurred.  (Windows 10 x64; AMD Ryzen 2400G; GTX 1660Ti; 16GB RAM; 105GB free on KSP install drive, 1.84GB free on system install drive; KSP 1.12.5; 128 mods, for modlist see attached files.)

Situation:
When KSP is loading, or when in space (usually switching craft or views), but especially when opening a save, the game will suddenly and unexpectedly crash.  Sometimes a runtime error window will appear, not always though.  (I'm afraid I don't have a screenshot of the error window.)  However, one thing always seems to happen:  all open tabs in Firefox will also crash, needing to be reloaded, and if Windows nightlight is on, it will crash too (needing to be turned off and on again to start working).

I have on a couple of occasions attempted to narrow down the culprit, but because of the unpredictable nature of the bug, and the number of mods in this save, this has so far been unsuccessful.

This evening, I had enough free time to play KSP for the first time in about 8 months.  I first ensured that all of my mods were fully up-to-date (I needed to use this workaround to force CKAN to notice the most recent versions of each mod).  But then KSP crashed 4 or more times in a row due to this bug and I decided I needed to go to bed instead.

I have attached all the logs I could find from the most recent crash.  This crash occurred during loading rather than during opening a save, but hopefully it is still helpful.  Firefox itself did not crash this evening (sometimes it does), only all the open tabs did.  The only logs for Firefox I can find are records of Firefox crashing completely, therefore this evening's error is not recorded, and so I have not included it in the attached logs.

 

All logs that I could find from this evenings latest crash (some may be from earlier crashes).  Modlist is also included in this drive folder: https://drive.google.com/drive/folders/1GtKJdmbLnmkG7A6-wAy4crQpU0yDnd-1?usp=sharing

 

Hopefully someone is able to make some sense of this issue, it has been frustrating me for a long time.

Thanks

I.M.

Edited March 16, 2023 by Infinite_Maelstrom
Made system specs clearer

[jimmymcgoochie]

Have you tried not running other programs at the same time as KSP? Does it still crash if you’re only running KSP and nothing else?

With over a hundred mods and only 16GB RAM it’s possible that you’re just running out of memory, which would explain why the logs end abruptly with no obvious signs of an error during loading. Try removing some mods (especially those with lots of parts/large parts/highly detailed interior views) as this could reduce memory use by KSP.

This may be unrelated, but your system drive is pretty full and clearing some space might make things run a bit faster which could help?

[Lisias]

Sounds like the swapfile is getting without space for growing.

Usually the swapfile is set to be on the boot drive, and it's nearly full in your case. 

@jimmymcgoochie's hunch appears to be right.

You need to have at least the same amount of memory as free disk space on the drive where th4 swapfile is to be safe on heavy demand.

Alternatively you move the swapfile itself to other disk. I like to create a dedicated volume on my fastest disk for the swapfile. 

Give this a try:

https://www.windowscentral.com/how-move-virtual-memory-different-drive-windows-10

[Infinite_Maelstrom]

So I haven't gone to bed yet...

I emptied windows' recycle bin, this cleared up 22GB on my KSP drive and 240MB on my C: drive.  Then I closed all the non-essential programs I could find, and restarted KSP (also closing CKAN afterwards).  This worked a little better, getting to the home screen.  I opened a save, and after waiting a long time, KSP crashed.  This time the night light did not crash though.  Shortly afterwards, however, when I was trying to get the logs from file explorer, the "windows application" crashed (according to the pop-up error dialogue).  The PC did not BlueScreen, but the screen did go blue with nothing but an empty firefox window for a few minutes, then it recovered.  I think this lends credence to the theory that the lack of space on my C: drive might be amplifying the issue, even if it is not causing it directly.

In any case, the logs for this crash - including windows error reports from a minute or two later - are here: https://drive.google.com/file/d/1p9AecmZzsKeKPlHIpauFUxsbPRCtFwJc/view?usp=share_link 

 

One additional potentially interesting note is that often, just before KSP crashes, it will show a screen that is like an extremely over-exposed version of the current screen, except with all colours above a fairly low-ish luminosity shown as black.  This does not always happen before a crash, and a crash does not always occur after this happens, but they are very strongly correlated.

[Lisias]

11 hours ago, Infinite_Maelstrom said:

 

One additional potentially interesting note is that often, just before KSP crashes, it will show a screen that is like an extremely over-exposed version of the current screen, except with all colours above a fairly low-ish luminosity shown as black.  This does not always happen before a crash, and a crash does not always occur after this happens, but they are very strongly correlated.

I'm away from my personal machine now, so I'm guessing - will inspect the logs later.

But what you describes looks like GPU device driver crash. Perhaps due VRAM exhaust,  perhaps due overheating, or Kraken knows what else.

Install a temperature monitor and be sure to have it visible all the time. Stop gaming in the very instant it reaches the manufacturer's advised temperature limit.

If the problem is overheating, it's time to do some preventive maintenance on the GPU card (cleaning up dust from the sinkers and replacing old fans - better hire a skilled technician for that).

If temperature never reaches the "boiling" point on crash, then my next guess is VRAM exhaust together RAM exhaust. When you abuse the VRAM, the GPU device driver starts to borrow RAM from the CPU to use as a poor's man VRAM extension.

This is troublesome because, besides pressuring an already under heavy pressure Virtual Memory Manager, the speed in which the GPU access CPU's RAM is some orders of magnitude slower than when accessing its VRAM. And this makes everything slower including to the CPU that is suffering from contention (i.e., it can't access its RAM when the GPU is doing it). Special code under Critical Sections that doesn't expect being halted on accessing its RAM may go bonkers and crash.

Try to lower the Texture Quality on the KSP Settings page to see if things get better.

Humm... An old AMD GPU driver is known to induce GPU overheating. If you are using AMD, check if your device driver version is prone to this problem.

https://community.amd.com/t5/drivers-software/driver-causes-overheating/td-p/402596

— — — 

@Infinite_Maelstrom,  I got the logs.

Source
Windows

Summary
Hardware error

Date
.17/.03/.2023 01:33 AM

Status
Not reported

Description
A problem with your hardware caused Windows to stop working correctly.

Problem signature
Problem Event Name:     LiveKernelEvent
Code:   1b8
Parameter 1:    1
Parameter 2:    0
Parameter 3:    0
Parameter 4:    0
OS version:     10_0_19045
Service Pack:   0_0
Product:        768_1
OS Version:     10.0.19045.2.0.0.768.101
Locale ID:      5129

Looks like an issue in the NVIDIA's device driver:

https://linustechtips.com/topic/1404005-1a8-and-1b8-livekernelevents-crashing-the-gpu-drivers-please-help/

When did you updated the gpu device driver by the last time? In the link I posted, it's suggested that rolling back to version 473 fixed the issue for someone...

 

Edited March 17, 2023 by Lisias
got the log

[Infinite_Maelstrom]

I tried deleting the CKAN cache, that cleared up about 30GB of space on my C: drive.  Instead of rolling back to an old GPU driver, I updated to the latest one (531.41), hoping that would contain a fix.

Despite issues with CKAN, KSP managed to load further than before - I got to the sound of birds chirping before it crashed.  (The strange colour issue also occurred, but had resolved by the time the crash actually happened.)

Unfortunately I had absent-mindedly closed my HWMonitor 'to save RAM,' so I don't know if the GPU overheated or anything like that.

When it crashed, this time Firefox's tabs did not crash, but windows 10 nightlight did.  A popup appeared with the following:
    Error in GC
        Virtual alloc remapping failed

(forgot to screenshot, pretty sure that's what it said though)

When I clicked 'OK', then Firefox's tabs crashed.

I have included the same selection of logs as last time (this time with 7-zip for better compression).
https://drive.google.com/file/d/1zC0tZWxGzckXHxSUSJ6ac9Wlf7xEPh6R/view?usp=sharing
This time, the Windows logs STRONGLY suggest that the failure was caused by a lack of RAM.  (Note, in the .evtx file, the event that occurred at 11:48:07pm was when I clicked "OK" on the error pop-up window - I had left it open for a while to try to understand what it meant.  The other errors are roughly simultaneous with the observed crash.)

 

Edit:  scrolling back through the windows logs to the last time I opened KSP, I see that there are 4 very similar memory-related errors there too.  I must have missed them when uploading the last set of logs.

Edited March 29, 2023 by Infinite_Maelstrom

[Lisias]

9 hours ago, Infinite_Maelstrom said:

I tried deleting the CKAN cache, that cleared up about 30GB of space on my C: drive.  Instead of rolling back to an old GPU driver, I updated to the latest one (531.41), hoping that would contain a fix.

Despite issues with CKAN, KSP managed to load further than before - I got to the sound of birds chirping before it crashed.  (The strange colour issue also occurred, but had resolved by the time the crash actually happened.)

Everything pinpoints you exhausted your memory space:

		<EventData>
			<Data Name="Caption">Windows - Out of Virtual Memory</Data>
			<Data Name="Message">Your system is low on virtual memory. To ensure that Windows runs properly, increase the size of your virtual memory paging file. For more information, see Help. </Data>
		</EventData>

----

		<UserData>
			<MemoryExhaustionInfo xmlns="http://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events">
				<SystemInfo>
					<SystemCommitLimit>29705191424</SystemCommitLimit>
					<SystemCommitCharge>29649117184</SystemCommitCharge>
					<ProcessCommitCharge>26075082752</ProcessCommitCharge>
					<PagedPoolUsage>701210624</PagedPoolUsage>
					<PhysicalMemorySize>17122279424</PhysicalMemorySize>
					<PhysicalMemoryUsage>16458653696</PhysicalMemoryUsage>
					<NonPagedPoolUsage>397881344</NonPagedPoolUsage>
					<Processes>200</Processes>
				</SystemInfo>
				<PagedPoolInfo>
					<Tag_1>
						<Name>MmSt</Name>
						<PoolUsed>36415600</PoolUsed>
					</Tag_1>
					<Tag_2>
						<Name>Ntff</Name>
						<PoolUsed>33440000</PoolUsed>
					</Tag_2>
					<Tag_3>
						<Name>FMfn</Name>
						<PoolUsed>24068640</PoolUsed>
					</Tag_3>
				</PagedPoolInfo>
				<NonPagedPoolInfo>
					<Tag_1>
						<Name>smNp</Name>
						<PoolUsed>20934656</PoolUsed>
					</Tag_1>
					<Tag_2>
						<Name>Temp</Name>
						<PoolUsed>17890064</PoolUsed>
					</Tag_2>
					<Tag_3>
						<Name>NVRM</Name>
						<PoolUsed>16757056</PoolUsed>
					</Tag_3>
				</NonPagedPoolInfo>
				<ProcessInfo>
					<Process_1>
						<Name>KSP_x64.exe</Name>
						<ID>8788</ID>
						<CreationTime>2023-03-29 10:32:24.314713</CreationTime>
						<CommitCharge>19761094656</CommitCharge>
						<HandleCount>1763</HandleCount>
						<Version>2019.4.18.4260</Version>
						<TypeInfo>201</TypeInfo>
					</Process_1>
					<Process_2>
						<Name>Accountable2You.exe</Name>
						<ID>15096</ID>
						<CreationTime>2023-03-29 06:01:21.798798</CreationTime>
						<CommitCharge>1238036480</CommitCharge>
						<HandleCount>7146</HandleCount>
						<Version>5.1.3.0</Version>
						<TypeInfo>210</TypeInfo>
					</Process_2>
					<Process_3>
						<Name>firefox.exe</Name>
						<ID>4860</ID>
						<CreationTime>2023-03-29 08:50:06.754276</CreationTime>
						<CommitCharge>612888576</CommitCharge>
						<HandleCount>3042</HandleCount>
						<Version>111.0.1.8480</Version>
						<TypeInfo>219</TypeInfo>
					</Process_3>
					<Process_4>
						<Name/>
						<ID>0</ID>
						<CreationTime>0001-01-01 00:00:00</CreationTime>
						<CommitCharge>0</CommitCharge>
						<HandleCount>0</HandleCount>
						<Version>0.0.0.0</Version>
						<TypeInfo>0</TypeInfo>
					</Process_4>
					<Process_5>
						<Name/>
						<ID>0</ID>
						<CreationTime>0001-01-01 00:00:00</CreationTime>
						<CommitCharge>0</CommitCharge>
						<HandleCount>0</HandleCount>
						<Version>0.0.0.0</Version>
						<TypeInfo>0</TypeInfo>
					</Process_5>
					<Process_6>
						<Name/>
						<ID>0</ID>
						<CreationTime>0001-01-01 00:00:00</CreationTime>
						<CommitCharge>0</CommitCharge>
						<HandleCount>0</HandleCount>
						<Version>0.0.0.0</Version>
						<TypeInfo>0</TypeInfo>
					</Process_6>
				</ProcessInfo>
				<ExhaustionEventInfo>
					<Time>2023-03-29 10:44:33.037483</Time>
				</ExhaustionEventInfo>
			</MemoryExhaustionInfo>
		</UserData>

----

		<EventData>
			<Data Name="Caption">dwm.exe - System Error</Data>
			<Data Name="Message">The process has terminated because it could not allocate additional memory. </Data>
		</EventData>

And KSP_x64 was the first victim/perpetrator.

Well, this makes things simple - not a hardware error, not a device driver problem. Just memory running out.

Last time I had to mess with Windows' swapfiles was on the Windows 7 era, so I don't think I will be of too much help now - but I found this article that can help you a bit:

https://support.robertsspaceindustries.com/hc/en-us/articles/360000083387-Out-of-memory-errors-set-your-pagefile

Check how your Virtual Memory settings are configured, I suspect they are deactivated.

You can also try to save memory by editing the KSP_ROOT/settings.cfg file. Open it on Notepad and find and change the following value to 3:

TEXTURE_QUALITY = 3

On a side note: anyone on Linux or Mac in need to parse Windows log files : https://github.com/net-lisias-p/python-evtx (a SAS saver, I say)

[Infinite_Maelstrom]

Ok, so I didn't really feel like troubleshooting the virtual memory, so I just went and bought 16GB more RAM ($86nzd, less than half the price I got the first 16GB for!). 

Guess what?  No more crashes!  Nice!  Thanks for the help @Lisias.

KSP is currently using 12.6GB RAM, with ~5GB used by the OS.  The quantity of RAM used by KSP jumps massively when it first loads into the menu screen, and again when you load into a save, explaining why my game crashed in those situations. I suspect that there was a Windows 10 update that increased the RAM requirements, after which KSP stopped being able to load.