[0.90] Possible Windows Memory Wall Crash Workaround

[bernh]

Hey Everyone,

I may have a workaround for the dreaded 3.6GB-3.7GB memory wall in Windows when running KSP 32bit

First a little background in case it makes a difference.

My system:

i5-2320 running at 3GHz

8GB RAM

Intel DZ77GA-70K

nVidia GT610 2GB Graphics Card driving 2 monitors at 1280x1024 each

Motherboard/CPU Intel graphics running a third monitor at 1024x768

Windows 7 Ultimate x64 SP1

KSP 0.90.0.705

49 mods installed, including some large ones like B9 Aerospace, KW Rocketry, Alchemy Labs, Karbonite, Regolith, KAS, KSO, KAX, MKS and OKS to name a few.

I was at a point where the game would not load, crashing about 80%-90% through the loading progress bar.

I now have the game running, with a RAM footprint of about 1.2GB sitting in the SPH or VAB.

This is what I did:

In the launcher settings

-Terrain Detail default

-SM3 Terrain Shaders Enabled

-Terrain Scatters Disabled

-Render Quality Good

-Texture Quality Half Res

-Fallback Part Shaders Disabled

-Aerodynamic FX Quality Normal

-Screen Resolution 1280x1024 (my monitor's native resolution)

-Full Screen Disabled

-Anti-Aliasing Off

-VSync Every VBlank

-Frame Limit 80FPS

Installed ddsloader - http://forum.kerbalspaceprogram.com/threads/96729-0-90-DDSLoader-1-9-%28Mar-9%29-Loads-DDS-Texture-Boringly-fast-edition

Installed KSP to DDS Texture Converter - http://forum.kerbalspaceprogram.com/threads/98672-WIN-KSP-to-DDS-texture-converter

Ran the texture converter on the GameData folder picking 1/2 resolution for the conversion

Created a shortcut to start KSP using the -force-opengl -popupwindow switches

Installed a RAM optimizer called Minimem - http://main.kerkia.com/products/minimem/download.aspx

With the Minimem GUI open, I set it to optimize every 30 seconds, and disabled the two options for Do not optimize the foreground process and Do not optimise if overall RAM available is above...

I ran KSP using the created shortcut, then found the KSP process in the right hand side of the Minimem GUI and added it to the optimize list on the left side.

Correct me if I'm wrong, but from what I have read, using ddsloader allows KSP to use the dds textures made by KSP to DDS. The force-opengl switch allows those textures to be loaded onto the RAM on the graphics card. The Minimem application then releases the RAM that those textures used while they were making their way to the GPU, but since they are now on the GPU they are no longer needed in RAM.

I have been running this setup for 5 or 6 days now with great success. I have not played around with my settings much beyond what I have outlined, so feel free to experiment if you want to try this approach.

One thing to note though, is that if you install any additional mods after the initial setup, you will need to re-run the KSP to DDS converter to convert the new textures.

Happy Gaming!

[funk]

Wow, Minimem is great it keeps ram usage always at low level... time to add more mods!!!. Thx for the hint!

[Stone Blue]

Hmmm...Think I'll wait on Minimem...

I've done everything you listed except Minimem, and have seen some nice improvements...

But that Kerkia website seems a little sketchy...And Minimem itself hasnt seen an update in over 3 years...??...Is it even still being developed or supported?...Not to mention it relies on .NET 2.0??.....

[bernh]

I agree that Minimem is a bit old, but it works, and, as an Electronics Technician and Computer Technician for the last 20 plus years, I am a firm believer in the old saying, "If it's not broke, don't fix it."

You don't need to install .NET 2.0, as long as you have a newer version. I personally have .NET 4.5.

It has not raised and flags with AVG or Malwarebytes.

I didn't jump to Minimem right away myself. After I did the other steps, and was still hitting the RAM wall, I looked for a RAM optimizer, actually trying several different ones, but Minimem is the one that worked, and unlike all the others I tried, it lets you target a specific process, instead of only working globally. Essentially, it was the last step I tried, after realizing that the other steps, while definitely helpful, were still letting the textures to keep a footprint in RAM, and while smaller than the non-DDS footprint, was still large enough to keep me from running the game.

If you are happy without having to resort to the RAM optimization step, there is no need to do it.

[Honeybadga]

Does DDS loader make ActiveTextureManagement obsolete? Is there a point in running the two side by side?

[Stone Blue]

Does DDS loader make ActiveTextureManagement obsolete?

I dont believe so...

Is there a point in running the two side by side?

Yes...If you dig around in one of the DDS threads, there are quite a few mods that wont handle being converted by DDS...

There is an exception list you can use, so DDS converter will skip them...I guess that is where ATM comes in, to pick up the slack...

Also, if I am reading some info in that thread correctly, at this time, DDS converter only handles certain file types, not ALL of them?....

As to not feeling totally comfortable downloading Minimem, I DO have a streamlined, separate install of Win7 that I use to ONLY run KSP...I guess I could install it on that partition, and if it IS malware or worse, no big loss for me to wipe that Windows install and start fresh...

Edited April 9, 2015 by Stone Blue

[sarbian]

Minimem looks a lot like snake oil. Nothing in their description actually explain how it works. If fixing leak was as easy as launching an app it would be in Windows...

[bernh]

I don't know what Minimem does under the hood. If the info about the authors is correct, it was developed by some independent programmers, that do custom software and websites for people. Any tools they develop outside of a contract they put on their website for free download.

There are a lot of testing, diagnostic, and fix it apps out there that have been developed because the functionality that is required is not in windows.

Seriously guys, I am not employed by them, and gain nothing from mentioning them. If you don't want to try it, don't. I am 43 years old, a trained electronics and computer technician. I have had a computer since I was 9 years old. In all that time, I have only be been hit by 3 viruses. I always have virus scanners and malware scanners going. All I can say is that Minimem worked for me and allowed me to play my heavily modded game when included with the other steps I mentioned.

[Van Disaster]

Does DDS loader make ActiveTextureManagement obsolete? Is there a point in running the two side by side?

There's no point using both provided you preconvert everything. ATM will just take memory up for no benefit.

Try -force-opengl or -force-d3d11 ; both have their own issues ( opengl has no shadows, dx11 needs aero effects turned to min/has occasional shadow glitches/turns minmus into a black hole ) but they will offload textures to video RAM which saves about a GB.

Install Texture Replacer, it'll unload textures again after the game has loaded them at startup.

Edited April 9, 2015 by Van Disaster

[sarbian]

I had a look at what Minimem actually does. It calls EmptyWorkingSet at regular intervals on all the app that are selected. This force some of the process memory to go to swap. It give the illusion that you have more free mem but actually you just have more used swap and those page will get back to the memory since there was a good reason Windows Kernel did not put them there in the first place. As I said : Snake Oil.

Edited April 9, 2015 by sarbian

[funk]

I also have shutdowns of KSP with Minimem after some time. It doesn't really effect playtime as sarbian already mentioned. So I will migrate to Linux for 1.0 if I get my raid 0 working...

[ColKlonk]

One could avoid all this garbage if KSP managed it's objects properly

[dewin]

heh, I guessed correctly at what Minimem does...

And as a result, it could be beneficial for KSP's performance if your overall system is low on RAM in general -- but if you're running 8GB+, it's not going to do much for you (nor will it reduce KSP's memory usage.), and it's not going to resolve out-of-memory crashes regardless.

As far as a process's address space goes, each 32-bit process has a virtual 4GB of memory. Paging to disk isn't going to reduce that at all, it just means that when it requests that chunk of memory it now has to be retrieved from disk.

[Fel]

I had a look at what Minimem actually does. It calls EmptyWorkingSet at regular intervals on all the app that are selected. This force some of the process memory to go to swap. It give the illusion that you have more free mem but actually you just have more used swap and those page will get back to the memory since there was a good reason Windows Kernel did not put them there in the first place. As I said : Snake Oil.

Even if it didn't do that, it would be impossible for it to be anything but snake oil. RAM Optimizers cannot safely dynamically deallocate memory currently in use by a program. (Well, correction: An optimizer directly made for KSP could do it... but it wouldn't be pretty and would probably cause more crashes unless it froze the game [Re: Garbage Collected Memory]. You know, it wouldn't be pretty but... if (and only if) all of the loaded textures were on the GPU's memory... correcting the memory to properly reflect this might be possible [note: this is an issue with C# and Unity], still... getting a stable pointer in garbage collected memory.)

It is interesting to know that what they tend to do is push everything back into swap. In my youth I liked them, then I grew out of them (I never noticed a performance increase), and later I learned that the kernel would optimize memory for the currently running application anyways.

[Van Disaster]

@Fel: "[LOG 02:40:24.702] [TR.Loader] Texture unloading freed approximately 1119.8 MiB = 1174.2 MB of system RAM" after a big list of "[LOG 02:40:24.699] [TR.Loader] Unloaded VenStockRevamp/Squad/Parts/Engine/Mk55LUM". I can watch it doing it using GCMonitor.

That's from Texture Replacer, runs after asset loading & before MM starts compiling. I run DX11, it might not do anything for DX9 I guess, never bothered checking. I've no idea if it dynamically unloads textures if you swap scene later on either.

Edited April 10, 2015 by Van Disaster

[Fel]

@Fel: "[LOG 02:40:24.702] [TR.Loader] Texture unloading freed approximately 1119.8 MiB = 1174.2 MB of system RAM" after a big list of "[LOG 02:40:24.699] [TR.Loader] Unloaded VenStockRevamp/Squad/Parts/Engine/Mk55LUM". I can watch it doing it using GCMonitor.

That's from Texture Replacer, runs after asset loading & before MM starts compiling. I run DX11, it might not do anything for DX9 I guess, never bothered checking. I've no idea if it dynamically unloads textures if you swap scene later on either.

That isn't exactly what I meant. A Ram Optimizer should be an external program, not a plugin; just a quick look and it seems the plugin works by converting mod's textures to DDS with enhancements (i.e. compression).

Unity's problem's are much harder to address because you don't have any real control over the memory from C#. If something gets "stuck" because C# isn't convinced that ALL namespaces that reference it no longer are referencing it, the only way you can free it would be to either restart the game, or have an external program hack and do some really stupid things and hope it all works out.

The case I was referencing was how OpenGL's implementation doesn't have a shadow cache of textures. In theory, you could manually "free" that space by writing it as unassigned. In practice, it is a very bad idea.

[Van Disaster]

Converting something to DDS is nothing like freeing 1GB or so of memory in one go. TR only occasionally converts anything, it's obvious in the log when it does - given the game converts everything to DDS which isn't already converted anyway that is not in itself going to save anything.

TR unloads 72MB of assets from an otherwise stock game irrespective of version of DX - unfortunately I don't have a more modded game I can start in dx9 mode at the moment so I'm not sure if it's dumping out pre-offloaded textures or just unloading regardless. I've not really any desire to deconstruct TR myself either.

I'm well aware of reference count GC issues :S

[sarbian]

Unity's problem's are much harder to address because you don't have any real control over the memory from C#. If something gets "stuck" because C# isn't convinced that ALL namespaces that reference it no longer are referencing it, the only way you can free it would be to either restart the game, or have an external program hack and do some really stupid things and hope it all works out.

I would like to point out that 95% of Unity is not C#. The only things that run on mono in the game scripting stuff. All the engine is pure native code. You can create native object from the C# and if you don't free them manually they never go. It s not a garbage collector problem.

[NathanKell]

If you're using OpenGL mode that, and converting to DDS, is about all you need--since that means you won't leak loading textures, and textures will properly be removed from system RAM when uploaded, you needn't scale them down to save RAM.

[Jovzin]

Well I must say my 2 cents too.

I really hate this memory stacking and not flushing the textures. Bcs of this I was able to do only 6- 7 times of scene switching ( from KSP to ship and back from ship to KSP or to another ship) and after those 7 times of scene switching I got crash bcs our beloved KSP is not handling the textures well and instead of flushing them it stacks them.

This was with forced DX11 and aggresive memory management from rbay89 and converted textures to DDS and used DDSloader.

But after some time of googling on the internet I found this little program called cleanmem ( mainly used in Battlefield games ) http://www.pcwintech.com/cleanmem

Now I am using DX11 with ATM and wiht DDSloader and every 30 minutes cleanmem will do its job and I must say no more crashes for me. I can do a lot of those scene switching and no crash.

But I really hope that Squad will fix this in 1.0 cause it is really a game breaking thing.

Edited April 14, 2015 by Jovzin