Possible workaround for memory issues with 32 Bit KSP D3D9 Mode

[theersink]

I need some voluteers to verify this is not just a fluke. As we all know what may work for one may not work for all. Anyhow, on to my revelation...

So I have been trying to optimize and get D3D9 working at a stable level as I prefer to have shadows and a steady simulation rate. After hours upon hours of tinkering and swapping and trying different approaches something dawned on me....

I play a modded version of the sim Falcon4, now this came out in 1998 and has been upgraded and improved into one of the best flight sims around for those who know about it. Anyhow I digress. Back in the early days of the heavy modding we were having the exact same problems with memory usage and getting out of memory errors with all the new content. Now those gurus figured out that you could "cheat" by simply adjusting your virtual memory settings in windows to use a set amount rather than have windows allocate it on its own. It has something to do with how the program sees the swapfile when windows isn't managing the size, I'm not sure the mechanics of it but it did allow us to continue modding the sim and was a major breakthrough.

Now tonight I was almost ready to give up and just deal without shadows and a laggy OpenGL mode when I figured I would try this simple trick. Perhaps windows is no better now at managing a swapfile now than it was in the windows xp days? (Windows has always been horrible at this)

Well I set my virtual memory amount on my non system disk to twice my system memory. I figured I would re-cache all the textures so I did "The Dance" as we used to call it and noticed that while ATM was doing its thing my system memory did not have the extreme spikes and crashes while compressing the textures like it did before. In addition, when it was done with the texture it would not increase memory usage by the amounts it did before when loading large textures.

Now you ask, what about in game? Note: After compressing you must exit and restart.

Well after loading these mods (ATM X86 Aggressive):

KSP: 0.25 (Win32) - Unity: 4.5.2f1 - OS: Windows 8.1 (6.3.9600) 64bit

USI Tools - 0.2.4

Zero-Point Inline Fairings - 0.8.1

B9 Aerospace - 5.2.6

Chatterer - 0.7.1.86

CIT - CIT_Util - 1.1

CIT - KERT - 1.2.1

Community Resource Pack - 0.2.3

Ferram Aerospace Research - 0.14.4

Fine Print - 0.59

Firespitter - 6.3.4

RasterPropMonitor - 0.18.3

Kerbal Engineer Redux 1.0 - 1.0.12

Kerbal Joint Reinforcement - 2.4.4

KineTechAnimation - 1.1.1

KSP-AVC Plugin - 1.1.5

Final Frontier - 0.5.9.177

Nano Gauges - 0.5.17.163

PlanetShine - 0.2.2

Procedural Parts - 0.9.20

Protractor Continued - 2.4.14

RCS Build Aid - 0.5.2

RealChute - 1.2.6

RealSolarSystem - 8.0

RemoteTech - 1.5.1

ResGen - 0.28.2

SCANsat - 1.0.8

StageRecovery - 1.5.1

TAC Fuel Balancer - 2.4.1.5

TextureReplacer - 2.0.2

TAC Life Support - 0.10.1.13

Trajectories - 1.0

Universal Storage (Core) - 0.9.3.25

Universal Storage (TAC) - 0.9.2.7

In addition I'm running Proot's texture pack(not resized), KW Rocketry and FASA launch clamps in addition to a few others.

I did some testing. Before i could watch my memory usage slowly creep up when changing screens and especially when in the TS viewing other planets and crafts. I must have done 50 transitions all around the board and not once did I get a crash. This includes reverting to launch and VAB. Memory usage hovered around 3.0-3.1 GB but usually went down after transition was loaded.

In conclusion I am now able to use proots pack with the high resolution planets, EVE and several large part mods in a fairly stable environment.

Now I cannot and will not guarantee this will work for you and

I do not reccommend changing your windows virtual memory settings unless you know what you are doing. If you do it wrong you can seriously hose your windows environment.

Now here are some screens of the process and conclusion.

​If anyone tries this let me know here how it works for you.

Edited December 13, 2014 by theersink

[MartGonzo]

I can't confirm this works specifically for KSP because I've always used a fixed size swap file (6Gig, same as my RAM)

never done any testing but it's quite a common tweak on performance optimization threads so I'm guessing it is faster.

Sometimes you see threads that recommend removing the swap file entirely but I don't recommend that.

[theersink]

I can't confirm this works specifically for KSP because I've always used a fixed size swap file (6Gig, same as my RAM)

never done any testing but it's quite a common tweak on performance optimization threads so I'm guessing it is faster.

Sometimes you see threads that recommend removing the swap file entirely but I don't recommend that.

Yes I know it's a fairly common practice in hardcore gaming circles. but it seems like doing this allows the system to swallow up the KSP memory leaks while loading and in game. It's possible it could be why some people have a better experience while others have a crash fest. For me this allowed the use of Proots textures, B9 Aero and provided much more stability in game. In addition it reduced my memory footprint at the menu by 350 Megs.

[MartGonzo]

Yes I know it's a fairly common practice in hardcore gaming circles. but it seems like doing this allows the system to swallow up the KSP memory leaks while loading and in game. It's possible it could be why some people have a better experience while others have a crash fest. For me this allowed the use of Proots textures, B9 Aero and provided much more stability in game. In addition it reduced my memory footprint at the menu by 350 Megs.

Any idea why it works?

It seems odd that a fixed size swap file would have any impact on a games memory usage.

[theersink]

Any idea why it works?

It seems odd that a fixed size swap file would have any impact on a games memory usage.

Well if I understand correctly, normally windows will adjust the size of the swapfile based on different kinds of calls from programs for memory. It is not very good at it. When you set it to a fixed size and take windows out of the equation some programs can "cheat the system" and use this area for background tasks (such as swapping textures to and from VRAM) when normally windows would not allow it. That is a very basic explanation and I could be completely wrong but as I said that is how I understand it. Now also if I remember correctly the full benefit can only be gained within gaming environment by setting it at 2x your memory. (Again something about windows allocation cheating), less than that and for some reason the program won't utilize it.

Edited December 13, 2014 by theersink

[MartGonzo]

Well if I understand correctly, normally windows will adjust the size of the swapfile based on different kinds of calls from programs for memory. It is not very good at it. When you set it to a fixed size and take windows out of the equation some programs can "cheat the system" and use this area for background tasks (such as swapping textures to and from VRAM) when normally windows would not allow it. That is a very basic explanation and I could be completely wrong but as I said that is how I understand it. Now also if I remember correctly the full benefit can only be gained within gaming environment by setting it at 2x your memory. (Again something about windows allocation cheating), less than that and for some reason the program won't utilize it.

Thanks, that explanation does seem plausible.

[theersink]

Follow up:

After playing the game for around an hour I was still hovering around 3.1 GB Mem usage. I recovered a flight and built a new rocket, after I hit the launch button the transition screen seemed to pause and I thought ah crap crash time but no, after a short pause I was presented with the launchpad and this little gem:

That pause successfully unloaded almost 700 MB from memory and did not crash the game nor were there any other adverse effects that I could see. This is the longest I have played in D3D9 32 Bit without crashing and I have 4 more mods loaded than I was using before.

[theersink]

End of session followup:

After about 2 Hours of non stop glitch free playing I need to sleep. Final report of the night:

Ended session using right around 2.6 GB:

Looked through the log and curiously found this entry, now I have not been playing long enough to know if this is common or not but it is interesting:

(Filename: C:/BuildAgent/work/d63dfc6385190b60/artifacts/StandalonePlayerGenerated/UnityEngineDebug.cpp Line: 49)

FF: storing window positions

(Filename: C:/BuildAgent/work/d63dfc6385190b60/artifacts/StandalonePlayerGenerated/UnityEngineDebug.cpp Line: 49)

FF: writing 5 window positions

(Filename: C:/BuildAgent/work/d63dfc6385190b60/artifacts/StandalonePlayerGenerated/UnityEngineDebug.cpp Line: 49)

[smokeScreen PersistentEmitterManager] : OnDestroy

(Filename: C:/BuildAgent/work/d63dfc6385190b60/artifacts/StandalonePlayerGenerated/UnityEngineDebug.cpp Line: 49)

UnloadTime: 25.689100 ms

Unloading 7 Unused Serialized files (Serialized files now loaded: 0 / Dirty serialized files: 0)

Unloading 417 unused Assets to reduce memory usage. Loaded Objects now: 185159.

Total: 133.203232 ms (FindLiveObjects: 7.802334 ms CreateObjectMapping: 2.418418 ms MarkObjects: 119.388725 ms DeleteObjects: 3.216867 ms)

ContractsWindow.ContractsWindow.CustomAddonLoader: PSYSTEM was loaded; instantiating addons...

(Filename: C:/BuildAgent/work/d63dfc6385190b60/artifacts/StandalonePlayerGenerated/UnityEngineDebug.cpp Line: 49)

ContractsWindow.ContractsWindow.CustomAddonLoader finished; created 0 addons

(Filename: C:/BuildAgent/work/d63dfc6385190b60/artifacts/StandalonePlayerGenerated/UnityEngineDebug.cpp Line: 49)

AddonLoader: Instantiating addon 'ActiveTextureManagement' from assembly 'ActiveTextureManagement'

Full log for anyone interested:

https://onedrive.live.com/redir?resid=5C2CF5B33E38865E%214008

[Van Disaster]

Sadly did not help me ( although it's a coin flip whether I can even start in DX9 ) although it did make me look at my VM settings and find that yes, they had somehow gone back to system managed. Will see if it helps DX11 which is still crashing sometimes.

[djnattyd]

My page file is currently set to 14gb (helps in FSX) and there is no difference in KSP stability. As soon as I hit 2.8gb RAM, KSP crashes.

Also, the difference in stability between DX9 and openGL is negligible for me at least.

[theersink]

My page file is currently set to 14gb (helps in FSX) and there is no difference in KSP stability. As soon as I hit 2.8gb RAM, KSP crashes.

Also, the difference in stability between DX9 and openGL is negligible for me at least.

Yes, I doubt it will help any if it is already set. Mine was not after an update and it seems to be working well for me. Of course it could just be my particular setup too.

On another note the new AMD/ATI Omega drivers seem to work really well with KSP. If you have an AMD/ATI card I would strongly reccommend d/l them.

[Red Iron Crown]

There's a lot of misinformation out there about virtual memory, the pagefile and how they work.

Windows NT based OSs (NT, 2000, XP, Vista, 7, 8, 10) use a memory system based on the old VMS operating system. Only the operating system has direct access to the physical RAM addressing scheme, programs never see it. Instead, the memory manager assigns each process an entire memory address range (32-bit or 64-bit, as determined by the program), this space is called virtual memory. The Windows memory manager does the job of translating those virtual memory addresses into physical memory addresses. As different programs make different demands for physical memory, the memory manager creates "pages" of each program's data, each of which can be removed from physical memory and covered by what is called a "backing store". A backing store can take two forms: For items that were changed or created after loading from disk, the page is written to disk in a special file called the pagefile, while items that haven't changed since loading from disk are backed by the original file. This is important, because it means that disabling the pagefile doesn't prevent paging, it just forces the OS to only page items that are backed by original files. Windows' memory manager is pretty sophisticated, and its algorithms for determining what to page and in what order are very good but are compromised if the pagefile is unavailable. So never disable the pagefile. Confusingly, Windows overloads the term "virtual memory" by calling the pagefile by the same term in the control panel, which has resulted in a lot of confusion.

Further, individual programs have no idea of what is paged and what is not, all they know about is their virtual memory space which as far as they are concerned is all RAM. The OS handles all the assignments to physical RAM and backing stores behind the scenes at a lower level than the programs can "see". The memory limit for an individual program is determined by the size of its virtual memory address space, it won't complain about running out of memory until that is exhausted, whether portions have been sent to a backing store or not. Changes to the pagefile size have no effect on the amount of virtual memory available to a program; the size of the pagefile only affects the memory manager's ability to do its job properly.

So, while I appreciate the OP's determination to find a workaround for 32-bit's memory issues, I can't see how this tweak can possibly help with them.

[theersink]

Thank you for the explanation.

All I can say is it has and does improve performance in some applications. This may not be "by the whitesheets" operation but it does. It is very common practice in hardcore PC gaming circles and has been since XP.

[enneract]

Thank you for the explanation.

All I can say is it has and does improve performance in some applications. This may not be "by the whitesheets" operation but it does. It is very common practice in hardcore PC gaming circles and has been since XP.

This is just cargo-cult behavior.

[magnemoe]

This is just cargo-cult behavior.

it might have some performase benefits, could imagine an fixed swap file would work better on an drive who is pretty full as the swapfile would not be fragmented.

However i'm not sure how important the swapfile is for running a 32 bit program on an 64 bit os with +8GB ram.

In short its no reason why any part of the game should be swapped to disk, as you have more free memory than the game can use.

One benefit of lots of memory is also that the used files will be in cache so in games with lots of voices and textures it don't have to wait on the disk.

And yes 64 bit will have another benefit here as it can keep more of the game in memory, it don't have to throw out stuff, them load and convert them again who takes time.