[1.9.x-1.12.x] HeapPadder - Pads the heap to minimize the garbage collection

[etoanrish]

A bit confused about this, even after reading all the comments.

Is this mod useful for an Apple MAC, OS/X system with 16GB? (NOT running Windows)

I frequently have Safari and Mail open and running while using KSP planet packs. Smaller planet packs run for quite some time before the stuttering starts to happen.  As expected, running larger planet packs, the stuttering starts earlier.  One of my planet packs just keeps grabbing more and more memory locking the computer for two or more minutes.

Does this mod grab a bunch more memory for KSP at the start?  Does it keep KSP from grabbing more and more system memory?  This would force Mail and Safari to swap to disk to avoid the stutter, right?  If so, what would be the appropriate setting 2GB or 4GB.?

Thanks in advance.

[darthgently]

At the risk of waking a dead thread (got the warning), if one has 64G of ram, would it make sense to set "total" to 16384 as the apparent pattern seems to follow?  Or would it make more sense to cap it below that somewhere?  I imagine there is a time/space trade off within the heap algo itself

[linuxgurugamer]

1 hour ago, darthgently said:

At the risk of waking a dead thread (got the warning), if one has 64G of ram, would it make sense to set "total" to 16384 as the apparent pattern seems to follow?  Or would it make more sense to cap it below that somewhere?  I imagine there is a time/space trade off within the heap algo itself

It adjusts dynamically.  Not really sure if it's needed if you have that much memory

[chris-kerbal]

Is this still relevant for current versions of KSP? I am not blessed with GBs of RAM and normally run around 9 out of 12 GBs.

[darthgently]

18 hours ago, linuxgurugamer said:

It adjusts dynamically.  Not really sure if it's needed if you have that much memory

Oh, I needed it.  I was getting horrible 5 to 10 second freezes every 30 secs or so that coincided with GC (going by Memgraph correlation).  I had installed HeapPadder but hadn't increased the "total" above 4k.  After adjusting to 16k the freezes went completely away.  I should add that the freezes pre-existed the HeapPadder install but didn't change noticeably after install with the "total" at 4k (or whatever the default is, I recall it was 4k)

[Gordon Dry]

FYI the description in the OP still uses the phrase "MemGraph" instead of "HeapPadder" at some places. one place.

Edited November 19, 2021 by Gordon Dry

[linuxgurugamer]

48 minutes ago, Gordon Dry said:

FYI the description in the OP still uses the phrase "MemGraph" instead of "HeapPadder" at some places. one place.

Fixed, thanks

[linuxgurugamer]

New release, 0.0.2.2

• Thanks to github user @lampeh for this:
  • Fixes negative values in screen message

[linuxgurugamer]

New release, 0.0.2.3

• Thanks to github user @Hauke Lampe for this
  • Fixed default config files not found
     

[linuxgurugamer]

New release, 0.0.2.4

• Fixed bad paths when used with new  launcher

[610yesnolovely]

2 hours ago, linuxgurugamer said:

New release, 0.0.2.4

• Fixed bad paths when used with new  launcher

... and you fixed it in 20 other mods. Respect and also betcha glad you automated your build/release.

[linuxgurugamer]

6 hours ago, 610yesnolovely said:

... and you fixed it in 20 other mods. Respect and also betcha glad you automated your build/release.

And a bunch more to go.  Really wish T2 hadn't broken the game this way so late in it's life, just a waste of time for me and other modders

[ThaGreenGoloco]

Does this still work with latest KSP version? CKAN thinks otherwise and I want to know if it's safe to install. It'd be saving my Kerbal experience.

[linuxgurugamer]

2 hours ago, ThaGreenGoloco said:

Does this still work with latest KSP version? CKAN thinks otherwise and I want to know if it's safe to install. It'd be saving my Kerbal experience.

yes

[GDorn]

I'm running linux on a system with 32GB of ram.  The default 32g file didn't help much at all, so I set the heappadder defaults to a 1 in every row and a total of 16384.  Not sure if that's the right way to go about it, but it only helps a little; before doing this, I'd end up with a 1s pause every 5s.  With this padding I now have a 1s pause every 10-12s.  So it helps a touch but it's still bad enough that I find I'm not interested in starting the game...

Should I keep increasing the padding?  Is there a more optimal way to go about this?  Should I be running KSP in Proton, or boot to (*shudder*) Windows?

[linuxgurugamer]

On 3/21/2023 at 12:56 PM, GDorn said:

I'm running linux on a system with 32GB of ram.  The default 32g file didn't help much at all, so I set the heappadder defaults to a 1 in every row and a total of 16384.  Not sure if that's the right way to go about it, but it only helps a little; before doing this, I'd end up with a 1s pause every 5s.  With this padding I now have a 1s pause every 10-12s.  So it helps a touch but it's still bad enough that I find I'm not interested in starting the game...

Should I keep increasing the padding?  Is there a more optimal way to go about this?  Should I be running KSP in Proton, or boot to (*shudder*) Windows?

How is the memory usage?  You can increase it, but seems that you got most of the improvement it can do.  Beyond that, Windows 

[kurgut]

Hi,

Is this still useful in 1.12 ?

I was being told that garbage collection had been improved in KSP, also that heap padder did apprently increased RAM usage, by allocating more RAM to KSP...

I'm running a laptop, with rtx3060 6gb vram, 32gb RAM, i7-12700H   2.70 GHz

So I'm wondering if I really need it... Like, the main symptom giving incentive to use this would be having freezes right ?
Generally speaking, would it increase performances ?

Thanks for your help  

peace

[KincaidFrankMF]

At risk of being annoying... Any chance someone could be kind enough to give the "idiotspeak" version of what this mod does, for the benefit of the technologically clueless among us?

I just don't know what garbage collection is in relation to software, and I suspect I'm not the only one. As a result I can't tell if it's something I need. Also whether the mod is still useful with the latest version of the game?

 

I gamer. Want game run fast. Need mod?

Edited May 3 by KincaidFrankMF

[Fizzlebop Smith]

3 hours ago, KincaidFrankMF said:

At risk of being annoying... Any chance someone could be kind enough to give the "idiotspeak" version of what this mod does, for the benefit of the technologically clueless among us?

I just don't know what garbage collection is in relation to software, and I suspect I'm not the only one. As a result I can't tell if it's something I need. Also whether the mod is still useful with the latest version of the game?

 

I gamer. Want game run fast. Need mod?

It's covered in the very first post 

 

On 7/19/2019 at 3:14 PM, linuxgurugamer said:

HeapPadder

This mod is designed to force Mono to keep significantly more free space in the heap, which can significantly reduce the frequency at which the heap fills up and the Mono garbage collection causes a stutter,

 Installation

1. Copy the HeapPadder folder from the zip file into the GameData folder of your KSP installation.

Regarding the padheap.cfg file:

1. If the default settings aren't working well for you, try the following steps:
2. Increase the padheap size in your install: \GameData\HeapPadder\PluginData\padheap.cfg, change the "total" number to 4096 or even 6144. See the next section for an example
3. For example, if you have 16 GB:
   1. Look at your task manager to find out how much free ram you have when the game is running. 
   2. Do this calculation 16 Gb - ~(5 to 6 GB from KSP) - (1 to 2 GB Windows) = 8 GB free
   3. This is the amount of RAM you can use for your padheap total size (in MegaByte 4GB=4096MB).  

 Always have some Ram to spare of course. 

Configuration

• The mod uses a config file called:  padheap.cfg.  This file is not distributed with the mod, it is created the first time running the mod based on the amount of system memory found.
• The mod has several default config files for different sizes of system memory.  These files are used if no padheap.cfg file is found, usually on the first time running.
• If the config file padheap.cfg is found, then it will be used.  The mod will NOT overwrite the file, this is done so that any future updates to the mod won't wipe out any local config changes
• There is nothing preventing you from use one of the suggested files, even if your system memory is different.  To use any of them, simply copy the desired file to:  padheap.cfg
• Current files supplied allocate the following memory to the heap:
  • SuggestedFor_32g.cfg    8 gig of memory, used by default if system memory is greater than 20 gig of memory
  • SuggesetdFor_16g.cfg    4 gig of memory, used by default if system memory is between 8 and 20 gig of memory
  • SuggestedFor_4g.cfg        1/2 gig of memory, used by default if system memory is less than or equal to 4 gig
  • default_padheap.cfg        1 gig of memory, used by default if system memory is greater than 4 and less than or equal to 8 gig of memory
• In the unlikely event that no files are found, a default configuration will be written out.

While this can be used with Memgraph, you risk doubling the memory usage if you allocation memory with Memgraph as well as HeapPadder.  To address this I've released an update to MemGraph, which disables the heap padding functionality of Memgraph if HeapPadder is installed

Pics

HeapPadder has no UI, so nothing to show

Availability

• Download:
  • https://github.com/linuxgurugamer/HeapPadder/releases
  • https://spacedock.info/mod/2190/HeapPadder
• Source Code: https://github.com/linuxgurugamer/HeapPadder
• License: MIT license (see https://github.com/linuxgurugamer/heapPadder/blob/master/Graph.cs)

 HeapPadder is also available via CKAN

 

 

If you have stutter, crashes or general issues due to memory allocation, you can use this mod to increase the RAM allocated to mono processes.

This may only make marginal difference in better machine but potentially significant gains in older systems

[KincaidFrankMF]

1 minute ago, Fizzlebop Smith said:

It's covered in the very first post

Um... I did say I need the idiotspeak version! What does Mono even do? What the heck is a heap and why do I need free space in it? Mono garbage collection is what we used to have before they brought in recycling, right?

We're not all software developers...

I guess what I'm really asking is, is this the kind of mod that will help the game run better for all players, or is it mainly for if you're, e.g., running a bunch of memory-intensive planet packs? Without understanding the technical stuff, I can't tell. Also, someone mentioned that the core game now handles garbage collection, whatever that is, better now, so I was wondering if the mod still makes a big difference?

[Jacke]

4 hours ago, KincaidFrankMF said:

At risk of being annoying... Any chance someone could be kind enough to give the "idiotspeak" version of what this mod does, for the benefit of the technologically clueless among us?

I just don't know what garbage collection is in relation to software, and I suspect I'm not the only one. As a result I can't tell if it's something I need. Also whether the mod is still useful with the latest version of the game?

 

I gamer. Want game run fast. Need mod?

A lot of this is in this Wikipedia article.
https://en.wikipedia.org/wiki/Heap_(data_structure)

But I skimmed it and it jumps in deep and will be hard for you to understand.  So I'll explain as best I can.

A running Program needs computer resources to operate.  Sometimes those resources need to be allocated for the full run of the Program, but often they are only need temporarily while doing a particular task.  So they are requested and then at some later time returned.

Using the clock cycles when processing is handled by various ways to start and stop code.  They matter, but aren't the issue here.

What's the most usual resource needed and then returned is memory.  Most programs in most parts see memory as a near-monolithic whole in chunks.  The underlying operating system can do other things, or the program can do explicit access of slower storage (e.g. read or write to files, which for slower file access has had whole books written on its optimization), but it's the general "memory" we're talking about here.

The most common way to control temporary allocation and return of memory is the Stack.  Call a function or subroutine, it asks for memory on the Stack to do its work, then when finished, it returns or frees what it used on the Stack.  It's a Last-Allocated--First-Freed memory reserve.

But sometimes extra memory needs can't be handled by a Stack.  When various chunks of memory are needed independent of one another and when they are freed is not in the inverse order of when they were allocated.  The most common way to then handle it is the Heap.

The Heap gets requests for memory and then allocates a Chunk of memory from its pool of available Chunks and passes back to the requesting code.  Internally, the Heap has pointers to connect all these Chunks of memory.  When a particular Chunk of memory is freed, it's often best to just Flag that Chunk as freed and not clean up the fiddly bits to make that Chunk available, perhaps unify unallocated Chunks of memory together.  The big issue is that most Heaps offer different sized Chunks to ideally minimize the over size of memory needed.  But making Chunks different sizes complicates the administration of the Heap.

At some point with this model of Heap, all those Chunks freed but not ready to be reallocated need to be sorted out, as the pool of available memory for use in new Chunks is running low.  What happens now is Garbage Collection, where the memory Chunks of the Heap are sorted out--but in a way that won't break the running of the Program.  But the impact of running Garbage Collection can impact the running of the rest of the Program, in that it takes up clock cycles and maybe other resources of the computer.  Ideally, Garbage Collection can be run on the Heap in parallel with its regular operation and the rest of the Program.  But not always.  But Garbage Collection should not make a Program noticeable Stutter.  With a Program like KSP where the program is displaying video and audio, having Garbage Collection not cause Stuttering may be very challenging.

One way to cut down on this Stuttering is to make the Heap bigger.  That way, the times when Garbage Collection is needed may be farther apart.  And may not be needed at all except at special times where it won't cause visible Stutter.  Also, with a larger Heap, it can be operated in different ways to make it need more memory overall (thus the bigger Heap) but that simplifies its operation and Garbage Collection.

I hope this explanation helped!

Edited May 3 by Jacke

[KincaidFrankMF]

1 hour ago, Jacke said:

I hope this explanation helped!

Legend. Thank you so much! Really kind of you to take the time to go through all that.

Does the bigger Heap not lead to a longer/more visible stutter when it does eventually get sorted out?

[Jacke]

1 hour ago, KincaidFrankMF said:

Does the bigger Heap not lead to a longer/more visible stutter when it does eventually get sorted out?

It's very complex and depends on a lot.  It's easier to just use this mod to expand the Heap and experiment and see what happens.

In a deeper context, when the Heap and its support code could be changed, with more memory other Heap schemes can be used that have less overhead demand to do the Garbage Collection.