linuxgurugamer

[1.5.*, 1.6.1, 1.7] MemGraph Updated with Stutter Reduction

Recommended Posts

Originally written by @padishar, original thread is here:  https://forum.kerbalspaceprogram.com/index.php?/topic/139128-12x13-memgraph-1103-with-stutter-reduction/

Note New Dependencies

This started as a simple plugin that displays a graph of the Mono heap allocation rate and garbage collection, mainly intended as a troubleshooting and development aid rather than for general use (hence why I originally released it in the Add-on Development sub-forum rather than this one).  However, I have since devised a way to force Mono to keep significantly more free space in the heap, which can significantly reduce the frequency at which the heap fills up and the Mono garbage collection causes a stutter, so I have added it to this mod and released it more publicly.  I will be continuing to work on this mechanism to improve it further.

The code has been highly optimised to create as little garbage as possible (given the constraints of the old OnGUI based API) so that it interferes with what it's measuring as little as possible.  I do have plans to add various "usability" type enhancements but only things that can be implemented cleanly and are directly useful for the mods purpose.

To install, simply download the zip from the link below and copy the MemGraph folder from the zip file into the GameData folder of your KSP installation.

  • Mod-KeypadMultiply toggles the display of the window.
  • Mod-KeypadPlus increases the vertical scale of the graph.
  • Mod-KeypadMinus decreases the vertical scale of the graph.
  • Mod-KeypadDivide runs a bit of test code controlled by MemGraph\PluginData\test.cfg
  • Mod-End pads the Mono heap with a configurable amount of headroom to reduce frequency of garbage collections.

Every second the plugin totals up all the memory allocated on the heap and whether any garbage collections have run.  It also displays the current total heap allocation, the maximum heap allocation just before the last garbage collection, the minimum just after the last collection, the number of "render" and "physics" updates and the time between the previous two collections.

The graph is 600 pixels wide and shows the last 10 minutes of the memory allocation in green.  If a garbage collection happens during the interval, that column of the graph will have a red background.

The test code basically allocates a number of blocks of the specified size and displays which allocations actually cause the allocated heap to change.  This allows us to deduce various characteristics of the memory allocator and garbage collection mechanisms (and was a crucial part of developing the heap padding mechanism).

Mod-End activates the heap padding.  The amount of padding is controlled by the MemGraph\PluginData\padheap.cfg.  The format of the file is very simple.  Each line controls the amount of padding allocated for blocks of each size range.  The first value is the size of each block allocated and the second is the number of blocks.  The first values are only present for illustration, they don't actually control the size of the blocks, these are hardwired to the sizes in the default configuration.  The total value at the end is the total number of megabytes of padding allocated (note, the memory usage of KSP will increase by more than this due to the free space headroom maintained by Unity).

Logging

The mod now has the ability to write the data to a set of log files. The files are written in the root of the game directory.  Logging can be enabled at startup by editing the settings.cfg file. Set the following to true to turn on logging, false to turn it off:

enableLogging = true

Logging Controls

  • Mod-Home writes a marker to the log file (if logging is enabled, see below)
  • Mod-PageUp toggles logging

The first set of files will only have a single value in them at any time. This will be the last value which was plotted. The name of the file corresponds to the data item as shown on screen:

  • dataLog.Cur.txt
  • dataLog.HeapMin.txt
  • dataLog.Interval.txt
  • dataLog.Last.txt
  • dataLog.Max.txt
  • dataLog.P.txt
  • dataLog.R.txt

This last file is a sequential log, and is overwritten every time the game starts. The log is formatted as a CSV file, and the first line contains the column headers

  • dataLog.Log.txt

I recommend that you run the game normally and load up a situation that has noticeable stutter.  Display the graph, setting the scale so the regular allocation rate fits nicely and the garbage collection red lines can be seen.  Let it run for several runs of the garbage collector and then hit Mod-End.  After a short pause, the game should continue with a considerably larger gap between the collections.  After another few collections hit Mod-End again and it may improve further.  I would appreciate feedback about how well this works, e.g. screenshots of the graph taken during this process and after another few collections would be very helpful along with details about your setup (and preferably, an output_log.txt/player.log file).  Many thanks to @Tig for the testing and very well presented data he posted over in the Grumpy Collector thread.

One other thing I should add, though it should be obvious with only a little thought, is that the heap padding mechanism is only intended for 64 bit versions of the game.  Trying to allocate 900 MB of extra heap space on the 32 bit version is unlikely to be successful and, if it is, then it will probably cause the game to crash before long due to running out of address space. It is also unlikely to work effectively if your machine has only 4GB of RAM as the total usage of KSP is likely to grow close to 4GB even without loading a save, resulting in virtual memory paging which will seriously hurt performance.

Regarding the padheap.cfg file:

If the default settings aren't working well for you, try the following steps:

Increase the padheap size in your install: \GameData\MemGraph\PluginData\MemGraph\padheap.cfg, change the "total" number to 4096 or even 6144. See the next section for an example

For example, if you have 16 GB:

  1. Look at your task manager to find out how much free ram you have when the game is running. 
  2. Do this calculation 16 Gb - ~5/6 GB from KSP - 1/2 GB Windows = 8 GB free
  3. This is the amount of RAM you can use for your padheap total size (in MegaByte 4GB=4096MB).  
  4. Always have some Ram to spare of course. 

Padheap is the volume memgraph is using to save the junk and than deletes it when its full.

Obligatory picture of mod in use:

tUUtX2i.png

Availability

Older versions

Previous releases on GitHub (versions 1.0.0.x are for KSP 1.1.x, versions 1.1.0.x are for KSP 1.2).

There's no obligation and it won't entitle you to any special treatment but, if you feel like it, then you can make a donation using the link below.

Donate Here

 

Edited by linuxgurugamer

Share this post


Link to post
Share on other sites

hoo men... did you sleep some time?

your work is pure gold.

this mod works perfect for me...

Tested on i7-4700HQ - RAM 8GB - GTX 680m
Ubuntu 18.04 Nvidia Drivers 390.48
Runing perfect (almost zero stuttering) with KSP.x86_64 (64bit) and more than 60 mod's

Share this post


Link to post
Share on other sites

Wow! I just stumbled into this because of @4x4cheesecake's post(thank you). downloading now. I need a serious stutter reduction (too many mods, too many parts, too many ships) and I have headroom for days (32GB). 

Share this post


Link to post
Share on other sites
1 minute ago, AngrybobH said:

Wow! I just stumbled into this because of @4x4cheesecake's post(thank you). downloading now. I need a serious stutter reduction (too many mods, too many parts, too many ships) and I have headroom for days (32GB). 

:D

Don't use too much of your RAM to increase the mono heap. If more memory is used, the GC will need more time so your game will 'freeze' for a second or two (tested with roughly 20gb added to the heap) At least, this will happen just every 10min^^ But feel free to test it on your own ;)

Share this post


Link to post
Share on other sites
8 hours ago, 4x4cheesecake said:

@linuxgurugamer the version file contains a wrong mod name, I guess you copied this one from Kerbal Slingshotter? ;)

Oops.

Wasn't in a significant place, but I'll get it fixed

Thanks

Edited by linuxgurugamer

Share this post


Link to post
Share on other sites
7 hours ago, 4x4cheesecake said:

:D

Don't use too much of your RAM to increase the mono heap. If more memory is used, the GC will need more time so your game will 'freeze' for a second or two (tested with roughly 20gb added to the heap) At least, this will happen just every 10min^^ But feel free to test it on your own ;)

Yup. One approach would be to add a little RAM. I think hitting the add RAM button will add RAM each time you hit it ie first time say 0.5Gb, next time another 0.5Gb, so the total would be 0.5Gb then 1.0Gb etc. Once you work out what value works for you, you can replace your increment value with that.

Share this post


Link to post
Share on other sites

New release, 1.1.1.1

  • Updated mod name in .version file

Share this post


Link to post
Share on other sites

So this works, such great work! Much Props. For a long time this was an insoluble, hard-wired problem. This is a small miracle.

 

The default heap settings extend the gap between GC from about 20s to about 50s, so far so good.

Im not certain on how to increase the amount of "heap", I changed the values in the heap textfile from 1 to 10, is that how you do it? Thats megabytes right?

Is 10 reasonable? I am not RAM limited as yet, 24gb installed. Could I do more?

I have anecdotal evidence howver, that the 10-10-10-10...etc setting in the heap textfile reduced stability? Is that to be expected? I had a few extra crashes happen soon after adding extra heap with those settings.

On the other hand, Im running something like 200mods and my system crashes randomly every couple of days anyway (I think its a hardware mismatch as it can happen at any time, but its rare enough that fixing it isnt really worth the bother) so it could easily be coincidence.

 

 

10 hours ago, 4x4cheesecake said:

:D

Don't use too much of your RAM to increase the mono heap. If more memory is used, the GC will need more time so your game will 'freeze' for a second or two (tested with roughly 20gb added to the heap) At least, this will happen just every 10min^^ But feel free to test it on your own ;)

I can easily live with a 2s pause if the gap is 10min! What settings did you use for this?

 

***

One more question - does it matter when you add the heap? Does it work the same from space centre or VAB that it does in-play? I get a lot of pauses in the VAB when building larger vessels.

Share this post


Link to post
Share on other sites
11 minutes ago, p1t1o said:

I can easily live with a 2s pause if the gap is 10min! What settings did you use for this?

I can't remember, sry. Haven't used the mod for a while since I'm no longer running 80+ mods.

You might want to read through the original thread, i.e. on page 13 is an discussion about different settings for the padhead.cfg (the last number in this file is basically the increment of your heap everytime you press 'alt + end', you can ignore the other values) Don't forget to enable 'applyPadding' in your settings.cfg.

Share this post


Link to post
Share on other sites
4 minutes ago, 4x4cheesecake said:

I can't remember, sry. Haven't used the mod for a while since I'm no longer running 80+ mods.

You might want to read through the original thread, i.e. on page 13 is an discussion about different settings for the padhead.cfg (the last number in this file is basically the increment of your heap everytime you press 'alt + end', you can ignore the other values) Don't forget to enable 'applyPadding' in your settings.cfg.

Ah, the last number you say, where it says "total" I think? I did not actually touch that number....maybe that explains the crashes perhaps? I havent touched "settings.cfg" but assume it is setup ok as the default heap settings work.

Thanks, I'll try some stuff later tonight :)

 

Share this post


Link to post
Share on other sites

@LGG: Just FYI, the version in the ingame window title says 1.1.0.3. I assume that just got forgotten to update. BTW, thx for the maintenance! :)

Edited by Jebs_SY

Share this post


Link to post
Share on other sites
5 minutes ago, Jebs_SY said:

@LGG: Just FYI, the version in the ingame window title says 1.1.0.3. I assume that just got forgotten to update. BTW, thx for the maintenance! :)

yes, i didn't realize he had that there.  I'll get it removed

Share this post


Link to post
Share on other sites

New release, 1.1.1.2

  • Removed version number from window title

Share this post


Link to post
Share on other sites

I've updated the OP with a short discussion on the padheap.cfg file

Share this post


Link to post
Share on other sites

New release, 1.1.1.3

  • Added help and legend window
  • Added 512mb & 1gb to the scale

Share this post


Link to post
Share on other sites

Thanks so much, the fix GC heap solved all my horrible stutter problems

Share this post


Link to post
Share on other sites

@linuxgurugamer OP says that this is available via CKAN; this does not appear to be the case.  The old version of this mod is there, but your version is not.

Share this post


Link to post
Share on other sites

You were correct, it's fixed now.  You will need to refresh CKAN to see the updates

Thank you

Edited by linuxgurugamer

Share this post


Link to post
Share on other sites

Pretty much a invaluble tool for a smooth sailing experience, so thank you for this mod.

It's pretty much optimal here after some fiddeling with no reds in sight, although im running a bare minimum asortment of mods on v1.4.5

Before:

screenshot0.png?raw=1

After:

screenshot3.png?raw=1

Edited by Cratzz

Share this post


Link to post
Share on other sites

My game didn't stutter much (only in flight, every 5-10 seconds) but oh boy did it annoy me. I have pretty good specs (R7 2700X, 16GB 3200mhz ddr4, GTX 1080, MX500 1TB SSD) and yet it stutters as much as it di on my 2600K rig. That is until I found this mod :D Linuxgurugamer saves the day, again

 

Edit: Do I need to pad the heap every time I play or does the mod do it automatically after setting it up to my liking?

Edit 2: If I understand correctly, enabling "applyPadding" in the settings.cfg does what I want. 

Edited by kendoka15

Share this post


Link to post
Share on other sites

I have 8Gb of RAM
Intel 5   3.00 Ghz

2GB GT 720 (not so great video card)

55 mods installed
 

My question is:

How should I edit my pad settings to be optimal to my computer?
THanks!

Share this post


Link to post
Share on other sites

So, after pressing MOD-END one time too often, it halts every 6 seconds ...

How to revert that without rebooting KSP?

Share this post


Link to post
Share on other sites

could someone tell me what Mod-KeypadMultiply is?

Edit-

I think I found it. Alt-key?

Edited by Sidestrafe2462

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.