[1.2.x/1.3] MemGraph 1.1.0.3 - with Stutter Reduction

[Padishar]

This started as a simple plugin that displays a graph of the Mono heap allocation rate and garbage collection, mainly intended as a troubleshooting and development aid rather than for general use (hence why I originally released it in the Add-on Development sub-forum rather than this one).  However, I have since devised a way to force Mono to keep significantly more free space in the heap, which can significantly reduce the frequency at which the heap fills up and the Mono garbage collection causes a stutter, so I have added it to this mod and released it more publicly.  I will be continuing to work on this mechanism to improve it further.

The code has been highly optimised to create as little garbage as possible (given the constraints of the old OnGUI based API) so that it interferes with what it's measuring as little as possible.  I do have plans to add various "usability" type enhancements but only things that can be implemented cleanly and are directly useful for the mods purpose.

To install, simply download the zip from the link below and copy the MemGraph folder from the zip file into the GameData folder of your KSP installation.

Mod-KeypadMultiply toggles the display of the window.
Mod-KeypadPlus increases the vertical scale of the graph.
Mod-KeypadMinus decreases the vertical scale of the graph.
Mod-KeypadDivide runs a bit of test code controlled by MemGraph\PluginData\test.cfg
Mod-End pads the Mono heap with a configurable amount of headroom to reduce frequency of garbage collections.

Every second the plugin totals up all the memory allocated on the heap and whether any garbage collections have run.  It also displays the current total heap allocation, the maximum heap allocation just before the last garbage collection, the minimum just after the last collection, the number of "render" and "physics" updates and the time between the previous two collections.

The graph is 600 pixels wide and shows the last 10 minutes of the memory allocation in green.  If a garbage collection happens during the interval, that column of the graph will have a red background.

The test code basically allocates a number of blocks of the specified size and displays which allocations actually cause the allocated heap to change.  This allows us to deduce various characteristics of the memory allocator and garbage collection mechanisms (and was a crucial part of developing the heap padding mechanism).

Mod-End activates the heap padding.  The amount of padding is controlled by the MemGraph\PluginData\padheap.cfg.  The format of the file is very simple.  Each line controls the amount of padding allocated for blocks of each size range.  The first value is the size of each block allocated and the second is the number of blocks.  The first values are only present for illustration, they don't actually control the size of the blocks, these are hardwired to the sizes in the default configuration.  The total value at the end is the total number of megabytes of padding allocated (note, the memory usage of KSP will increase by more than this due to the free space headroom maintained by Unity).

I recommend that you run the game normally and load up a situation that has noticeable stutter.  Display the graph, setting the scale so the regular allocation rate fits nicely and the garbage collection red lines can be seen.  Let it run for several runs of the garbage collector and then hit Mod-End.  After a short pause, the game should continue with a considerably larger gap between the collections.  After another few collections hit Mod-End again and it may improve further.  I would appreciate feedback about how well this works, e.g. screenshots of the graph taken during this process and after another few collections would be very helpful along with details about your setup (and preferably, an output_log.txt/player.log file).  Many thanks to @Tig for the testing and very well presented data he posted over in the Grumpy Collector thread.

One other thing I should add, though it should be obvious with only a little thought, is that the heap padding mechanism is only intended for 64 bit versions of the game.  Trying to allocate 900 MB of extra heap space on the 32 bit version is unlikely to be successful and, if it is, then it will probably cause the game to crash before long due to running out of address space. It is also unlikely to work effectively if your machine has only 4GB of RAM as the total usage of KSP is likely to grow close to 4GB even without loading a save, resulting in virtual memory paging which will seriously hurt performance.

Obligatory picture of mod in use:

Download MemGraph for KSP 1.2.x and 1.3

Download MemGraph for KSP 1.1.x

Source on GitHub

Previous releases on GitHub (versions 1.0.0.x are for KSP 1.1.x, versions 1.1.0.x are for KSP 1.2).

The code is released under the MIT license (see https://github.com/Gerry1135/MemGraph/blob/master/Graph.cs).

There's no obligation and it won't entitle you to any special treatment but, if you feel like it, then you can make a donation using the link below.

Donate Here

 

Edited June 19, 2017 by Padishar

[Padishar]

Just released a small update to 1.0.0.1 to slightly optimise the UI code (changed from using GUILayout to GUI).

[Padishar]

Another small update to add the current total of heap allocated (HWM).  The new screenshot in the OP shows an "interesting" (not to mention, in need of some serious optimisation) issue.  The point earlier in the graph where the memory allocation rate was ~5 MB/s was when flying the identical vessel to in the screenshot but in a save that also contains 4 fairly large space stations (2 of ~300 parts and 2 of ~700 parts).  The point where the shot was taken was in a copy of the original save with the stations terminated and shows a rate of ~500 KB/s.  That's a lot of memory usage for 4 vessels outside of physics range that should be needing little CPU and certainly shouldn't need to be allocating anything like that much memory.

[drtedastro]

Thanks.

I have been looking for something like this.

Cheers.

[Padishar]

11 hours ago, Padishar said:

The new screenshot in the OP shows an "interesting" (not to mention, in need of some serious optimisation) issue.  The point earlier in the graph where the memory allocation rate was ~5 MB/s was when flying the identical vessel to in the screenshot but in a save that also contains 4 fairly large space stations (2 of ~300 parts and 2 of ~700 parts).  The point where the shot was taken was in a copy of the original save with the stations terminated and shows a rate of ~500 KB/s.  That's a lot of memory usage for 4 vessels outside of physics range that should be needing little CPU and certainly shouldn't need to be allocating anything like that much memory.

I've now done a more detailed examination of this behaviour and conclude that, during flight, there is a rate of memory allocation that is proportional to the total part count of all the on-rails vessels in the save.  This results in large and complex saves suffering significantly more with GC related stutter.

This file contains the various saves the tests were run on.  They are all based on the "Platonic Stations" save which contains the top stage of a KerbalX in a ~200km orbit and four stations with part counts of 308, 308, 642 and 1017.  The other saves all have one or more of the stations terminated to reduce the overall part count with names that indicate what is present, e.g. TetraOcto is the version with 1 tetrahedron (308 parts) and the octahedron (642 parts).

This table shows the approx. allocation rate from the screenshots above.

Save		On-Rails Parts	Alloc Rate KB/s
JustJeb			0		600
Tetra			308		1400
TetraTetra		616		2100
Octo			642		1900
TetraOcto		950		2600
TetraTetraOcto		1258		3300
Platonic		2275		4900

This shows a pretty consistent increase with overall part count.  The only exception being TetraTetra allocating more than Octo despite the slightly lower part count.  I suspect this is due to there being a per-vessel component as well as a per-part one.

This only seems to occur in the flight scene.  In KSC and the astronaut complex all of the saves allocate around 200-240 KB/s.  In the tracking station they allocate slightly more but still fairly constant with this small number of vessels though I suspect the tracking station will allocate quite a bit more with many vessels.

The code that determines if an on-rails vessel enters the physics bubble is the only bit I can think of right now that relates to on-rails vessels and only happens in the flight scene so this may be a good place to start looking.

I'll be posting this to the bug tracker in the near future...

[stk2008]

Hey

Just read your post in the other thread and thought WOW gonna try this :).

So you seem to be really smart at this stuff sorry to ask here but is the stutter even possible to fix on Squads end or is it really down to Unity?

 

Thanks again

 

P.s

Is there a way to find out exactly what Mod,File,Texture etc etc is being read etc.

So kind of like the windows performance tab but based on KSP`s files and mods loaded?

Edited May 11, 2016 by stk2008

[Padishar]

1 minute ago, stk2008 said:

but is the stutter even possible to fix on Squads end or is it really down to Unity?

Most of the problem is caused by Squad's code and by mods allocating and discarding too much memory.  KSP does push Unity considerably beyond what it's really designed for but there really is no excuse for any game to be constantly allocating and discarding megabytes of memory every second.  This would cause issues whatever game engine (with whatever underlying memory allocation model) was used.  As I said in the other post, people hoping that Unity will update Mono to use a better GC algorithm would be disappointed by the effect (or lack of) it would have on KSP...

[stk2008]

Damn would it be wise to get t he KSP community together and start a partition to get Squad to fix the stuttering.

I dont mean be rude,mean etc about it I mean sign a partiotion to show how many of us are effected by it so they can see thats it worth fixing?.

I for one need it fixed the stutter is terrible and a real deal breaker

[Padishar]

2 minutes ago, stk2008 said:

Damn would it be wise to get t he KSP community together and start a partition to get Squad to fix the stuttering.

I dont mean be rude,mean etc about it I mean sign a partiotion to show how many of us are effected by it so they can see thats it worth fixing?.

Not really.  As I have posted in various other related threads, Squad are aware of the problem and of some of the specific problem areas and are intending to fix them.  The best way to increase the priority of that work would be to add information (and/or votes if that is working) to one or more of the existing bugs in the bug tracker, complete with good information that demonstrates why it is such a bad problem.

[stk2008]

Awesome thank you.

Is there a KSP Github used by the Devs if not I assume just vote here on the forums?.

 

Sorry for all the questions I have recently got KSP and WOW its so much fun ( I am terrble at it) but still its awesome :).

Really cant thank the people that make mods for KSP enough awesome community

 

Edited May 11, 2016 by stk2008

[Padishar]

There is the official Squad bugtracker at http://bugs.kerbalspaceprogram.com/

You should be able to search and read bug reports without an account but you would need to create one to be able to post or vote on anything.  If you do create an account then the best idea is to use the same name as your forum account as this makes it easier for people to work out who you are...

[sarbian]

Fixing memory use is bit more complex than "oh right, let's change those 2 lines"... They are aware of the stutter but this kind of issue is C O M P L E X. No need to pester them...

[Blacks]

well i will add on this is not only about memory, in equation it need to be track more thing like cpu, gpu, graphic engine, what function it use & in what condition or how (see unity + dx9,dx11, opengl), what version it use ... & i agree with sarbian is not a easy task to be done, but i agree with padishar to

[Padishar]

I have just released version 1.0.0.3 that eliminates a couple of bits of garbage creation hidden in the UI code and moves various initialisers to Awake (I think it's only static ones that might cause upset but I've moved every reference member init to Awake to be sure).

Edit: Another quick update to 1.0.0.4 to fix the window dragging (and I've actually updated the download link this time )

Edited May 12, 2016 by Padishar

[Padishar]

Another small update to 1.0.0.5, this time to add a simple allocation test function run on Mod-NumPadDivide.

https://github.com/Gerry1135/MemGraph/blob/master/Graph.cs#L333-L417

This basically allocates a specified number of blocks of a specified size controlled by the MemGraph\PluginData\test.cfg file and reports which allocations result in what changes to the total of allocated memory.  This allows various deductions to be made about how the memory allocation works in Unity/Mono.  I will make a followup post later (well, probably tomorrow some time) describing some of what I have deduced so far...

[Tig]

@Padishar So I don't double post you...

http://forum.kerbalspaceprogram.com/index.php?/topic/139484-112-grumpycollector-101-stutter-what-stutter/&do=findComment&comment=2595367

Related to a feature request for MemGraph, but I thought the primary gist of the post was better suited to the garbage collection thread. 

Edited May 22, 2016 by Tig

[Padishar]

Updated to 1.0.0.6.  Now includes a mechanism to increase the Mono heap headroom to reduce the frequency of garbage collections.  Just hit Mod-End to assign approx 1.5 GB of extra ram to the Mono heap...

[Tig]

Hey @Padishar,

Thought I'd move further testing of MemGraph to well, uhh, the MemGraph thread.

But to anyone interested in garbage collection in Kerbal/Unity/Mono and the steps taken to try and at least stem the bleeding, the Grumpycollector thread is a goldmine of info, even though the mod itself is not useful for the average user, it helped point everyone in the right direction.

 

So after actually playing the game for a while I thought I'd report in on my experience.  First, the game remains incredibly more playable, but I've noticed a few I-wouldn't-call-problems, but I thought they'd be of interest to you.

After a play session of about 3 hours or so - many launches, many landings, many scene changes - I checked in on MemGraph and GCMonitor.

• HWM is moving between ~3100-3850MB. 
  • Obviously quite a bit more than where I start out (hitting Mod-End 3 times and HWM ~2500-3000).
• My stutters still stay in the 55-60 second range, and my fps is locked at 60 - except of course when it dips on a stuttered frame.
  • Stuttered frames seem maybe a bit longer - I wouldn't be surprised if they were that's traditionally been the case as the heap grows
  • That said, same pattern as before, even if they are longer the relative increase in ms compared to the heap size is small
• GCMonitor - Perhaps most interesting, my memory usage has climbed to 13.1GB, lol. 
  • For reference in my short testing sessions, my usage was about 6.8GB. 
  • Windows reports a working set of 10.6 and commit of 11.2
    • As I'm typing this windows now reports 8.5/8.7, GCMonitor hasn't changed.
• Long scene transitions: Moving from the MC or the Lab back to the main screen sometimes takes longer than usual. 
  • In a few instances the typical windows program crash pops up, simply clicking "wait for program to respond" actually works and Kerbal comes back a few moments later. 
  • The problem is my favorite type: Non-repeatable intermittent...  it does appear to get worse the longer I play.  But, sometimes the transition is immediate even 2-3 hours in.  Strange.
• Some exceptional memory usage spikes
  • Brief moments where MemGraph reports 40-50MB/sec demands, see below
• Unfortunately, I don't have hard data for you on frametimes. 
  • I wasn't running fraps, I've heard anecdotally that it can affect Kerbal's preformance, no idea if its true - but since I was playing for my own enjoyment I thought I'd leave it off. 
  • If Kerbal is running and I start up fraps, then Kerbal immediately crashes guaranteed.  Don't know if that's common or an issue with just me, but in any event no precise data on frametimes for you.  sorry.

But I got a picture of MemGraph - looks pretty clockwork on the intervals still:

Note the scale of 64MB, yeah those are 40-50MB spikes.  And the trend line is pretty clear; the memory demanded per second is rising with time, but each GC resets the "clock" if you will.

Is there some reason for the positive slope in between the GCs?  I guess what I'm asking is: why isn't the trend between GCs flat?  Some growing array/list that instead of getting appended to is being reallocated brand new every frame?

Just seems unfortunate with the KB/s being a function of time, its turning the the total heap between GCs into a quadratic.  ick.

[stk2008]

This is probably a stupid question but

What is the Mod+End key?

I see End key on my keyboard but whats the Mod key on windows as I assume Mod key is for Linus?.

 

Thanks again

[Padishar]

10 minutes ago, Tig said:

So after actually playing the game for a while I thought I'd report in on my experience.  First, the game remains incredibly more playable, but I've noticed a few I-wouldn't-call-problems, but I thought they'd be of interest to you.

Thanks for more great feedback, very helpful.  I'll try to get a more detailed reply up in the next few hours, including the detailed description of how the memory allocation works but, for now, I will say that most of the "not-really-problems" are understandable consequences of how the Mono heap works.

As an example, yes, there is a reason for the positive slope between collections.  The actual rate of memory allocation is a lot more constant but the memory allocator works on 4 KB pages and it groups allocations of similar sizes together into the same 4KB pages.  E.g. if some code allocates an object that is 120 bytes in size then this will be rounded up to 144 bytes and allocated from a page that already contains other allocations of 113-144 bytes but isn't totally used (due to the extra padding the allocator adds to each block, 25 such blocks can be allocated in each 4KB page).  Larger allocations (anything over 2032 bytes, where no more than one block can fit in a 4KB page) are handled a little differently, simply using as many 4KB pages from the free page list as are required.

If there isn't an existing page with empty space for the correct size range of block then the allocator will take a page that is currently not used at all and add it to the list of pages used for that block size range.  It's only when a page is added like this that the memory usage can be detected.  If there are no free pages then the collector runs to free up space, either by freeing up a block in a page that holds the correct size range or by freeing up all the blocks in any page so the whole page gets returned to the free list and can then be assigned to a different block size range.  So, immediately after a collection, there will be lots of 4KB pages that have some free space in them which results in some of the memory demand being taken from already allocated pages and they don't show up in the usage rate.  E.g. it could be possible for hundreds of 144 byte blocks to be allocated without needing any new 4KB pages to be assigned to that size range.  Once all the free space in the already assigned pages are used up, a new page will be assigned for every 25 such blocks that are allocated and the reported allocation rate will be much closer to the "real" rate.

A couple of other observations:

58 minutes ago, Tig said:

After a play session of about 3 hours or so - many launches, many landings, many scene changes

So much for the people that complain that the game is "unplayable" (without qualifying it with "for me")...

43 minutes ago, Tig said:

In a few instances the typical windows program crash pops up, simply clicking "wait for program to respond" actually works and Kerbal comes back a few moments later. 

The "not responding" dialog isn't really a "crash" warning.  It simply tells the user that the program has not responded to a Windows message that it was sent, usually because it is busy doing something and isn't bothering to run the main message processing loop in the program.  Most programs that do any kind of long calculation will do the same thing but often only when the user tries to interact with the program while it is busy (e.g. if you click on a busy window you will often trigger this dialog instantly because Windows sends the program a "click" message but it doesn't respond to it immediately).

49 minutes ago, Tig said:

Brief moments where MemGraph reports 40-50MB/sec demands, see below

I suspect these may be caused by the game auto-saving (though every minute seems a little frequent so it may be some other regular housekeeping task that KSP does).

50 minutes ago, Tig said:

If Kerbal is running and I start up fraps, then Kerbal immediately crashes guaranteed.  Don't know if that's common or an issue with just me, but in any event no precise data on frametimes for you.  sorry.

Not a problem, you've been more than helpful.  I used to use fraps every time I played the game and I've only ever seen this happen once though I've hardly used fraps with KSP 1.1.  I spent a considerable time trying to repeat it but I couldn't.  I switched from using fraps to using a mod to get better accuracy at low frame rates (fraps only displays integer values in its overlay and there's a lot of difference between 2 and 3 fps).

 

[UnanimousCoward]

1 hour ago, stk2008 said:

This is probably a stupid question but

What is the Mod+End key?

I see End key on my keyboard but whats the Mod key on windows as I assume Mod key is for Linus?.

 

Thanks again

On Windows, "Mod" is the Alt key. (On Linux, it's the right Shift key, and on Mac it's the Option key.)

So "Mod+End" means (on Windows) "hold Alt and press End".

[Padishar]

1 hour ago, stk2008 said:

What is the Mod+End key?

My apologies, I totally missed this earlier.  UnanimousCoward is totally correct...

[stk2008]

Awesome thanks for the fast reply

 

[p1t1o]

Garbage collection is probably a necessary evil - but why does the game have to freeze solid whilst garbage collection is done? Surely it is no slower or faster than any of the other processes that are occurring? I'm assuming a lot of games have some kind of garbage collection process, why don't they stutter?

[Broco]

35 minutes ago, p1t1o said:

Garbage collection is probably a necessary evil - but why does the game have to freeze solid whilst garbage collection is done? Surely it is no slower or faster than any of the other processes that are occurring? I'm assuming a lot of games have some kind of garbage collection process, why don't they stutter?

They either use more advanced Garbage Collectors (Unity's is pretty old), they create less garbage, they use call the GC manually in certain events (example Skyrim: When you get in or out of a cave, aka loading screen).