[1.2.x/1.3] MemGraph 1.1.0.3 - with Stutter Reduction

[Padishar]

15 minutes ago, p1t1o said:

Garbage collection is probably a necessary evil - but why does the game have to freeze solid whilst garbage collection is done? Surely it is no slower or faster than any of the other processes that are occurring? I'm assuming a lot of games have some kind of garbage collection process, why don't they stutter?

Firstly, while the garbage collection process (at least all the ones that are in common use) is occurring, nothing else must be allowed to allocate any memory or change the contents of any reference variable because the collector has to determine which blocks of memory are still being used and it does this by first assuming that all blocks are no longer needed and then scanning through all the memory looking for block references which it then marks as still used.  Theoretically, there are ways to implement a garbage collector that doesn't "stop the world" for any appreciable time but they would result in the allocation of memory taking significantly longer which is one of the main reasons for using a garbage collector mechanism in the first place.

The garbage collection process in KSP can take anywhere from <10ms to >500ms depending on the speed of the CPU, the speed of the memory access and the size and number of memory blocks that are allocated. At 60fps there are <17ms per rendered frame so, if the game is paused for even 10ms, it is quite likely that the next rendered frame will not get done in that 17ms resulting in a stutter.  A more usual (for KSP) time would be 80-120 ms so you will miss between 5 and 7 frames of the 60 that should occur in a second.  If the GC takes 500ms (for whatever reason) then you will lose half the frames...

Garbage collection is only used by some programming languages (well, strictly speaking, language runtimes) such as Java and C# (or other .NET/Mono based languages).  A lot of games (and underlying game engines) do not use it precisely because of the stalling issue.  If you have to explicitly code the releasing of unwanted memory (e.g. in C or C++ using the standard allocators) then the programmer has control over when the CPU is used to do it rather than letting the language runtime decide to do it when it feels like it.  Those games that are GC based and don't stutter have simply been written in such a way that they do not create significant amounts of garbage during normal gameplay.  As you can see from Tig's graph, his game produces somewhere between 10 and 20 MB of garbage every second, so code is allocating and discarding more than 300 KB of memory for every 60fps frame that is rendered.  After using the heap padding feature his stutters are approx. one minute apart rather than 15 seconds apart but, if KSP were only creating 30KB of garbage each frame then they would be 10 minutes apart and could be deliberately triggered when the user wont notice the pause.

1 minute ago, Broco said:

They either use more advanced Garbage Collectors (Unity's is pretty old), they create less garbage, they use call the GC manually in certain events (example Skyrim: When you get in or out of a cave, aka loading screen).

Yes, all of these things can reduce the perceived stutter and, in combination, can reduce it to a level where it is effectively unnoticeable.  However, the most important part is creating less garbage.  The same is true in any language, if you're trying to write a program that requires consistent, fast performance, then you don't waste CPU cycles constantly allocating and releasing temporary memory.  If you do it in any GC based system you will get a stutter, do it in a non-GC system and you will simply get poorer general performance.  If any game written in C or C++ did 10000+ calls to malloc/free/new/delete every frame, it would be very unlikely to manage 60 fps...

[p1t1o]

@Padishar @Broco Thanks guys.

Is the 1.5gb "padding" arbitrary? Could one, for example, if you had a lot of excess RAM - lets say KSP runs at 7GB but I have 16 - set the "padding" to 5gb or more to further increase the time between stutters? Or could this introduce further problems, like the GC taking longer each cycle?

Does the "padding" function simply because there is a larger garbage pool so it doesn't have to be collected as often? Or is it more involved?

If the period in between stutters could be increased arbitrarily (based on RAM capacity I suppose), it could easily reach a point of minimal invasiveness without having to be eliminated entirely. I think mine stutters about every 20-30s, if it was 5mins I don't think I would notice it at all, or if I did, its annoyance would be orders of magnitude less.

Cheers

[Padishar]

2 minutes ago, p1t1o said:

Is the 1.5gb "padding" arbitrary? Could one, for example, if you had a lot of excess RAM - lets say KSP runs at 7GB but I have 16 - set the "padding" to 5gb or more to further increase the time between stutters? Or could this introduce further problems, like the GC taking longer each cycle?

The 1.5 GB is as a result of my testing in a limited range of situations and I do plan to tune the padding mechanism further and also allow the user some control over it.  Due to how the memory allocator and GC work, it isn't currently possible to forcibly pad the heap for large block allocations (this is anything over 2032 bytes) and it is very inefficient on memory usage to pad it for blocks above 500 bytes.  These factors combine to make the improvement highly dependent on the pattern of memory usage (specifically, how many blocks of what different sizes are being allocated).  If the memory usage was (virtually) all in small blocks then the improvement would be considerably better.  Large blocks have to be allocated directly from the normal heap headroom that Mono maintains (rather than the "pre-allocated blocks" that the padding mechanism creates) and this is simply a proportion of the total heap size (and it looks like Unity deliberately reduces this proportion as memory use increases to prevent too much free memory being retained ).  Increasing the padding does generally always increase the time between stutters but the benefit shrinks rapidly as the heap gets larger and, yes, the padding results in a lot more work for the collector (though I'm also investigating a cunning way to minimise this).

[p1t1o]

@Padishar

Cheers, good luck and thanks for the hard work!

[Tig]

 

11 hours ago, stk2008 said:

This is probably a stupid question but

What is the Mod+End key?

I see End key on my keyboard but whats the Mod key on windows as I assume Mod key is for Linus?.

 

Thanks again

Although its been answered.  Its not a stupid question at all.    I started using the platform agnostic "Mod" instead of "Alt" because pre 1.1 I tried out Kerbal on Linux and was cursing at why I couldn't load up the debug.  Different "Mod" key.  Took me 10 minutes to figure it out.   *facepalm*.  You'll get no judgment from me, lol. 

 

11 hours ago, Padishar said:

As an example, yes, there is a reason for the positive slope between collections.  The actual rate of memory allocation is a lot more constant but the memory allocator works on 4 KB pages and it groups allocations of similar sizes together into the same 4KB pages.  [snip]

If there isn't an existing page with empty space for the correct size range of block then the allocator will take a page that is currently not used at all and add it to the list of pages used for that block size range.  It's only when a page is added like this that the memory usage can be detected.

Thank you that makes perfect sense now.  Looking at my graph above I perceive, albeit somewhat poorly defined, small stair-steps in my MemGraph chart.  I'm wondering if that's happening as it runs out of larger block sizes.

I can see how managed memory languages were probably really popular with guys/gals coming from C/C++ into something like C# for some applications.  But if there's anything I've learned from this discussion its that even if your memory is "managed" if you don't understand memory management and specifically how your memory management works you can really screw things up still.  I've seen memory issues in other managed games, perhaps there is a new generation that never had to deal with unmanaged languages and some of them are exacerbating the problem.  At least with C/C++ if you messed up your memory management typically the consequences were very bad very quickly - hence the bug got found and smashed.   Now it seems like managed languages can allow people to use very poor techniques, especially in how/where they create, but it still works... and if the program is small enough the issue is not perceptible.

Or maybe its just that I'm crusty and bitter that I had a professor who had us learn SPARC assembly...

11 hours ago, Padishar said:

So much for the people that complain that the game is "unplayable" (without qualifying it with "for me").

And with 101 mods! Although to be fair some are dependencies, plus about 10 custom MM cfgs.

11 hours ago, Padishar said:

The "not responding" dialog isn't really a "crash" warning.  It simply tells the user that the program has not responded to a Windows message that it was sent, usually because it is busy doing something and isn't bothering to run the main message processing loop in the program.  Most programs that do any kind of long calculation will do the same thing but often only when the user tries to interact with the program while it is busy (e.g. if you click on a busy window you will often trigger this dialog instantly because Windows sends the program a "click" message but it doesn't respond to it immediately).

So true, I wrote that in my post because so many people think it is always a crash warning... to be fair, with windows 9 out of 10 times (or more) it probably is a crash.

I wonder how many of the "my game is crashing, i quit" reports are just people being impatient with this warning and not waiting a few seconds.

11 hours ago, Padishar said:

I suspect these may be caused by the game auto-saving (though every minute seems a little frequent so it may be some other regular housekeeping task that KSP does).

Not a problem, you've been more than helpful.  I used to use fraps every time I played the game and I've only ever seen this happen once though I've hardly used fraps with KSP 1.1.  I spent a considerable time trying to repeat it but I couldn't.  I switched from using fraps to using a mod to get better accuracy at low frame rates (fraps only displays integer values in its overlay and there's a lot of difference between 2 and 3 fps).

Autosave interval is set to 5 minutes, although there is something in the settings.cfg i hadn't noticed before called AUTOSAVE_SHORT_INTERVAL, its set to 30.  I changed it to a few different values, didn't appear to have any effect - found a few other posts with the same results.  In any event I also use the S.A.V.E. mod, but its set to 20 minutes intervals.

I use fraps mainly for the frametimes; I'm big into the frametime analysis and happy to see that more and more gaming websites are focusing on the metric. 

Its subjective, it always is with fps, but my personal preference is for consistency over average speed.  In theoretical gaming world I'd rather a rocksolid 40 fps over 45 fps that is bouncing between 35 and 55.  Totally subjective.

And yes, the integer does bother me with fraps.  Especially since its not only the overlay but the freaking logs as well, which i just don't understand since the frametime log is very precise.  Sure, I could recreate my own more accurate fps from the frametime log but why make me go through the hassle... sigh.

-----

Thank you as always.  If I may, what aspect of Kerbal/Unity/Mono is causing the huge increase in total memory being used?  From the beginning of my play session to the end the HWM grew about 800MB or so (from a max of ~3000 to 3800ish) so unless I totally misunderstood you on how blocks are freed and new pages allocated that shouldn't be the issue (or rather only 800MB of it), but my total memory went from 6.8GB to 13.1GB.  What else is in there?  I thought the vast majority of the rest of the memory being used was for models/textures.  My understanding was that all that gets loaded at the start and then should just remain static in memory.  But clearly I'm missing something... about 5 gigs of something.  

 

 

Edited May 27, 2016 by Tig
typographical

[spacehorse]

Thank you for the good work, I am not technical enough but looks like work for me and the "gameplay" feeling is a lot of better after activation (Looks alos in the graph) THANKS!

Edited May 28, 2016 by spacehorse

[N1njawarrior]

2 hours ago, spacehorse said:

Thank you for the good work, I am not technical enough but looks like work for me and the "gameplay" feeling is a lot of better after activation (Looks alos in the graph) THANKS!

from my experience as a coder, this seems like it would work, but my mac is too slow for it to do much, in fact it's so slow that this mod makes it worse, which is a shame, I was really looking forward to something like this, but I must keep my scene switching times below 2 minutes.

But again, good work, I like what you're doing here.

[Padishar]

41 minutes ago, N1njawarrior said:

from my experience as a coder, this seems like it would work, but my mac is too slow for it to do much, in fact it's so slow that this mod makes it worse, which is a shame, I was really looking forward to something like this, but I must keep my scene switching times below 2 minutes.

But again, good work, I like what you're doing here.

There is a possibility that the Mac build of Mono uses slightly different private data in each memory block resulting in different block sizes or simply tracks different block sizes to the windows build.  If this is the case then there may be no padding created for one or more block sizes which would reduce the effectiveness.

I am surprised that it would make the performance worse though.  What do you actually mean?  Are the GC stutters closer together after padding the heap or does the game just run slower in general (the U value in the window is basically fps)?  How much RAM does your machine have?  Any chance you can do a test exactly as described in the first post and then upload your player.log file?

[N1njawarrior]

Just now, Padishar said:

There is a possibility that the Mac build of Mono uses slightly different private data in each memory block resulting in different block sizes or simply tracks different block sizes to the windows build.  If this is the case then there may be no padding created for one or more block sizes which would reduce the effectiveness.

I am surprised that it would make the performance worse though.  What do you actually mean?  Are the GC stutters closer together after padding the heap or does the game just run slower in general (the U value in the window is basically fps)?  How much RAM does your machine have?  Any chance you can do a test exactly as described in the first post and then upload your player.log file?

This mod doesn't make the stutters any worse, (that would be pretty hard) or cause a hit in frames/fps, (again, that would be hard to do.) The performance hit I spoke of doesn't actually happen while playing the game, on when switching scenes, it cause the loading times when doing things like launching a vessel, or entering to the SPH to be significantly longer.

In terms of the machine I'm running, it's a six-year-old mac-mini, and one of the ram slots is broken, I currently have 4GB of ram. (8GB if both slots worked) I will try to run run that test as soon as I get the chance, if there is anything else I can do to help, just let me know.

[Padishar]

3 hours ago, N1njawarrior said:

I currently have 4GB of ram.

Ahhh, the heap padding increases the memory usage of KSP by a little over 1.5GB (at the main menu my x64 KSP on Windows has a commit size of 2.2 GB and this rises to 3.8 GB after doing the padding).  This will cause a machine with only 4GB of RAM to have a bit of a fit so that isn't really a surprise.  I will be modifying the padding mechansim to make it tunable which should allow it to be used on a machine with only 4GB, though since the whole point of it is to trade memory usage against GC frequency, the benefit with so little RAM will never be great.

Edit: I have edited the first post to make it clear that it currently wont work well with only 4GB of RAM.

Edited May 28, 2016 by Padishar

[Gordon Dry]

Could it be that MemGraph after "calibrating" has the side-effect of KSP using much more RAM than without?

[Padishar]

22 minutes ago, Gordon Dry said:

Could it be that MemGraph after "calibrating" has the side-effect of KSP using much more RAM than without?

What are you trying to imply?  In what way is that not totally obvious from all these:

On 5/6/2016 at 10:52 AM, Padishar said:

However, I have since devised a way to force Mono to keep significantly more free space in the heap,

On 5/6/2016 at 10:52 AM, Padishar said:

Mod-End pads the Mono heap with approx 1.5 GB of headroom to reduce frequency of garbage collections.

20 hours ago, Padishar said:

Ahhh, the heap padding increases the memory usage of KSP by a little over 1.5GB (at the main menu my x64 KSP on Windows has a commit size of 2.2 GB and this rises to 3.8 GB after doing the padding).

 

Edited May 29, 2016 by Padishar

[Gordon Dry]

Okay, the last sessions my RAM usage was about 11.9 GB, now I had 16 GB RAM usage - that's my physical RAM and I had problems to manage the situation - I don't like reset buttons, I prefer killing hanging processes, I needed 10 minutes to manage that...

Edited May 29, 2016 by Gordon Dry

[Padishar]

6 minutes ago, Gordon Dry said:

Okay, the last sessions my RAM usage was about 11.9 GB, now I had 16 GB RAM usage - that's my physical RAM and I had problems to manage the situation - I don't like reset buttons, I prefer killing hanging processes, I needed 10 minutes to manage that...

I'm sorry, I simply don't understand what you're trying to say.  If you're trying to report an apparent problem caused by this mod then you'll need to accurately describe what you did in the session (with, I presume, details of the memory usage of KSP at different times during the session, including exactly which measure of memory use you are quoting) and, preferably, post an output_log.txt/player.log file of the session.  To be really thorough, you should repeat the whole session without using the heap padding so a valid comparison can be made.

[Padishar]

I have now released version 1.0.0.7.  This adds readouts of the minimum (just after GC) and maximum (just before GC) heap sizes and the time between the last two collections. It also makes the heap padding configurable via a text file.  The default settings now allocate approx. 900 MB of padding and the screenshot in the OP now shows the effect of running this.  Before the padding the collections were happening at 13 second intervals and after the padding this increased to 44 seconds.  Modifying the configuration to allocate around 2.5 GB of padding increased it to 112 seconds.

Edit: @p1t1o, now that it's configurable you can simply edit the values in padheap.cfg, e.g. if you add a 0 on the end of all the counts then it should allocate more like 9GB extra.  If it does work and does give a decent extra effect then you might want to tune it back a bit to give you more headroom.  The counts in the file are simply counts of the number of pre-allocated blocks to create for each block size.  The current values are fairly arbitrary, simply based on sharing the total padding fairly evenly between the block sizes with a considerable bias towards smaller blocks which was experimentally determined to improve the effect in the limited number of situations I tested.

Edited May 31, 2016 by Padishar

[Gordon Dry]

Seems to work better now, I have to investigate further.

I suggest to change the settings files from .cfg to.xml to avoid MM patch rebuilding after settings were changed.

[Padishar]

22 minutes ago, Gordon Dry said:

Seems to work better now, I have to investigate further.

Thanks for the reply.

22 minutes ago, Gordon Dry said:

I suggest to change the settings files from .cfg to.xml to avoid MM patch rebuilding after settings were changed.

What makes you think that changing any of the settings files causes MM to ignore the cache?  As far as I'm aware, MM only rebuilds when files that are loaded into the GameDatabase change and the ones from this mod aren't due to them being in the PluginData folder.  In any case, MM specifically logs the names of the files that cause it to rebuild so, does your log explicitly say that one of this mod's ,cfg files caused a rebuild?

[p1t1o]

19 hours ago, Padishar said:

Edit: @p1t1o, now that it's configurable you can simply edit the values in padheap.cfg, e.g. if you add a 0 on the end of all the counts then it should allocate more like 9GB extra.  If it does work and does give a decent extra effect then you might want to tune it back a bit to give you more headroom.  The counts in the file are simply counts of the number of pre-allocated blocks to create for each block size.  The current values are fairly arbitrary, simply based on sharing the total padding fairly evenly between the block sizes with a considerable bias towards smaller blocks which was experimentally determined to improve the effect in the limited number of situations I tested.

Thanks for the heads-up! Sounds great! Unfortunately I'm at 97% RAM usage at the moment (why did I have to save just a few bucks and only spring for 8GB? Talk about bottleneck...) so I'm guessing this isn't going to do much for me until I can bulk that up a bit?

[Padishar]

3 minutes ago, p1t1o said:

Thanks for the heads-up! Sounds great! Unfortunately I'm at 97% RAM usage at the moment (why did I have to save just a few bucks and only spring for 8GB? Talk about bottleneck...) so I'm guessing this isn't going to do much for me until I can bulk that up a bit?

Ahhh, sorry, I thought the example in your post indicated you had 16GB.  Yeah, if you haven't got much physical memory free then it might well do more harm than good, even at the default settings...

[Guest]

KSP has indirectly trained me over time to keep my "garbage" low. I usually don't let my number of active flights get over 15, and I clean up debris as quickly as possible in the tracking center. The information in this thread explains why I have been having to do this. I had noticed over time that the more on rails flights that I had, the worse my game performed, but I never would have suspected on-rails flights were causing garbage collection problems. I'm really excited now that this information has been brought to light. Padishar, thank you for creating this tool. I feel it's only a matter of time now that we can all start seeing major benefits from this. This thread deserves a sticky and should be required reading for all KSP developers moving forward, IMO.

[Wiseman]

First of all, I wanted to thank you for this stutter reduction experiment. I haven't been able to play at all since 1.1.2 because of the frantic garbage collection going on. I was curious, though, if you know what's causing this behavior:

Spoiler

Here's a link to my log file.

I have a pretty hefty modlist (142 according to CKAN, and a few manually installed visual mods as well), including Kopernicus and OPM, as you can see in the screenshot above. I tripled the size of the padding in the config file, and reduced my frame limit to 60. In flight view, that increased my interval to about 72~ seconds, which is great! (The wide swaths of the second third of the log) But, when switching to map view, it dropped down to 20 seconds briefly (the end of the log and back to the left side), then went back up if I stopped touching things. The real issue issue was when I focused on another planet, the garbage collector seems to have gone into overdrive and was stuttering every few frames. 

Update: I think this has to do with Kopernicus, OPM, and/or the OPM visual overhaul mod, or some combination thereof. The framerate gets cranky every time I scroll in or out from a particular body, but that seems in line with my thinking as it coincides with the loading/unloading of those bodies. However, anytime I'm focused on a planet with rings, the memory usage and resultant garbage collection shoots through the roof. In any case, this seems unlikely to have anything to do with your mod, so my apologies!

Edited May 31, 2016 by Wiseman
Update with further testing

[Tig]

On 5/30/2016 at 6:43 PM, Padishar said:

I have now released version 1.0.0.7.  This adds readouts of the minimum (just after GC) and maximum (just before GC) heap sizes and the time between the last two collections. It also makes the heap padding configurable via a text file.  The default settings now allocate approx. 900 MB of padding and the screenshot in the OP now shows the effect of running this.  Before the padding the collections were happening at 13 second intervals and after the padding this increased to 44 seconds.  Modifying the configuration to allocate around 2.5 GB of padding increased it to 112 seconds.

Edit: @p1t1o, now that it's configurable you can simply edit the values in padheap.cfg, e.g. if you add a 0 on the end of all the counts then it should allocate more like 9GB extra.  If it does work and does give a decent extra effect then you might want to tune it back a bit to give you more headroom.  The counts in the file are simply counts of the number of pre-allocated blocks to create for each block size.  The current values are fairly arbitrary, simply based on sharing the total padding fairly evenly between the block sizes with a considerable bias towards smaller blocks which was experimentally determined to improve the effect in the limited number of situations I tested.

Hey @Padishar, thanks for the update.  Going to do more full testing on different sizes when I have time, busy week...

I've done some limited testing though and just wanted to give the TLDR version since you know my settings and the way I test by now

Everything works *exactly* the way you would have theorized in advance.  More heap = longer interval = longer stutter frame in ms, but the increase in said frame is slight compared to the increase in the intervals.

Thx again.

PS.  To all other users, I'll say this a bit stronger than the mod author, this mod is in no way for people with 4GB of ram, and frankly be careful with your MemGraph settings with 8GB of ram if you're running stock.  Mods will run you out of 8GB of ram very quickly in my experience with MemGraph and GCMonitor.  I'm seeing my total ram usage (not heap) grow even in stock with every scene change still and mods exacerbate that issue, at least for me.  The moment you run out of physical ram by using this mod, which I've tested, you'll see a big drop in performance even on an ssd.  (I have a Samsung EVO, maybe the swap on an m.2 would be fast enough, i don't know, but I'll take donations towards buying one for testing purposes. )

 

Edited June 1, 2016 by Tig

[Galenmacil]

I was going to write how I was unable to understand the new feature in MemGraph 1.0.7 until I finally figured it out. I used the "extra 0 at the end of each line" Padishar suggested and It seems to make a huge difference on the frequency of KSP garbage collection.

Edit: Reduced garbage collection frequency to 120s in between.

Edited June 1, 2016 by Galenmacil

[Tig]

26 minutes ago, Galenmacil said:

I was going to write how I was unable to understand the new feature in MemGraph 1.0.7 until I finally figured it out. I used the "extra 0 at the end of each line" Padishar suggested and It seems to make a huge difference on the frequency of KSP garbage collection.

Glad you figured it out and glad it's solving this nasty issue for lots of us.

For anyone confused by the numbers, my personal recommendation would be to double or triple @Padishar's numbers after the colon in padheap.cfg if you're running 16GB of ram.  Watch your ram usage with GCMonitor and adjust as you see fit.  I wouldn't change his numbers much with mods and 8GB of ram, but again the mods - how many/which ones - can have a profound effect.  There is NO one-size-fits-all settings with MemGraph to lessen with the stutter issue.   I'm just trying to give you a good starting point for anyone confused... hopefully it helps a few.

I wouldn't recommend just adding a zero to the end unless you're in @Galenmacil's range of 32GB+ ram.  ( hehe, saw your post before your edit   )

Edited June 1, 2016 by Tig
clarification.

[p1t1o]

14 hours ago, Padishar said:

Ahhh, sorry, I thought the example in your post indicated you had 16GB.  Yeah, if you haven't got much physical memory free then it might well do more harm than good, even at the default settings...

Haha no it was literally an example/wishful thinking on my part! I do intend to increase my RAM when I get a chance so I am keeping an eye on this for later

Mo' address space, Mo' problems