[1.12.x] MemGraph Updated with Stutter Reduction

[Gargamel]

Some posts have been edited or removed.   Please refrain from making personal comments. 

[vardicd]

16 hours ago, Jognt said:

Interesting train of thought. What it boils down to is whether it's memgraph specific or due to one (or a combination) of its dependencies.

@Tonka Crash Keep in mind that memgraph was primarily designed to diagnose GC issues. You still have to deal with any Mono peculiarities even when increasing the padheap (and maybe even exactly when increasing the padheap)

How often do you get GC with 4GB of padheap? Because you really are pumping up those numbers which makes me wonder whether you're using it to mask excessive Garbage Creation. Like, how high does "Last" usually go in the Space Center/Flight Scene?

8 hours ago, Tonka Crash said:

@Jognt Your questions are completely irrelevant to the problem I posted. Watch the video. Or even better, test it yourself. I'd like someone else to confirm the problem. With MemGraph I get a lag spike dropping items in KIS, without it, I don't. It's that simple. This is in a stripped install that doesn't at all represent my normal playing environment.  

I don't care what MemGraph was "designed" to do. What it's being used for is to spread out GC intervals, which is variable depending  on what you are doing. I also don't keep track of what MemGraph is doing. As I've stated before I don't have MemGraph open by default and rarely look at it at all. As long as GC stutter is infrequent enough to not annoy me I don't care what the settings are at.

Here's a question I asked in January that went unanswered.

 

6 hours ago, Jognt said:

Did you miss the part where @vardicd said he has the same problem even though he doesn’t use memgraph?

i understand it’s easy to yell “your mod is broken”, but if you really want to get this sorted I suggest you either use the info and questions provided to help find the cause or to just stop using any mods. 

With regards to “I don’t care what it was designed to do” -> you’re using a mod for a purpose it was not (mainly) intended. It doesn’t matter that all of us use it for that purpose. 

[snip]

 

5 hours ago, linuxgurugamer said:

I don't have an answer as to why that happen. I've updated the mogg recently to remove some inefficiencies, but regarding U the stuttering I have no idea because nothing I saw when I was looking at it seemed to indicate that that would cause it.

This is very hard to diagnose because it's working differently depending on different people's systems.

In the interests of informing everyone that is now currently involved in this discussion [and because I feel a bit responsible for what I'm interpreting as a bit of hostility creeping into this discussion {Did not mean for my obsessive questions to cause that. Sorry }] 

From what tests I've been able to run today, it seems that i have a completely independent problem from what Tonka Crash is reporting, even though the symptoms are basically the same. Even with all of my mods that require toolbar controller, unblur, and clickthroughblocker removed, and those 3 dependencies removed, I'm still seeing the lag/stutter and camera freezing that Tonka had originally reported. I can only conclude, at this point that I have another mod, or combination of mods that are not Linuxgurugamer's mods that is causing a similar conflict within KSP. My line of investigation seems to have been a wild goose chase.

I did however, ironically discover that unblur causes conflicts with a few of my other mods, which causes their ingame icons to be blurry and not display properly. 

 

 

[Jognt]

My apologies to @Tonka Crash. It was not my intention to doubt the existence of the lag. I experience it myself as well after playing for a few hours. I always attributed it to "Oh well, KSP" but admit I have not done actual testing with/without MemGraph.

Since @vardicd experiences something similar without memgraph I am wondering what type of code could cause this. I'm used to restarting KSP once or twice to it's not a big deal for me personally, but I can understand it being very annoying.

I did some reading about Unity's GC by the way and I think the values in padheap.cfg are related to spawning objects to manually increase heap space. Apparently different sized objects/heapsizes get treated differently by Unity.

[Tonka Crash]

@vardicd Have you identified a minimum set of mods to create the problem? 

It's tedious to spend hours of time I'd rather be playing compulsively starting KSP over and over to test different combinations of mods to narrow it down to a minimum set that creates a problem. After coming up with MemGraph as a likely candidate. I then spent another day playing with the MemGraph code trying to understand what it was doing and wrote up a critique on some aspects I found odd.

It's incredibly frustrating to dump a dozen hours of my time into trying to identify a problem in someone else's code and then to be dismissed as having a poor attitude.

[Tonka Crash]

@vardicd To put an end to the speculation on dependencies I repeated the test on 1.7.3 with the 1.1.6 version of MemGraph before the toolbar was added. This should still be a valid test as the bulk of the code is otherwise the same from the version where I first reported the problem. The issue I'm seeing is unchanged. I can't reduce the list of mods any further as it is now just KIS, Module Manager and Memgraph. I've also run it with a locally compiled version incorporating the latest updates to MemGraph, but with the toolbar removed, also no change to the behavior.

The spoiler is from the KSP.Log showing the list of mods installed that can reproduce this problem. Nothing else has changed in my test methodology. Every time I start KSP it's a new game to rule out something being carried over in a save file. The response is just like the video. The MemGraph settings.cfg and padheap.cfg are the same as above as they were copied from my normal install. This morning I tested with the total padheap set at 2048, 4096 and 8192. The behavior is exactly the same as the video, but as I've mentioned the lag is a function of the total padheap. The smaller the heap, the shorter the lag. The tradeoff is I pay for it in my real game where a small padheap leads to more frequent GC stutter in a realistic flight scene.

At 2048 the GC interval is long enough I never see a GC cycle while on the pad during these tests. It runs as the flight scene loads, but then there's no need for one during a typical test. As I was typing this post, it finally did a GC: 1209s. For these tests there are only 4 parts in the scene: lander can, Kerbal, Screwdriver and a KIS ground block to move around.

Spoiler

Environment Info
Win32NT 7FFFFFFFFFFFFFFF  Args: KSP_x64.exe -popupwindow

Mod DLLs found:
Stock assembly: Assembly-CSharp v0.0.0.0
ModuleManager v4.0.2.0
KIS v1.22.7117.19606 / v1.22 for KSP 1.7+
KSPDev_Utils.1.2 v1.2.7031.33522 / v1.2 for KSP v1.6+
MemGraph v1.1.1.6

Folders and files in GameData:
KIS
MemGraph
Stock folder: Squad

ModuleManager.4.0.2.dll
ModuleManager.ConfigCache
ModuleManager.ConfigSHA
ModuleManager.Physics
ModuleManager.TechTree

 

@linuxgurugamer  I'm just reporting my observations for the peanut gallery. At this point I don't expect you to find a fix for this problem. 

[linuxgurugamer]

36 minutes ago, Tonka Crash said:

@linuxgurugamer  I'm just reporting my observations for the peanut gallery. At this point I don't expect you to find a fix for this problem. 

Not a problem, maybe someone will have an idea.

Speaking of which, I had one:

As I understand it, you use Memgraph to pad the heap only.  How about if I come up with a minimal HeapPadder mod, which will only pad the heap and do nothing else.  Would you be willing to help test that if/when I get it done?

[Tonka Crash]

2 minutes ago, linuxgurugamer said:

Not a problem, maybe someone will have an idea.

Speaking of which, I had one:

As I understand it, you use Memgraph to pad the heap only.  How about if I come up with a minimal HeapPadder mod, which will only pad the heap and do nothing else.  Would you be willing to help test that if/when I get it done?

No problem.

[Jognt]

3 hours ago, linuxgurugamer said:

Not a problem, maybe someone will have an idea.

Speaking of which, I had one:

As I understand it, you use Memgraph to pad the heap only.  How about if I come up with a minimal HeapPadder mod, which will only pad the heap and do nothing else.  Would you be willing to help test that if/when I get it done?

I dont know what .NET ksp runs on, but the Unity GC article mentioned an experimental GC routine that can be enabled for those wanting to play around with it. 

Might be be an idea to look at that? Looks like it GC’s over time to eliminate stutters completely by checking objects on the fly in the ‘left over’ time of each frames (so let’s say there’s 16ms until next refresh and the frame is done after 10, it’ll spend 6ms on GC.)

Source: https://docs.unity3d.com/Manual/UnderstandingAutomaticMemoryManagement.html#incremental_gc

[Tonka Crash]

@Jognt KSP is .NET 3.5 on Unity 2017.1.3p1 - The incremental garbage collector is for Unity 2019, so not available to us.

[vardicd]

7 hours ago, Tonka Crash said:

@vardicd Have you identified a minimum set of mods to create the problem? 

It's tedious to spend hours of time I'd rather be playing compulsively starting KSP over and over to test different combinations of mods to narrow it down to a minimum set that creates a problem. After coming up with MemGraph as a likely candidate. I then spent another day playing with the MemGraph code trying to understand what it was doing and wrote up a critique on some aspects I found odd.

It's incredibly frustrating to dump a dozen hours of my time into trying to identify a problem in someone else's code and then to be dismissed as having a poor attitude.

I have not at this point managed to discover a minimum set of mods to reproduce my issue. Frustratingly enough it seems as if multiple configurations of mods can trigger it, thus I'm having trouble pinning it down. 

On one install for example i can load mods: A B C and be fine, load mod D and poof lag/stutter. I suspect Mod D is the problem, so start a new install, Start with mod D, no issue. so one at a time i add in A B and C, with all 4 mods installed, no lag/stutter. I go back to my first install and the issue is still there. I remove mod D in install 1, to test, problem remains, even though it wasn't there before Mod D was installed in that version of game. Start a 3rd clean install, put mods A B C D E and F into the install before the problem shows up, so start a new install and begin loading mods starting with mod F, get all 6 mods loaded into the 4th version, no lag/stutter, but the problem remains on install 3 with the exact same mods loaded.

[jpinard]

On 7/13/2019 at 6:40 PM, Jognt said:

It depends on what you want to use memgraph for. 

If you’re using it to reduce GC intervals and the default padding amount doesn’t get you in ram-usage trouble then you can just open the settings.cfg, set visible to false and applypadding to true, and never look back as it does it’s thing. 

I recommend you keep at _least_ 10-15% of your total ram free so windows has room to shuffle stuff around. 

@linuxgurugamer @Jognt

Don't I still need to go into the padheap.cfg and set some paramaters?  or in test.cfg?  

 I'd expected to go into padheap.cfg and manually input 4096 so my system sets 4 Gig of RAM available to GC, but I see there's a collection of numbers, not just one setting.  Or do I change the test.cfg from size=984 to size=4096?

Edited July 15, 2019 by jpinard

[linuxgurugamer]

All,

I have a new mod to address any issues here with memgraph, it's in a closed beta right now, will provide more details as the testing proceeds.

[linuxgurugamer]

5 hours ago, jpinard said:

@linuxgurugamer @Jognt

Don't I still need to go into the padheap.cfg and set some paramaters?  or in test.cfg?  

 I'd expected to go into padheap.cfg and manually input 4096 so my system sets 4 Gig of RAM available to GC, but I see there's a collection of numbers, not just one setting.  Or do I change the test.cfg from size=984 to size=4096?

The left column is the size in K for that block
The right column is the weight for that block, essentially think of it as the right column specifies how many of the left column to allocate before going to the next larger size.

After some minimal testing, try using the following values for your game:

8		: 1
16		: 1
24		: 1
32		: 1
40		: 1
48		: 1
64		: 1
80		: 1
96		: 1
112		: 1
144		: 1
176		: 1
208		: 0
240		: 0
296		: 0
352		: 0
432		: 0
664		: 0
800		: 0
1008	: 0
1344	: 0
2032	: 0
total	: 4096

 

[linuxgurugamer]

16 hours ago, vardicd said:

I have not at this point managed to discover a minimum set of mods to reproduce my issue. Frustratingly enough it seems as if multiple configurations of mods can trigger it, thus I'm having trouble pinning it down. 

On one install for example i can load mods: A B C and be fine, load mod D and poof lag/stutter. I suspect Mod D is the problem, so start a new install, Start with mod D, no issue. so one at a time i add in A B and C, with all 4 mods installed, no lag/stutter. I go back to my first install and the issue is still there. I remove mod D in install 1, to test, problem remains, even though it wasn't there before Mod D was installed in that version of game. Start a 3rd clean install, put mods A B C D E and F into the install before the problem shows up, so start a new install and begin loading mods starting with mod F, get all 6 mods loaded into the 4th version, no lag/stutter, but the problem remains on install 3 with the exact same mods loaded.

I wonder if it has anything to do with the order in which they were first loaded in the save file.

What happens if you copy the save file from the 4th version into the 3rd version and start it there?

[vardicd]

3 hours ago, linuxgurugamer said:

I wonder if it has anything to do with the order in which they were first loaded in the save file.

What happens if you copy the save file from the 4th version into the 3rd version and start it there?

Had not considered such a thing, will test when able.

[Jognt]

22 hours ago, jpinard said:

@linuxgurugamer @Jognt

Don't I still need to go into the padheap.cfg and set some paramaters?  or in test.cfg?  

 I'd expected to go into padheap.cfg and manually input 4096 so my system sets 4 Gig of RAM available to GC, but I see there's a collection of numbers, not just one setting.  Or do I change the test.cfg from size=984 to size=4096?

The “Total: “ amount can be user-adjusted. I recommend only touching the other numbers if LLG recommends it or if you know what you’re doing (or are just _very_ curious)

[vardicd]

16 hours ago, linuxgurugamer said:

I wonder if it has anything to do with the order in which they were first loaded in the save file.

What happens if you copy the save file from the 4th version into the 3rd version and start it there?

12 hours ago, vardicd said:

Had not considered such a thing, will test when able.

So here's an extra layer of weirdness to this story. So if I transfer the save file from one install that does NOT have the glitch to an identical install with a save file that's displaying the glitch, the save file will become glitched, but how long it takes is completely variable. The fastest it's happened so far is about 5 real world minutes, the longest it's taken to happen is about 20~ish real world minutes. If I transfer the save file from a install with the glitch, to an install that does not have the glitch, it's a 50/50 split on if the glitch will persist, or clear up, and if it persists, all new saves on that previously clear install have the glitch.

It seems to me, admittedly I don't know how it works, almost as if there's some sequence of events that are firing {sometimes? randomly? not always?} between my mods and if this random confluence happens it causes the lag/stutter, but if it doesn't happen the install stays clear, unless i introduce a save where it's happened, then the install acts as if the events fired, or it causes the specific event to happen, "corrupting" the install.

[Tonka Crash]

@vardicd I always test mod interaction problems in sandbox with a new save every single time I launch with a different combination of mods just to avoid weirdness creeping into tests through an old save with different mods. It's the only way to guarantee the test conditions are both known and repeatable.

[linuxgurugamer]

Ok, I now have a public beta for the new mod:

HeapPadder - Pads the heap to minimize the garbage collection

https://github.com/linuxgurugamer/HeapPadder/releases/tag/0.0.1.2

HeapPadder

1. This mod is designed to force Mono to keep significantly more free space in the heap, which can significantly reduce the frequency at which the heap fills up and the Mono garbage collection causes a stutter,

 Installation

1. Copy the HeapPadder folder from the zip file into the GameData folder of your KSP installation.

Regarding the padheap.cfg file:

1. If the default settings aren't working well for you, try the following steps:
2. Increase the padheap size in your install: \GameData\MemGraph\PluginData\MemGraph\padheap.cfg, change the "total" number to 4096 or even 6144. See the next section for an example
3. For example, if you have 16 GB:
   1. Look at your task manager to find out how much free ram you have when the game is running. 
   2. Do this calculation 16 Gb - ~5/6 GB from KSP - 1/2 GB Windows = 8 GB free
   3. This is the amount of RAM you can use for your padheap total size (in MegaByte 4GB=4096MB).  
4.  Always have some Ram to spare of course. 

Configuration

The mod has a default config file, in the Plugins folder called:  default_padheap.cfg.  This is needed only if there isn't a config file present, the config file should be placed into the same folder and is called: padheap.cfg.  If the config file is not present, then the default_padheap.cfg is copied to padheap.cfg.  This done so that any updates to the mod  will not overwrite any local changes to the config file

While this can be used with Memgraph, you risk doubling the memory usage if you allocation memory with Memgraph as well as HeapPadder.  I am planning on releasing an update to Memgraph, so that the Memgraph padding code is disabled if HeapPadder is installed.

HeapPadder has no UI.

The code is released under the MIT license (see https://github.com/linuxgurugamer/heapPadder/blob/master/Graph.cs).

 

Edited July 17, 2019 by linuxgurugamer

[gamerscircle]

I am going to download this and test it on my current save.  I have a stutter that I can't seem to nail down as to which mod it might be.  From the reading, it seems that this might help some.

[linuxgurugamer]

34 minutes ago, gamerscircle said:

I am going to download this and test it on my current save.  I have a stutter that I can't seem to nail down as to which mod it might be.  From the reading, it seems that this might help some.

@gamerscircle There is a small problem which you need to fix before using this (everyone else as well)

In the file(s) in the PluginData directory, the default_padheap.cfg has some comment lines, you MUST delete them before using the mod.  If you've already started the game one time, then you MUST delete those lines from the padheap.cfg as well.

Working on a fix now

Beta updated with correct default_padheap.cfg

https://github.com/linuxgurugamer/HeapPadder/releases/tag/0.0.1.3

[Jognt]

3 hours ago, linuxgurugamer said:

Ok, I now have a public beta for the new mod:

HeapPadder - Pads the heap to minimize the garbage collection

https://github.com/linuxgurugamer/HeapPadder/releases/tag/0.0.1.2

HeapPadder

1. This mod is designed to force Mono to keep significantly more free space in the heap, which can significantly reduce the frequency at which the heap fills up and the Mono garbage collection causes a stutter,

 Installation

1. Copy the HeapPadder folder from the zip file into the GameData folder of your KSP installation.

Regarding the padheap.cfg file:

1. If the default settings aren't working well for you, try the following steps:
2. Increase the padheap size in your install: \GameData\MemGraph\PluginData\MemGraph\padheap.cfg, change the "total" number to 4096 or even 6144. See the next section for an example
3. For example, if you have 16 GB:
   1. Look at your task manager to find out how much free ram you have when the game is running. 
   2. Do this calculation 16 Gb - ~5/6 GB from KSP - 1/2 GB Windows = 8 GB free
   3. This is the amount of RAM you can use for your padheap total size (in MegaByte 4GB=4096MB).  
4.  Always have some Ram to spare of course. 

Configuration

The mod has a default config file, in the Plugins folder called:  default_padheap.cfg.  This is needed only if there isn't a config file present, the config file should be placed into the same folder and is called: padheap.cfg.  If the config file is not present, then the default_padheap.cfg is copied to padheap.cfg.  This done so that any updates to the mod  will not overwrite any local changes to the config file

While this can be used with Memgraph, you risk doubling the memory usage if you allocation memory with Memgraph as well as HeapPadder.  I am planning on releasing an update to Memgraph, so that the Memgraph padding code is disabled if HeapPadder is installed.

HeapPadder has no UI.

The code is released under the MIT license (see https://github.com/linuxgurugamer/heapPadder/blob/master/Graph.cs).

 

Nice! Is this identical to MemGraph but without the GUI? Or does it do it differently? (Better?)

[linuxgurugamer]

23 minutes ago, Jognt said:

 

Nice! Is this identical to MemGraph but without the GUI? Or does it do it differently? (Better?)

It's  just the padding, no gui

[vardicd]

10 hours ago, Tonka Crash said:

@vardicd I always test mod interaction problems in sandbox with a new save every single time I launch with a different combination of mods just to avoid weirdness creeping into tests through an old save with different mods. It's the only way to guarantee the test conditions are both known and repeatable.

yeah i do too, but in this particular case linuxgurugamer had specifically asked what happened if i transfered saves around.

[linuxgurugamer]

7 hours ago, gamerscircle said:

I am going to download this and test it on my current save.  I have a stutter that I can't seem to nail down as to which mod it might be.  From the reading, it seems that this might help some.

@gamerscircle Here is a config which should be reasonable for a 32 gig system, this will pad the heap with 8 gig of memory.  If you want to try 16, then just replace the 8192 with 16384 on the last line.

Put this into a file:  HeapPadder/PluginData/padheap.cfg

Do NOT delete the file:  HeapPadder/PluginData/default_padheap.cfg

8	  	: 1
16		: 1
24		: 1
32		: 1
40		: 1
48		: 1
64		: 1
80		: 1
96		: 1
112		: 1
144		: 1
176		: 1
208		: 1
240		: 0
296		: 0
352		: 0
432		: 0
664		: 0
800		: 0
1008	: 0
1344	: 0
2032	: 0
total	: 8192

 

Edited July 18, 2019 by linuxgurugamer