[1.12.x] MemGraph Updated with Stutter Reduction

[Tonka Crash]

@linuxgurugamer I've tracked down a performance glitch I've been seeing to MemGraph or one of its dependencies. (ClickThroughBlocker ToolbarControl UnBlur). 

Problem Synopsis: Logs 

Using KIS/KAS I see a lag or stutter placing objects or aborting out of placing objects with KIS. I don't know when the behavior started as it been there long enough it just seemed to be the norm to me. I mentioned it to @IgorZ yesterday and he said it's not normal. I checked a stripped install and sure enough, KIS is much more responsive. So I began the mod interaction hunt.

I narrowed it down to MemGraph or one of it's dependencies. With Memgraph installed I see a lag or stutter with KIS. No Memgraph, no lag. When I tested without Memgraph I left the dependencies installed, but nothing else should be calling them.

Video:

In the video below I drop an object while moving the camera dragging the mouse across the screen. With MemGraph there is a noticeable lag before the game catches up to the mouse. The second half of the video is how it should behave doing the same operation. I can barely move the mouse before the game catches up.

I forgot to look at the MemGraph stripchart until I was in the middle of writing this, well after logs and video had been produced and uploaded. This lag is not due to garbage collection. 

Memgraph is one of my must have mods, so it would be nice to figure out what's causing this. I'd like to test a stripped down MemGraph without a GUI or dependencies to see if that improves performance. I rarely look at the GUI anyway since I have a padheap.cfg that seems to work for me. 

Test Configuration: Latest versions installed from CKAN in my main game, copied to the stripped install path.

• KSP 1.7.2 - No expansion packs
• ClickThroughBlocker
• ToolbarControl
• Easy Vessel Switch 
• KIS & KAS
• MemGraph padheap.cfg configured for 8GB with 32GB of RAM.
• ModuleManager.4.0.2.dll
• unBlur.0.5.0.dll

[Tonka Crash]

@linuxgurugamer @IgorZ I played with this for hours yesterday. What I found is the lag seems to depend on the total pad size you have defined in padheap.cfg. I have been running 8192MB with 32GB of ram which was giving several minutes between GC cycles in my normally modded game. Getting rid of padhead.cfg entirely and KIS behaves instantaneously as if MemGraph weren't installed but then garbage collection happens several times a minute in my modded game.

linuxgurugamer, since this is one of your adopted mods I don't know if you've ever spent time trying to understand how it works. I downloaded the code and experimented with it trying to understand how it worked and what would cause a hiccup with KIS. The only thing I do understand is that the majority of graph.cs is only needed for the graph and is called more frequently than needed while the game is running. The actually padding from padhead.cfg happens once on game load and never again unless the user manually requests a pad size increase.

I still don't understand how the padding works at all or the significance of the majority of padheap.cfg. It seems that only the final line defines the total pad size, but the other rows are used in the padding algorithm. I just can't make any sense of what they are supposed to be doing. I don't understand the logic of padarray function at all. Do the actual numbers set in the first column have some special significance? I have zero experience with languages that use garbage collection.

General comment: None of the MemGraph log messages identify themselves with "[MemGraph]" to start the message. The log messages functionality is not typical of most mods and doesn't make use of UnityEngine.Debug.Log method. Comments claim this is to reduce garbage collection, but there are so few message any impact would be negligible.

By the way, graph.cs itself is rather poorly thought out, but changes to it didn't affect my issue: 

• It appears the data for the graph is updated in AddMemoryIncrement, but this is triggered by both Update and FixedUpdate events where I think it should be one or the other.
• AddMemoryIncrement itself does about half it's work in an if block limited to once per second to match the graph's update rate, but what I would think are the more expensive calls are outside that time limited block.
• UpdateGuiStr that updates the graph's header seems to be called anytime one of it's values is recalculated, but could done only if the graph is visible in OnGui.
• The settings file (but not padheap) are re-read on scene changes. It seems like an odd behavior as I (and I'd think most users) would assume .cfg file changes would require a game restart to take effect.

[linuxgurugamer]

1 hour ago, Tonka Crash said:

The settings file (but not padheap) are re-read on scene changes. It seems like an odd behavior as I (and I'd think most users) would assume .cfg file changes would require a game restart to take effect.

I believe it was designed that way on purpose, so that changes could be made and tested without restarting the entire game.

I don't have time right now, but will put this on my (ever enlarging ) list of things to do.  Some of the oddities you are seeing are due to memory management and padding.  I know I looked at it a while ago, some of what you are pointing out could definitely be improved

[linuxgurugamer]

On 6/20/2019 at 10:30 AM, Tonka Crash said:

It appears the data for the graph is updated in AddMemoryIncrement, but this is triggered by both Update and FixedUpdate events where I think it should be one or the other.

Fixed in upcoming release

On 6/20/2019 at 10:30 AM, Tonka Crash said:

AddMemoryIncrement itself does about half it's work in an if block limited to once per second to match the graph's update rate, but what I would think are the more expensive calls are outside that time limited block.

It's necessary to do it this way, those "expensive" calls are getting the best information, and aren't that expensive

On 6/20/2019 at 10:30 AM, Tonka Crash said:

UpdateGuiStr that updates the graph's header seems to be called anytime one of it's values is recalculated, but could done only if the graph is visible in OnGui.

Fixed in upcoming release

Edited July 6, 2019 by linuxgurugamer

[linuxgurugamer]

On 6/20/2019 at 10:30 AM, Tonka Crash said:

General comment: None of the MemGraph log messages identify themselves with "[MemGraph]" to start the message. The log messages functionality is not typical of most mods and doesn't make use of UnityEngine.Debug.Log method. Comments claim this is to reduce garbage collection, but there are so few message any impact would be negligible.

Memgraph is writing to it's own log file, no need to identify the messages since no other mod will write to it.

Since the whole idea of this mod is to report on garbage collection, I'm not going to change something "just because" it's not using Debug.Log.

Memgraph literally has one call to Debug.Log, at the end of the InitToolbar, which I'm removing since it is rather unneeded

New release, 1.1.2

• Disabled call to AddMemoryIncrement from the Update method
• Split the UpdateGuiStr into two, new one called WriteLog()
• Added a call to UpdateGuiStr just before the generated string is written
• Removed unnecessary Debug.Log line at the end of the InitToolbar method
• Updated the AssemblyVersion.tt

[tsaven]

Thank you so much for the continued updates and support

[vossiewulf]

Ditto, without this the GC delays are horrendous. One question, when I hit alt-End to increase the heap, it goes from 512MB min to 1500MB min. One that's a weird number for mem allocation, and two, hitting alt-End after that doesn't seem to do anything. The mod description implies you can progressively increase the heap allocation.

[Jognt]

2 hours ago, vossiewulf said:

Ditto, without this the GC delays are horrendous. One question, when I hit alt-End to increase the heap, it goes from 512MB min to 1500MB min. One that's a weird number for mem allocation, and two, hitting alt-End after that doesn't seem to do anything. The mod description implies you can progressively increase the heap allocation.

I'm guessing it says to tap it twice to make sure the button press is registered. Just in case it didn't work the first time.

By the way, you can set it to automatically apply the padding and hide the window by default so you don't have to think about it anymore by opening settings.cfg and setting ApplyPadding to true and Visible to false.

The default padding is set to 1GB. You can modify this number in the padheap.cfg by adjusting the final number after total: to a higher or lower value. Be sure not to go for crazy numbers that you don't have the spare RAM for. (I adjusted it down to 512 due to ram concerns)

Edited July 12, 2019 by Jognt

[vossiewulf]

1 hour ago, Jognt said:

I'm guessing it says to tap it twice to make sure the button press is registered. Just in case it didn't work the first time.

By the way, you can set it to automatically apply the padding and hide the window by default so you don't have to think about it anymore by opening settings.cfg and setting ApplyPadding to true and Visible to false.

The default padding is set to 1GB. You can modify this number in the padheap.cfg by adjusting the final number after total: to a higher or lower value. Be sure not to go for crazy numbers that you don't have the spare RAM for. (I adjusted it down to 512 due to ram concerns)

Thanks Jognt for the answer and the additional useful info. Will be nice to have it instanced correctly without having to intervene. Although I do like sometimes watching the endless GCs go away once I pad the heap. I might increase it a bit more, but as you know there's a limit to  the utility of doing so even if you have the mem.

Edited July 12, 2019 by vossiewulf

[jpinard]

I'm a little confused on just how exactly this works.  Once I install it, do I need to do anything after it examines how often garbage collection is happening?  Or later does it do it automatically or do I need to make changed based on what the graph shows?

[vardicd]

On 6/18/2019 at 10:44 AM, Tonka Crash said:

@linuxgurugamer I've tracked down a performance glitch I've been seeing to MemGraph or one of its dependencies. (ClickThroughBlocker ToolbarControl UnBlur). 

Problem Synopsis: Logs 

Using KIS/KAS I see a lag or stutter placing objects or aborting out of placing objects with KIS. I don't know when the behavior started as it been there long enough it just seemed to be the norm to me. I mentioned it to @IgorZ yesterday and he said it's not normal. I checked a stripped install and sure enough, KIS is much more responsive. So I began the mod interaction hunt.

I narrowed it down to MemGraph or one of it's dependencies. With Memgraph installed I see a lag or stutter with KIS. No Memgraph, no lag. When I tested without Memgraph I left the dependencies installed, but nothing else should be calling them.

Video:

In the video below I drop an object while moving the camera dragging the mouse across the screen. With MemGraph there is a noticeable lag before the game catches up to the mouse. The second half of the video is how it should behave doing the same operation. I can barely move the mouse before the game catches up.

I forgot to look at the MemGraph stripchart until I was in the middle of writing this, well after logs and video had been produced and uploaded. This lag is not due to garbage collection. 

Memgraph is one of my must have mods, so it would be nice to figure out what's causing this. I'd like to test a stripped down MemGraph without a GUI or dependencies to see if that improves performance. I rarely look at the GUI anyway since I have a padheap.cfg that seems to work for me. 

Test Configuration: Latest versions installed from CKAN in my main game, copied to the stripped install path.

• KSP 1.7.2 - No expansion packs
• ClickThroughBlocker
• ToolbarControl
• Easy Vessel Switch 
• KIS & KAS
• MemGraph padheap.cfg configured for 8GB with 32GB of RAM.
• ModuleManager.4.0.2.dll
• unBlur.0.5.0.dll

Late to the party, but i also see this camera lag behavior, and i don't use memgraph myself, though i do use the dependencies, clickthroughblocker, unblur, and toolbar control, as well as the easy vessel switch mod. I'm curious if you still see this issue. I've had this issue for a while, but never gave it a thought till today when i found this mod and got curious as to what it was about and saw your post.

Edited July 13, 2019 by vardicd

[Tonka Crash]

38 minutes ago, vardicd said:

Late to the party, but i also see this camera lag behavior, and i don't use memgraph myself, though i do use the dependencies, clickthroughblocker, unblur, and toolbar control, as well as the easy vessel switch mod. I'm curious if you still see this issue. I've had this issue for a while, but never gave it a thought till today when i found this mod and got curious as to what it was about and saw your post.

Nothing significant has changed since I first posted my observation, and I'm still seeing the lag. 

[Jognt]

4 hours ago, jpinard said:

I'm a little confused on just how exactly this works.  Once I install it, do I need to do anything after it examines how often garbage collection is happening?  Or later does it do it automatically or do I need to make changed based on what the graph shows?

It depends on what you want to use memgraph for. 

If you’re using it to reduce GC intervals and the default padding amount doesn’t get you in ram-usage trouble then you can just open the settings.cfg, set visible to false and applypadding to true, and never look back as it does it’s thing. 

I recommend you keep at _least_ 10-15% of your total ram free so windows has room to shuffle stuff around. 

Edited July 13, 2019 by Jognt

[linuxgurugamer]

3 hours ago, Tonka Crash said:

Nothing significant has changed since I first posted my observation, and I'm still seeing the lag. 

If you turd visible to off as described above, do you still get the stuttering?

[Tonka Crash]

13 minutes ago, linuxgurugamer said:

If you turd visible to off as described above, do you still get the stuttering?

The GUi is off by default in my config. I don't like clutter on my desktop. If the GC interval was saved in the KSP.Log I could do without a GUI entirely. I rarely have the GUI open unless the game seems to be bogging down. I set the total in padheap.cfg and ignore it. 

This is on a slightly overclocked Hades Canyon NUC i7-8809G @ 4.2GHz with 32GB Ram & NVMe Samsung 960 EVO SSD.

Currently my system reports 26GB in use (4GB) Compressed 5.9GB available. KSP has been open for almost exactly 11 hours.

settings.cfg:

visible = false
applyPadding = true
keyToggleWindow = KeypadMultiply
keyScaleUp = KeypadPlus
keyScaleDown = KeypadMinus
keyRunTests = KeypadDivide
keyPadHeap = End
enableLogging = false
keyMark = Home
keyToggleLogging = PageUp

padheap.cfg:  I've experimented with other totals (1024,2048,4096). Lag is related to the amount of padding, but reducing padding gets more frequent GC stutter. I usually have it set at 4096. I must have left it at 8192 after the last time I messed with this. I don't really understand what the other lines are supposed to be doing. The total is the only one that really seems to matter,  but from my experimentation they do have an affect in the code, I just don't understand what it's doing. I've tried mostly off, and currently have them all on. I'm just not sure if it matters.

Spoiler

8    : 1
16    : 1
24    : 1
32    : 1
40    : 1
48    : 1
64    : 1
80    : 1
96    : 1
112    : 1
144    : 1
176    : 1
208    : 1
240    : 1
296    : 1
352    : 1
432    : 1
664    : 1
800    : 1
1008    : 1
1344    : 1
2032    : 1
total    : 8192
 

[jpinard]

1 hour ago, Jognt said:

If you’re using it to reduce GC intervals and the default padding amount doesn’t get you in ram-usage trouble then you can just open the settings.cfg, set visible to false and applypadding to true, and never look back as it does it’s thing. 

I recommend you keep at _least_ 10-15% of your total ram free so windows has room to shuffle stuff around. 

This is exactly what I needed.  Thanks!

[vardicd]

5 hours ago, Tonka Crash said:

Nothing significant has changed since I first posted my observation, and I'm still seeing the lag. 

Okay, that makes me wonder, as i said i see a similar lag issue on my games, but i don't use memgraph, only found it today, and have never installed it. I'm going to start running tests of my own tonight, to see if i can narrow down the source of my lag/stutter, and thought I'd ask if you'd considered or tested the possibility of your issue coming from the dependencies of the mod, rather than the mod itself?

I'm curious if we've got 2 different issues causing the same symptoms, or if we have the same issue, but it's coming from one of the dependencies doing something in the background?

[Tonka Crash]

@vardicd When I narrowed it down to memgraph I tested with no memgraph but all it's dependencies and only KIS and the hiccups were gone. You can't run memgraph without it's dependencies as it won't load. The only way to do that is by editing code to remove the dependencies.

[vardicd]

3 hours ago, Tonka Crash said:

@vardicd When I narrowed it down to memgraph I tested with no memgraph but all it's dependencies and only KIS and the hiccups were gone. You can't run memgraph without it's dependencies as it won't load. The only way to do that is by editing code to remove the dependencies.

Okay, one final question for you on this subject, {and sorry if i'm being annoying } when you did your test without memgraph, did you think to include any other mod that needed the dependencies [that would cause them to do anything?] I ask because my first test I ran was to copy my main game folder and just rip out toolbar controller, unblur, and clickthroughblocker, thereby shutting down any mod that needed those dependencies, and I immediately saw a performance increase, but I wasn't sure if that was just because a number of mods aren't functioning any more and my system isn't working as hard, or if the removed dependencies where causing trouble. I'm currently installing a fresh copy of the game, and going to build a minimum install of KIS/KAS, toolbar controller, unblur, and clickthroughblocker and one other mod that needs those and see if i get performance issues. 

And again, I'm sorry if i'm coming off as being annoying or beating a dead horse. When i saw your post above, it just triggered a gut feeling of "This is important" and I'm fully self aware that i'm obsessive when i get like that and i have to run every possibility into the ground to convince myself that i'm wrong, and to let it go.  

[Tonka Crash]

@vardicd

When I tested this it was stock squad, no DLC. Only KIS and MemGraph. It was stripped to the minimum to show the problem.

[vardicd]

58 minutes ago, Tonka Crash said:

@vardicd

When I tested this it was stock squad, no DLC. Only KIS and MemGraph. It was stripped to the minimum to show the problem.

Okay, thank you

 

[Jognt]

7 hours ago, vardicd said:

Okay, one final question for you on this subject, {and sorry if i'm being annoying } when you did your test without memgraph, did you think to include any other mod that needed the dependencies [that would cause them to do anything?] I ask because my first test I ran was to copy my main game folder and just rip out toolbar controller, unblur, and clickthroughblocker, thereby shutting down any mod that needed those dependencies, and I immediately saw a performance increase, but I wasn't sure if that was just because a number of mods aren't functioning any more and my system isn't working as hard, or if the removed dependencies where causing trouble. I'm currently installing a fresh copy of the game, and going to build a minimum install of KIS/KAS, toolbar controller, unblur, and clickthroughblocker and one other mod that needs those and see if i get performance issues. 

And again, I'm sorry if i'm coming off as being annoying or beating a dead horse. When i saw your post above, it just triggered a gut feeling of "This is important" and I'm fully self aware that i'm obsessive when i get like that and i have to run every possibility into the ground to convince myself that i'm wrong, and to let it go.  

Interesting train of thought. What it boils down to is whether it's memgraph specific or due to one (or a combination) of its dependencies.

@Tonka Crash Keep in mind that memgraph was primarily designed to diagnose GC issues. You still have to deal with any Mono peculiarities even when increasing the padheap (and maybe even exactly when increasing the padheap)

How often do you get GC with 4GB of padheap? Because you really are pumping up those numbers which makes me wonder whether you're using it to mask excessive Garbage Creation. Like, how high does "Last" usually go in the Space Center/Flight Scene?

Edited July 14, 2019 by Jognt

[Tonka Crash]

@Jognt Your questions are completely irrelevant to the problem I posted. Watch the video. Or even better, test it yourself. I'd like someone else to confirm the problem. With MemGraph I get a lag spike dropping items in KIS, without it, I don't. It's that simple. This is in a stripped install that doesn't at all represent my normal playing environment.  

I don't care what MemGraph was "designed" to do. What it's being used for is to spread out GC intervals, which is variable depending  on what you are doing. I also don't keep track of what MemGraph is doing. As I've stated before I don't have MemGraph open by default and rarely look at it at all. As long as GC stutter is infrequent enough to not annoy me I don't care what the settings are at.

Here's a question I asked in January that went unanswered.

 

[Jognt]

3 hours ago, Tonka Crash said:

[snip]

Did you miss the part where @vardicd said he has the same problem even though he doesn’t use memgraph?

i understand it’s easy to yell “your mod is broken”, but if you really want to get this sorted I suggest you either use the info and questions provided to help find the cause or to just stop using any mods. 

With regards to “I don’t care what it was designed to do” -> you’re using a mod for a purpose it was not (mainly) intended. It doesn’t matter that all of us use it for that purpose. 

[snip]

 

Edited July 14, 2019 by Gargamel
Portions Redacted by Moderator

[linuxgurugamer]

I don't have an answer as to why that happen. I've updated the mogg recently to remove some inefficiencies, but regarding U the stuttering I have no idea because nothing I saw when I was looking at it seemed to indicate that that would cause it.

This is very hard to diagnose because it's working differently depending on different people's systems.