[1.2.x/1.3] MemGraph 1.1.0.3 - with Stutter Reduction

[DeadOnDuna]

2 hours ago, damerell said:

The best you can do is to ensure your swap space (Linux) / page file (Windows) is on the SSD. That's used when physical memory is exhausted.

Edited to insert correct quote.

already done it. I've set  my paging file to 8GB manualy ... works for other applications but i didn't noticed any difference  on ksp ...

Ksp loads everything in ram and if you fill your ram and you start "leaking"  to sdd the game becomes unresponsive and eventually crashes (maybe it's just me and my potato pc)

thanx for your time @damerell

 

[damerell]

3 hours ago, DeadOnDuna said:

Ksp loads everything in ram and if you fill your ram and you start "leaking"  to sdd the game becomes unresponsive and eventually crashes (maybe it's just me and my potato pc) @damerell

Applications don't in general choose whether to use RAM or paging. They request memory from the OS and it's up to the OS whether any given bit of allocation is presently in physical memory or swapped out. You have a problem when the working set (the amount of this allocated memory which is being used at any given moment in time) is more than the amount of physical memory you have - then the machine "thrashes", constantly transferring stuff between swap and physical memory. Conversely if the working set fits into physical memory you're fine even if large amounts of swap are being used. This is why different applications may allocate similar amounts of memory but behave very differently - do they want to refer to it all at once, or not?

This then makes memory access from the point of view of the application very slow. It can handle this gracefully or not; I infer KSP doesn't handle it gracefully and eventually chokes.

I strongly suspect your "memory cleaner" is snake oil. The best such a thing could do is expunge cached disc pages from memory, which might give a bit of an improvement to new memory allocation shortly thereafter but isn't really going to do you much good in the medium term.

[DeadOnDuna]

29 minutes ago, damerell said:

You have a problem when the working set (the amount of this allocated memory which is being used at any given moment in time) is more than the amount of physical memory you have - then the machine "thrashes", constantly transferring stuff between swap and physical memory. Conversely if the working set fits into physical memory you're fine even if large amounts of swap are being used. This is why different applications may allocate similar amounts of memory but behave very differently - do they want to refer to it all at once, or not?

Correct! I totaly agree!!!

 

29 minutes ago, damerell said:

I infer KSP doesn't handle it gracefully and eventually chokes.

My virtual memory setting is " automatically manage paging file size for all drives " and i have 4 GB of RAM (a joke i know ...) Do you suggest a diffrent setting???(for playing KSP)

 

29 minutes ago, damerell said:

I strongly suspect your "memory cleaner" is snake oil.

I see your point but... when i play without "cleaning my memory", after some reverts in the game, my memory usage goes up to 99% and the game crashes... when i "clean" manualy the game never crashes ... even after 3 hr ...

So i am not sure ... 

 

Edited January 14, 2017 by DeadOnDuna

[DeadOnDuna]

Searching the forum i stumbled upon some posts saying that mechjeb and GC dont work so great together . Is this true?

Will uninstalling MJ improve GC intervals ???

[damerell]

1 hour ago, DeadOnDuna said:

My virtual memory setting is " automatically manage paging file size for all drives " and i have 4 GB of RAM (a joke i know ...) Do you suggest a diffrent setting???(for playing KSP)

Earlier you said "I've set  my paging file to 8GB manualy" (sic), so you seem a little confused here. It would be best (if space permits) to have a paging file only on the SSD.

However, the import of the discussion above is essentially that if KSP is not well behaved there may be nothing much that can be done.

[DeadOnDuna]

 

6 hours ago, DeadOnDuna said:

already done it. I've set  my paging file to 8GB manualy ... works for other applications but i didn't noticed any difference  on ksp ...

I did it ... didn't noticed any difference  on ksp ... so i turned it back to defaults

32 minutes ago, damerell said:

so you seem a little confused here.

Not confused (...ok maybe sometimes :p) just experimenting a lot...

 

33 minutes ago, damerell said:

if KSP is not well behaved there may be nothing much that can be done.

The only thing i can do to improve Ksp performance is to buy a new PC (at least some RAM) but unfortunately this is not an option for me so i try to squeeze everything out of my old rig ...

tanx for your help

[Padishar]

2 hours ago, DeadOnDuna said:

Searching the forum i stumbled upon some posts saying that mechjeb and GC dont work so great together . Is this true?

Will uninstalling MJ improve GC intervals ???

MJ isn't really too bad garbage-wise though a garbage related issue was fixed in a very recent dev build.  All mods that have a lot of old-style UI will generate more garbage than they should but some are a lot worse than others (and mods often cause garbage in non-UI related code too).  Only having those bits of UI showing that you really need should improve things.  Otherwise, I would suggest following all the advice you can find on reducing the memory usage of KSP (use OpenGL and reduce the texture resolution to minimum) to free up as much as possible and then increase the MemGraph heap padding total value as much as you can without the game crashing (or the performance crashing due to VM thrashing).  As KSP's memory usage tends to increase over time, you will need to allow for this, not much point if you can extend the intervals but then only play for half an hour before it dies.

[DeadOnDuna]

17 hours ago, Padishar said:

a garbage related issue was fixed in a very recent dev build.

Thank you  i'll try the dev version of MJ

17 hours ago, Padishar said:

 All mods that have a lot of old-style UI will generate more garbage than they should but some are a lot worse than others (and mods often cause garbage in non-UI related code too).

Maybe a list of "suspicious|" mods would be helpfull but i'll undrstand if you dont want to refer to other mods...

17 hours ago, Padishar said:

Otherwise, I would suggest following all the advice you can find on reducing the memory usage of KSP (use OpenGL and reduce the texture resolution to minimum)

I used OpenGL and low resolution textures with earlier versions of KSP (0.90, 1.0, 1.1) and the result was disappointing  ( i like playing with  EVE , scatterer and so on ... )

Ok i know i expect a lot from my old rig ... but i'll spent the waiting time looking at beautiful clouds and auroras

17 hours ago, Padishar said:

increase the MemGraph heap padding total value as much as you can

3 times alt+End is the sweet spot for me ...

MemGraph made my game playable again (from 7sec to 30+sec intervals is a miracle to me  ) so <3 @Padishar

Edited January 15, 2017 by DeadOnDuna

[Padishar]

37 minutes ago, DeadOnDuna said:

3 times alt+End is the sweet spot for me ...

Hitting it multiple times doesn't really do much unless the game has allocated more long-lived objects since the first time.  The way to increase it is to edit the padhead.cfg file in the MemGraph/PluginData/MemGraph folder and change the value on the total= line.  The value is in MB so you may be able to increase it from the default of 1024 to something like 1200 or even 1500 without causing immediate crashes.  Also, wait until you've loaded your save and gone into the flight scene before hitting Alt-End for maximum benefit.

42 minutes ago, DeadOnDuna said:

MemGraph made my game playable again (from 7sec to 30+sec intervals is a miracle to me  ) so <3 @Padishar

You're welcome, I'm glad it's helping...

[DeadOnDuna]

47 minutes ago, Padishar said:

Hitting it multiple times doesn't really do much

weird i was under the impression that every time i was pressing  alt+End GC interval was increasing ...

 

47 minutes ago, Padishar said:

The value is in MB so you may be able to increase it from the default of 1024 to something like 1200 or even 1500 without causing immediate crashes.

I'll try 2048 ... i like breaking things and spending hours in fixing them

1 hour ago, DeadOnDuna said:

 All mods that have a lot of old-style UI will generate more garbage

maybe you could PM me the list...

much obliged

 

Edited January 15, 2017 by DeadOnDuna

[Raphaello]

There are two options in settings.cfg:

visible = false
applyPadding = false

The first one controls visibility of the window during startup (default is true). I've switched it off as it is not required for me during game launch.

The second one, I guess, allows automatic padding (automatic press of Mod+End) - does it make any sense to set it to true? Based on @Padishar comment above it probably doesn't...

EDIT: Great work on this little mod, thanks a lot!!

Edited January 15, 2017 by Raphaello

[Padishar]

Just now, Raphaello said:

The second one, I guess, allows automatic padding (automatic press of Mod+End) - does it make any sense to set it to true? Based on @Padishar comment above it probably doesn't...

Yes, applyPadding causes the padding to be applied very soon after the game starts (before most of the loading process).  This can help to (slightly) speed up the loading process but a substantial amount of the padding will get clogged up by the data the game loads so it is only really advised if you have lots of memory and have increased the amount of the padding.  You can reapply it at any time but this causes the old padding to be released and new padding to be allocated and this can cause the overall memory usage of KSP to increase so it should be avoided unless really necessary.  It can result in longer intervals but this is because some of the initial padding still serves as padding because the pages are held in memory despite MemGraph "releasing" them due to the core game having allocated some blocks from those pages.  If you have limited memory then you will probably be better off waiting until in the editor or flight scene.

[Jacke]

10 hours ago, DeadOnDuna said:

Thank you  i'll try the dev version of MJ.

I suggest using MJ #682.  #683 has some issues with resetting the layout of MJ windows.  From reading the current MJ topic I'm not sure if that has been fixed in #684.

EDIT: MJ #685 has the issues fixed, so it should be good to go.

 

Edited January 15, 2017 by Jacke

[DeadOnDuna]

I live in a dream...

 

Spoiler

Spoiler

Spoiler

Spoiler

Spoiler

with only 4GB of ram !!!

<3 @Padishar

Edited January 16, 2017 by DeadOnDuna

[ICirceI]

Val sells seashells down by the sea shore. She also only lags every 5 minutes instead of every 5 seconds, thanks to MemGraph.

Spoiler

 

Edited January 19, 2017 by ICirceI

[linuxgurugamer]

To all who identified the issue with the memory leak in Craft Import, thank you for your patience.

It's been fixed, thanks to @HebaruSan.  

I'd appreciate some confirmation that it is fixed.,thanks

Edited January 31, 2017 by linuxgurugamer

[dlrk]

Do I have to open up memgraph and hit alt-end every time I run KSP? Or is it saved between sessions?

[Calvin_Maclure]

Just now, dlrk said:

Do I have to open up memgraph and hit alt-end every time I run KSP? Or is it saved between sessions?

In the options, you can tell it to apply padding on start up. 

1 minute ago, dlrk said:

Do I have to open up memgraph and hit alt-end every time I run KSP? Or is it saved between sessions?

Go to: GameData -> MemGraph -> PluginData -> MemGraph -> Settings -> ''applyPadding = true''

[Helbrecht]

I just got this mod and am already seeing huge improvements in my sessions, I allocated 4gb extra to the heap and its working wonders, I have 12gb overall and am monitering my total usage to fine tune for max longevity of gaming sessions, once again hats off to the creator of this mod its a beautiful thing.

[orcaman]

I want to love this mod, but I just don't understand it. Do we add more heap every time we hit Mod+End or do we configure it using the config file? If the config file, then does Mod-End just toggle it? What on Earth IS that config file, anyway? I've gone back and forth through the thread a couple times, and depending on the post, couldn't tell if I was supposed to change what was after the colons next to the numbers, or just the end, or what.

On 9/11/2016 at 5:11 PM, Padishar said:

It also includes the changes I described earlier to the heap padding mechanism.  The values for each size of block are now simple weights and the total padding allocated in megabytes is specified separately (the "total" line at the end of padheap.cfg).  The total and weights are used to calculate how many 4k pages should be allocated for each block size range.  The default configuration splits the total equally between all the sizes up to 432 bytes and now allocates a bit more memory than it did before.

There was also: " if you add a 0 on the end of all the counts then it should allocate more like 9GB extra "

On 5/6/2016 at 5:52 AM, Padishar said:

Mod-End activates the heap padding.  The amount of padding is controlled by the MemGraph\PluginData\padheap.cfg.  The format of the file is very simple.  Each line controls the amount of padding allocated for blocks of each size range.  The first value is the size of each block allocated and the second is the number of blocks.  The first values are only present for illustration, they don't actually control the size of the blocks, these are hardwired to the sizes in the default configuration.  The total value at the end is the total number of megabytes of padding allocated (note, the memory usage of KSP will increase by more than this due to the free space headroom maintained by Unity).

The format is TOO simple. I"m not a comp-sci major. I don't know what in the heck you're talking about. How many values are "the first values"? Default config of what?

And lots more mess - and I can't even tell whether the heap padding is activating in game or not. When I just tried, I had to hit Mod+End 4 times across a couple minutes before I saw evidence that the GC interval had changed.

I don't need Up-Goer Five-speak; just tell us plebs how to get the maximum available for our RAM. I can watch Task Manager on another monitor to see how much room I've got left.

I'm sleepy and frustrated and I can't play the game because it's started locking up every 8 seconds. I never even had a freezing problem until 1.1 or 1.2, and then it just hit like a truck.

[Padishar]

9 hours ago, orcaman said:

I want to love this mod, but I just don't understand it. Do we add more heap every time we hit Mod+End or do we configure it using the config file? <snip>

Mod+End isn't a toggle, it simply causes the padding to be set up according to the values in the padheap.cfg.  Once the padding is applied it can only be totally removed by restarting the game.   However, changing the total value in padheap.cfg to 0 and hitting Mod+End again should remove most of it within a few cycles of the garbage collector.

The various descriptions are a bit technical but configuring it is now very simple.  Just run KSP normally and get into a situation where you get bad stutter,  Look in task manager to see how much free physical memory you have.  The only thing you are recommended to change is the number on the "total : " line at the end of padheap.cfg.  Set this to the number of megabytes that you want it to use for padding.  If your free memory is under 3 GB or so then you probably shouldn't increase the total value much from the default or you risk the game crashing (or the performance dropping through the floor due to virtual memory thrashing) fairly soon due to the tendency of KSP to use more and more memory as time (and scene switches in particular) goes by.  If you have a large number of large (in terms of texture size mostly) mods installed then you will have to use a lower number than of you only have a few mods.

As for general use, there is a setting to automatically apply the padding early in the startup of KSP but, if you have to use quite a small padding value then I recommend you don't use it and, instead, just hit Mod-End when you get the stutter.  The most important value in the window is "Last".  This shows how much memory was allocated by the C# parts of the game and mods in the last second (approx.).  To stand a decent chance of reducing the stutter significantly, this value needs to be reasonably small.  If your mod load and the state of your savegame results in this value being tens of megabytes (or more) then you will need to allocate a lot of padding to get the stutter intervals to a reasonable length.

Does this help?  If you still have questions then please post some screenshots showing the graph window (with the scale adjusted so the graph isn't totally green) and give some details about the spec of your machine and your mod load and I'll try to answer any questions you have and/or give advice on what to do...

[orcaman]

Thank you! I'm halfway there.

Your instructions in the first post mention Hitting Mod+End again a little while after the first time. What does that do?

What are "R" and "P" on the window?

Even if I don't get into it myself, I'm curious to know a bit more about the advanced config - which might all be answered by: What would the column headers be for the two columns in the config file?

I've got 6GB of RAM on Windows x64, no mods except Kerbal Alarm Clock by TriggerAu, and the Asteroid Day mod. I'm looking at buying more RAM just because of this problem.

[Padishar]

6 hours ago, orcaman said:

Your instructions in the first post mention Hitting Mod+End again a little while after the first time. What does that do?

It basically releases the old padding and allocates the padding again.  This may result in there being more padding available as some of the original padding will still stay around due to other memory allocations.

6 hours ago, orcaman said:

What are "R" and "P" on the window?

They are counts of the number of rendered and physics frames that happened in the last second (approx.).  So, R is basically your FPS and P is a measure of how much time is being slowed by the game (50 means it is running at real speed, 25 means it is running at half real speed).

6 hours ago, orcaman said:

Even if I don't get into it myself, I'm curious to know a bit more about the advanced config - which might all be answered by: What would the column headers be for the two columns in the config file?

The two columns would be "size of memory allocation" and "proportion of total padding created for blocks of that size".  Memory is allocated in 4kB pages which can hold a certain number of allocations, e.g. if the block allocated is 8 bytes or less, it gets put in a 4kB page that holds blocks between 1 and 8 bytes, if a block of 123 bytes is allocated it gets put in a page for blocks between 113 and 144 bytes.  The padding works by deliberately allocating a single block from each 4kB page so there is lots of free space still unused but the page can't get freed because it always has one block in use.  Each row in the config sets the amount of the total that is used for that block size so, with the default settings, the total of 1024 MB is split equally between the first 17 block sizes, making ~61680 kB for each size of block so 15420 4kB pages get allocated for each block size.  Changing the first numbers makes no difference, they are just labels, the block sizes are defined by the Unity/Mono memory allocator.  If you changed just one of the lines (e.g. the first one) to be 4 instead of 1, then 4 times as many blocks would be allocated for that size than the others.  As there would now be 20 weight in total, the 1s would represent 13107 pages and the 4 would be 52428 pages.  So, these lines allow you to set the "shape" of the padding without needing to do the calculation of page counts manually.

7 hours ago, orcaman said:

I've got 6GB of RAM on Windows x64, no mods except Kerbal Alarm Clock by TriggerAu, and the Asteroid Day mod. I'm looking at buying more RAM just because of this problem.

I assume you have quite a complex savefile (lots of ships, and/or big ships) to be getting bad stutter with just those mods.  I would expect the default padding to make an obvious difference, what does the Last value say (and the graph itself, which is just of the Last value)?

[orcaman]

Thank you! I finally understand it all, now. I've gone up to 2GB of padding which still leaves a comfortable amount of headroom, and turned down my max framerate, because if I understood correctly, the garbage is generated per frame, right? That's gotten me up to 40-50 seconds per GC, which is fine, I think. My save file is 8MB, if that's an adequate metric. If you count flags and asteroids, I've probably got 150 flights. Reducing the number of flags is another measure I took to speed the game up after reading this thread. I get from 8 to 15 MB/s in the "last" field. P is 50 or 51. Thank you for developing this, and also for taking the time to answer my questions!

[Oshika]

On 10/12/2016 at 8:34 PM, Oshika said:

Thank you i will try that and keep you informed on the situation. 

Three months after, 

I'm here to say it worked perfectly and the game runs well since i use your mod, thank you very much.