[1.2.x/1.3] MemGraph 1.1.0.3 - with Stutter Reduction

[kerbonaut7777]

Hi all guys, so my problems with this mod are:

1. in settings.cfg if i change applyPadding = false to applyPadding = true, game wont start, so actualy it is set as it was in default applyPadding = false with this i can run the game

2. this mod works, but only using mod+end (left alt + end) 1st time, it add that 900memory or something like that and intervals between lags increases, after using mod+end again after few garbage colletions cycles it do NOTHING

3. if i change in padheap.cfg ONLY total from 1024 to 2048 it works and it adds cca 2gb of memory = lag every 115s, but if i change TOTAL to 4096 game will crash when trying to apply padheap with mod+end

4. if i change in padheap.cfg rows 664,800,1008,1344,2032 from 0 as they are in default to 1 with TOTAL set to 4096, it will allow me to add that 4gb but game will crash when first loading screen happens (loading my save or change from hangar to plane on the airfield etc..)

 

32gb of memory, ONLY THIS MOD, nothing else

Edited October 17, 2016 by kerbonaut7777
correction

[Padishar]

1 hour ago, kerbonaut7777 said:

1. in settings.cfg if i change applyPadding = false to applyPadding = true, game wont start, so actualy it is set as it was in default applyPadding = false with this i can run the game

Do you have the latest version of this mod?  This was a bug in 1.0.0.9 (for KSP 1.1.3) and 1.1.0.0 (for KSP 1.2).  Versions 1.0.0.10 (for KSP 1.1.3) and 1.1.0.1 (for KSP 1.2) should work fine with applyPadding = true.

1 hour ago, kerbonaut7777 said:

2. this mod works, but only using mod+end (left alt + end) 1st time, it add that 900memory or something like that and intervals between lags increases, after using mod+end again after few garbage colletions cycles it do NOTHING

It isn't supposed to get a lot better the second and subsequent times.  Each time you apply it the old padding is discarded and new padding is allocated.

1 hour ago, kerbonaut7777 said:

3. if i change in padheap.cfg ONLY total from 1024 to 2048 it works and it adds cca 2gb of memory = lag every 115s, but if i change TOTAL to 4096 game will crash when trying to apply padheap with mod+end

Are you running the 64 bit version of KSP?  The 32 bit version can only ever allocate approx. 3.6 GB so asking it to allocate an extra 4 GB will certainly crash KSP.  If you are running the 64 bit version then how much free physical memory do you have when you are running KSP before applying the padding?  Using more than this will probably not crash KSP but it may cause it to start paging to virtual memory and this may slow it down enough to make Windows think it has crashed...

1 hour ago, kerbonaut7777 said:

4. if i change in padheap.cfg rows 664,800,1008,1344,2032 from 0 as they are in default to 1, game will crash when first loading screen happens (loading my save or change from hangar to plane on the airfield etc..)

There is no reason why it should crash due to changing those values unless you are running the 32 bit version and the extra is resulting in the total memory usage going over 3.6 GB.

It certainly sounds like you are running the 32 bit version (KSP.exe) rather than the 64 bit version (KSP_x64.exe).  Try running KSP_x64.exe directly from windows explorer rather than running from Steam or an old shortcut you may have set up.

If you still have problems then please supply an output_log.txt file (upload it to a file sharing service like dropbox or similar).  You will find this file in the KSP_Data or KSP_x64_Data folder depending on whether you are running the 32 or 64 bit version (and you should definitely be running the 64 bit version).

Edited October 17, 2016 by Padishar

[kerbonaut7777]

1 hour ago, Padishar said:

Do you have the latest version of this mod?  This was a bug in 1.0.0.9 (for KSP 1.1.3) and 1.1.0.0 (for KSP 1.2).  Versions 1.0.0.10 (for KSP 1.1.3) and 1.1.0.1 (for KSP 1.2) should work fine with applyPadding = true.

It isn't supposed to get a lot better the second and subsequent times.  Each time you apply it the old padding is discarded and new padding is allocated.

Are you running the 64 bit version of KSP?  The 32 bit version can only ever allocate approx. 3.6 GB so asking it to allocate an extra 4 GB will certainly crash KSP.  If you are running the 64 bit version then how much free physical memory do you have when you are running KSP before applying the padding?  Using more than this will probably not crash KSP but it may cause it to start paging to virtual memory and this may slow it down enough to make Windows think it has crashed...

There is no reason why it should crash due to changing those values unless you are running the 32 bit version and the extra is resulting in the total memory usage going over 3.6 GB.

It certainly sounds like you are running the 32 bit version (KSP.exe) rather than the 64 bit version (KSP_x64.exe).  Try running KSP_x64.exe directly from windows explorer rather than running from Steam or an old shortcut you may have set up.

If you still have problems then please supply an output_log.txt file (upload it to a file sharing service like dropbox or similar).  You will find this file in the KSP_Data or KSP_x64_Data folder depending on whether you are running the 32 or 64 bit version (and you should definitely be running the 64 bit version).

1. yes i do, 1.1.0.1. downloaded from this forum from 1st page, zip file says it also, but game wont launch with applyPadding = true  i coppied the MemGraph folder to gamedata folder, so now there are two Memgraph and Squad, i did not copy only MemGraph.dll as u have in README.md otherwise it only show memgraph window ingame, but cant do anything like changing scale or apply padding

2. OK got it

3. when i launch the game it says WindowsPlayer 64, game version 1.2.0.1586, i have 21-24gb free physical memory with game launched before padding

4. yet it does , can not find output_log.txt file in KSP_x64_Data

Edited October 17, 2016 by kerbonaut7777
correction

[Padishar]

2 hours ago, kerbonaut7777 said:

1. yes i do, 1.1.0.1. downloaded from this forum from 1st page, zip file says it also, but game wont launch with applyPadding = true  i coppied the MemGraph folder to gamedata folder, so now there are two Memgraph and Squad, i did not copy only MemGraph.dll as u have in README.md otherwise it only show memgraph window ingame, but cant do anything like changing scale or apply padding

2. OK got it

3. when i launch the game it says WindowsPlayer 64, game version 1.2.0.1586, i have 21-24gb free physical memory with game launched before padding

4. yet it does , can not find output_log.txt file in KSP_x64_Data

1. Very strange, the change I made in 1.0.0.10/1.1.0.1 fixed it when using the pre-release version but doesn't seem to work with the actual release.  The readme in the install zip is a bit out of date.  If you have copied the MemGraph folder then you don't need to copy the DLL separately (and this could cause an issue).

3. Well, that would appear to be the 64 bit version.

4. Very strange.  Is this installed in "Program Files"?  You could try copying the whole KSP folder to a different place on your disk (e.g. create a C:\Games folder and put it in there).

I have just released versions 1.0.0.11 and 1.1.0.2 with another fix for the applyPadding = true problem.

[kerbonaut7777]

23 minutes ago, Padishar said:

1. Very strange, the change I made in 1.0.0.10/1.1.0.1 fixed it when using the pre-release version but doesn't seem to work with the actual release.  The readme in the install zip is a bit out of date.  If you have copied the MemGraph folder then you don't need to copy the DLL separately (and this could cause an issue).

3. Well, that would appear to be the 64 bit version.

4. Very strange.  Is this installed in "Program Files"?  You could try copying the whole KSP folder to a different place on your disk (e.g. create a C:\Games folder and put it in there).

I have just released versions 1.0.0.11 and 1.1.0.2 with another fix for the applyPadding = true problem.

FIXED !!! ur new version is WORKING :)) rly good work and rly fast fix, all the problems are fixed by the new version 1.1.0.2, I also moved it to program files (64) but i tried this before and nothing changed, i think only this new version is enough, will try it in normal steam folder (program files 32) tomorrow, thanks again

changed only total to 4096, and applyPadding=true, works great, ksp around 7gb physical memory in task manager
 

Edited October 17, 2016 by kerbonaut7777

[Padishar]

11 hours ago, kerbonaut7777 said:

I also moved it to program files (64) but i tried this before and nothing changed, i think only this new version is enough, will try it in normal steam folder (program files 32) tomorrow

Both of the Program Files folders have strange file and folder permissions which can interfere with programs that attempt to write files in there.  You should try copying it to a completely new folder, e.g. create a new Games folder in the root of your drive and put the KSP folder in there.  Output_log.txt is by far the most important and useful file when trying to diagnose problems with KSP so I really would recommend you get it working to avoid headaches in the future.

[kerbonaut7777]

On 18. 10. 2016 at 0:49 PM, Padishar said:

Both of the Program Files folders have strange file and folder permissions which can interfere with programs that attempt to write files in there.  You should try copying it to a completely new folder, e.g. create a new Games folder in the root of your drive and put the KSP folder in there.  Output_log.txt is by far the most important and useful file when trying to diagnose problems with KSP so I really would recommend you get it working to avoid headaches in the future.

u get it, that its working with ur updated version right ? :))

[Padishar]

29 minutes ago, kerbonaut7777 said:

u get it, that its working with ur updated version right ? :))

Ahhh, no, I didn't understand from your last post that you were now getting an output_log.txt generated too (mainly because the change I made should have had no effect on that)...  Still, all good, glad it's all working for you now...

[Fwiffo]

After getting increasingly annoyed with stutter in the VAB and seeing the "stutter reduction" carrot dangling in your signature several times, I finally got around to trying out your mod tonight.  Congratulations, after a mere few minutes of use your mod has made it into my top-10 "must have" list.

Thank you for improving the quality of my life.

Next up, figuring out how to tweak your cfg to make KSP way more aggressive / greedy.  I have 48 GB of RAM on this machine and would love for KSP to grab another 20GB or so to make those evil red lines even more sparse. ;-)

Edit: After some reckless fiddling with padheap.cfg, life is pretty good...

Edited October 20, 2016 by Fwiffo

[Auriga_Nexus]

I'm playing the latest version and I get crashes whenever I transition to a new scene after applying heap padding settings. I can't tell if it's because I'm running out of memory or what. Currently all blocks are set to 1 and the total size is set to 4096. I'm running KSP 64-bit with 16 GB of memory. Gonna try moving to a smaller amount to see if it helps.

[Padishar]

2 hours ago, Fwiffo said:

Edit: After some reckless fiddling with padheap.cfg, life is pretty good...

Glad you like it...

1 hour ago, Auriga_Nexus said:

I'm playing the latest version and I get crashes whenever I transition to a new scene after applying heap padding settings. I can't tell if it's because I'm running out of memory or what. Currently all blocks are set to 1 and the total size is set to 4096. I'm running KSP 64-bit with 16 GB of memory. Gonna try moving to a smaller amount to see if it helps.

If you can provide an output_log.txt and/or a crash dump file then I may be able to help.  It sounds like KSP is probably running out of memory but without more info there's no way to tell...

[Auriga_Nexus]

15 minutes ago, Padishar said:

Glad you like it...

If you can provide an output_log.txt and/or a crash dump file then I may be able to help.  It sounds like KSP is probably running out of memory but without more info there's no way to tell...

Actually that's probably what it was. I have no evidence to confirm, but I changed the file to run off of a total memory of 2048 and set the lines from 660 to 2022 (or whatever those block sizes are) to 0 like in the default file, and haven't had any crashes yet.

[Alex55]

Thanks so much for making this fantastic mod! It really improved the gameplay.

A few tidbits from my experience with it:

1) It's a good idea to wipe out various dat/cache files that various mods put into the KSP_win64/GameData folder. (FinalFrontier.dat, ModuleManager.ConfigCache, etc) I was getting nasty Access Violation errors when loading the game and/or save files after buffing the memory. Wiping out those files fixed it. Example Error:

Unity Player [version: Unity 5.4.0p4_b15b5ae035b7]

KERNELBASE.dll caused an Access Violation (0xc0000005)
  in module KERNELBASE.dll at 0033:17a77788.

2) On my machine, I can get a "Too many heap sections" error if I increase the padding to high. (3 for all available sizes). Other users on this forum seem to be able to go much higher. Has anyone tried this fix and had it work for ksp? https://steamcommunity.com/app/242760/discussions/0/522728813818195362/

 

 

Edited October 22, 2016 by Alex55

[Padishar]

19 minutes ago, Alex55 said:

1) It's a good idea to wipe out various dat/cache files that various mods put into the KSP_win64/GameData folder. (FinalFrontier.dat, ModuleManager.ConfigCache, etc) I was getting nasty Access Violation errors when loading the game and/or save files after buffing the memory. Wiping out those files fixed it. Example Error:

Wiping out those files really shouldn't have fixed anything as they should have just been recreated by the relevant mods the next time you ran the game.  There was almost certainly something else going on here.

21 minutes ago, Alex55 said:

2) On my machine, I can get a "Too many heap sections" error if I increase the padding to high. (3 for all available sizes). Other users on this forum seem to be able to go much higher. Has anyone tried this fix and had it work for ksp? https://steamcommunity.com/app/242760/discussions/0/522728813818195362/

Unless you have read (and understood) all of the explanatory posts in this thread then you shouldn't be changing anything in padheap.cfg except for the total= line at the end.  Just set this to the number of megabytes of memory you want it to use.  Setting all the other values to 3 will have no different effect to setting them all to 1 (or any other number), they are simple weights specifying how the total amount is split between the different block sizes.

I would also recommend that you don't mess with the things that steam thread mentions unless you understand what it is they do.

[Alex55]

Thanks Padishar!

That makes life much easier. I was using an excel spreadsheet to recalculate the total amount of memory from the chunks, but it looks like I don't need to do that :). I'll leave the steam post alone. I actually have no idea what they're tweaking there.

RE: Deleting cache files from GameData Folder. Are you thinking that one or more of the cache files was corrupt and would've caused a crash eventually even without the padding? (The padding just caused it to happen sooner somehow)

 

Here is some more info on what was going on:

The crashes did not occur in the unmodded (except for MemGraph) game. The crashes weren't random. They would reliably occur on any load operation after padding:

1) applyPadding = false

1. start game
2. load game
3. apply padding at space complex
4. load game -> Crash

 

1. start game
2. apply padding at main menu
3. load game -> Crash

2) applyPadding = True

1. start game -> Crash

They seem to have gone away after I deleted the cache files. I was using ~14GB of padding on a machine with 60GB+ memory not in use. I'm running KSP 1.2.

The cache files were regenerated, but the game had no obvious problems with the regenerated versions. 

[Terensky]

@Padishar: thank you for making me enjoying the game MUCH more.

In case you'd need them:

Spoiler

output_log.txt

http://imgur.com/a/X9uoF

i7-4770K @3,50GHz, 16GB RAM, Windows10 x64, nVidia GTX 760 2GB

 

[Padishar]

17 hours ago, Terensky said:

 

@Padishar: thank you for making me enjoying the game MUCH more.

In case you'd need them:

I'm glad it's letting you enjoy the game more.  Thanks for the screenshot and log, I wouldn't usually need (or want ) the log unless you're having problems but, in this case, it has reminded me about a KER issue that I was investigating a few months back whose fixes are still sat in a separate branch on my fork (the exception from ScienceUtil.GetExperimentBiome).  I'll get that merged into my 1.2 development branch.

On the subject of this mod, given you have 16 GB of RAM, you would probably be better off increasing the total= line in GameData\MemGraph\PluginData\MemGraph\padheap.cfg to 2048, 3072 or even higher.  This should increase the interval between the stutters even further.

[Terensky]

Quote

On the subject of this mod, given you have 16 GB of RAM, you would probably be better off increasing the total= line in GameData\MemGraph\PluginData\MemGraph\padheap.cfg to 2048, 3072 or even higher.

Thanks, I will try for sure. Even with the current settings, the improvement is substantial: before I had stutter every 2 or 3 seconds, now it's almost unnoticeable. MemGraph has become one of my 'must have' mod (well, calling it a mod seems a little reductive). Not to sound apologetic (anyway I'm very ignorant on the subject), but Squad should look into this and eventually integrate it in the game.

[charliedrewitt]

This mod is the bees knees. Before installing I was stuttering every couple of seconds, now it's every 30-40 and seems to have barely changed the duration of the hitch. This has transformed the game for me.

Padishar, as a .NET programmer myself, I am thoroughly humbled by what you've achieved here. Excellent work and many thanks.

[OddFunction]

Just this morning I was thinking about how sad I was at the stutter. I was thinking about how it would probably never be fixed due to... reasons. I thought I'd have to make due, perhaps even one day buy an extremely overpowered computer just to run KSP at optimum.

 

I never imagined that by the end of the same day I'd find an extremely easy to install mod that just simply fixes the problem. I'm playing around now in my biggest station, mousing around in every single direction and am just amazed that there is no stutter. I'm sure I'll notice it again someday but for right now I honestly can't.  

 

And that's just on 16GB configured at a 3074 heap in the settings. If I slapped another 16GB into my open slots I could probably extend the stutter into oblivion. 

 

 

I guess what I'm trying to say is Thank You, which almost isn't enough. This mod feels like the moment when your headache subsides. It's too good to be true!

[Padishar]

20 hours ago, charliedrewitt said:

This mod is the bees knees. Before installing I was stuttering every couple of seconds, now it's every 30-40 and seems to have barely changed the duration of the hitch. This has transformed the game for me.

Padishar, as a .NET programmer myself, I am thoroughly humbled by what you've achieved here. Excellent work and many thanks.

1 hour ago, OddFunction said:

This mod feels like the moment when your headache subsides. It's too good to be true!

Thanks for the kind words (and to any others that I've not acknowledged previously).  This game is too good for me to avoid playing because of annoying stutter so I had to try and find a work-around...

[DaniDE]

@Padishar thank you for this wonderful mod. I have one simple question, do you have to apply the padding manually everytime you restart the game? I have noticed padding = false is set by default, and I am not sure if that is safe to turn on and if you still should manually invoke it by alt-end after a while even if that is set to true. (I am just assuming here that this setting does the auto-applying :-).

 

Thanks

[Padishar]

16 minutes ago, DaniDE said:

@Padishar thank you for this wonderful mod. I have one simple question, do you have to apply the padding manually everytime you restart the game? I have noticed padding = false is set by default, and I am not sure if that is safe to turn on and if you still should manually invoke it by alt-end after a while even if that is set to true. (I am just assuming here that this setting does the auto-applying :-).

You're welcome.  Indeed, you can have it applied automatically on startup by setting the applyPadding value in GameData\MemGraph\PluginData\MemGraph\settings.cfg to true.  The padding is then applied very early in the startup of KSP.  Because it's so early, a significant amount of the padding gets used by the permanent data that KSP loads which does slightly reduce the effectiveness.  For maximum benefit, the padding should be reapplied from time to time but, each time it is re-applied can cause the heap to become more fragmented, resulting in it tending to grow over time.  So, it's a bit of a balancing act and I've not decided what to do about it yet but, for the moment, you should probably apply the padding again with the hotkey after loading your save and then only try reapplying it if the stutter gets noticeably worse...

You can also hide the window by default by setting visible to false and you can change the hotkeys (or disable individual ones by setting them to None).

[DaniDE]

Thanks for clarifying this

[Padishar]

A little update on the sort of stutter improvement possible in a mostly stock install of KSP 1.2.1.  This has KER installed but I deliberately kept all its windows closed to avoid it skewing the results.  I ran KSP 1.2.1, loaded the save with the vessel in my avatar (this is a 1410 part vessel) and switched to it from the tracking station.  The garbage collections were happening at 20 second intervals.  I then applied the default MemGraph heap padding of 1024 MB and waited for the next garbage collection.

And waited, and waited some more.  This shows the initial game loading process and the first three garbage collections that happened at 20s intervals.  After applying the padding, it got to the end of the graph (10 minutes since starting the game, about 8 minutes since applying the padding) without another collection:

I left it running until the next collection happened:

Spoiler

The graph went all the way round again and started on its third lap.  Yes, that's 1210 seconds before the next collection, over 20 minutes, an approximately 60x improvement in the stutter frequency...

This improvement will not be the same in all cases, mods that allocate lots of large blocks of memory will significantly reduce the effectiveness, but this sort of improvement should be common in mostly stock installs...

Edited November 4, 2016 by Padishar