[1.2.x/1.3] MemGraph 1.1.0.3 - with Stutter Reduction

[katateochi]

On 04/11/2016 at 9:21 AM, Padishar said:

Yes, that's 1210 seconds before the next collection, over 20 minutes, an approximately 60x improvement in the stutter frequency.

I just loaded up your save in a completely fresh KSP install with just MemGraph and got almost identical results. Initial stuttering was ~28 second intervals and then after applying the default heap padding it went for 1239 seconds before the next GC event.  Here's a couple scr-shots;

Spoiler

I then copied my stock save into the same install and tried it with a group of craft parked around KSC.  There's 5 craft (one's just a command pod) but they only total 271 parts but this results in much more frequent GC events (~360 second intervals after applying heap padding).  Is this due to certain parts producing more garbage or is it due to it being around KSC?

Spoiler

 

 

[Padishar]

6 minutes ago, katateochi said:

Is this due to certain parts producing more garbage or is it due to it being around KSC?

Well, Squad did a lot of work to eliminate per-frame garbage creation but there may be some parts that do still cause more than others.  However, in this case, the difference is more likely due to the difference in frame rate, e.g. the amount of garbage created per frame is probably similar but your save is doing 4x the FPS that mine does so it generates ~4x the garbage per second.

[katateochi]

1 hour ago, Padishar said:

Well, Squad did a lot of work to eliminate per-frame garbage creation but there may be some parts that do still cause more than others.  However, in this case, the difference is more likely due to the difference in frame rate, e.g. the amount of garbage created per frame is probably similar but your save is doing 4x the FPS that mine does so it generates ~4x the garbage per second.

ah of course! that makes sense. With your save I got a steady 15fps, with my save it was at 60fps (vsync on).
Much love for your mod Sir, it's now an essential tool to have!
In my modded setup I've set the heap padding to 2GB, auto apply to true and then I find I need to hit alt+end a couple times to get to ~300s between stutters. I did try setting the heap padding waaay high (I've got the RAM for it), but while that gave longer intervals the stutter was much more pronounced (which is expected).

[Merit]

Hero mod, day and night saving an insanely modded save file, thanks so much @Padishar!

[Tr1gg3r]

thought you may find this info (Linkd) useful, i was mid stutter in flight, tried to Alt End to help then CTD, didnt get pics of your meter alas, but heres logs, hope they help.

https://www.dropbox.com/s/ep926w29tdt32f8/KSP crash.log?dl=0

https://www.dropbox.com/s/318z03dnh523ipe/output_logcrash.txt?dl=0

 

Edited November 10, 2016 by Tr1gg3r
Correct info

[Padishar]

7 hours ago, Tr1gg3r said:

thought you may find this info (Linkd) useful, i was mid stutter in flight, tried to Alt End to help then CTD, didnt get pics of your meter alas, but heres logs, hope they help.

https://www.dropbox.com/s/ep926w29tdt32f8/KSP crash.log?dl=0

https://www.dropbox.com/s/318z03dnh523ipe/output_logcrash.txt?dl=0

It doesn't look like the applying of the padding was directly responsible as the log shows it applied successfully and then quite a few other things happened before Unity/Mono reported an "out of memory" on a subsequent allocation that then led to the crash.  Do you have your Windows pagefile disabled or fixed to a small size?  The 64 bit version really shouldn't be getting out of memory errors unless you run out of free memory and Windows is unable to allocate space in the pagefile...

You have at least one mod that hasn't yet been updated to KSP 1.2 (it's still trying to use KSPUtil.dll).  IRSurfaceSampler and IRActiveStruts both appear to do this so I would be suspicious of all of the Infernal Robotics stuff (is this an unofficial build for 1.2?).  There are quite a few exceptions from IndicatorLights so that should also be investigated.  Almost a third of your log is full of exception spam from docking ports and, while I think these are probably related to the Infernal Robotics issues, they could be causing underlying issues (e.g. memory leaks or otherwise upsetting Unity in some way).

[Tr1gg3r]

Hi Padishar,

thats interesting, all my mods 'claim to be for 1.2.1 except mk2 station parts expansion which is for 1.2.

i need to remove IR surface sampler and IR struts as iirc theyre known not work atm, but i dont use them anyway so dont know why theyre throwing exceptions as theyre certainly not on any of my craft.

im not surprised by IR spam as i was testing limits of docking ports attached to IR parts (kraken juice)

i check for mod updates everytime before i play through ckan and avc.

re-page file etc, ive never changed it, so itll be on default, i remember pressing lots of buttons on your meter tho. Lol could i have scaled it wrong which affected things maybe.

shame indicator lights is causing issue, like those, ill look into that.

but all in all, your mod seems to help mate and shows theres still some underlying issues to be ironed out.

cheers

[Padishar]

I have just released version 1.1.0.3 for KSP 1.2.x.  The only change is to fix a bug where the graph window is duplicated each time the "Reload Database" feature is used in the stock debug console.

[FizzerUK]

OK..  for me thisis working in reverse. What am I doing wrong.

Waitied fior stutter and GC, hit mod-end, and it got worse, ht-meend agin now I have GC every few seconds?

Pretty sure I am doing something wrong here but what.

The end of the graph is what I started with.

Edited November 19, 2016 by FizzerUK

[Padishar]

42 minutes ago, FizzerUK said:

OK..  for me thisis working in reverse. What am I doing wrong.

Waitied fior stutter and GC, hit mod-end, and it got worse, ht-meend agin now I have GC every few seconds?

Pretty sure I am doing something wrong here but what.

The end of the graph is what I started with.

It sounds like you don't have enough free memory before you start.  How much actual RAM do you have?  Other machine specs?  What OS are you running?  What other mods do you have installed and how much memory is KSP using when you get to the main menu?  How much after loading your save game?  Have you changed the padding settings at all?  Can you upload an output_log.txt file and link it here?  What situation were you in when that screenshot was taken?  There were no physics updates during the previous second and 46 rendered frames, which implies that game time was suspended but the game wasn't locked up, was the game paused?

[Calvin_Maclure]

Im actually having a problem where my mod key (left Alt) doesnt work in game now for some reason?? Dont think this has anything to do with this mod but, kinda dont know who to ask... 

[Padishar]

Well, the tech support or general discussion areas would be more appropriate than here, but I'll suggest a couple of things.  First, make sure it works in other programs, Alt and the first letter of a menu should open it, e.g. Alt-F should open the File menu in Notepad (and most other editor programs).  Next, open the settings.cfg file in the KSP folder with notepad and search for MODIFIER_KEY.  This should tell you what the modifier key is configured as in the game.  You should also be able to see this in the in-game settings and change it back to the correct defaults if it is wrong.  Alternatively, you can just correct it in notepad.  This is the section from my settings.cfg:

MODIFIER_KEY
{
	primary = LeftAlt
	secondary = RightAlt
	group = 0
	modeMask = -1
	modeMaskSec = -1
}

 

[sh1pman]

Just want to toss my 50 cents here. I had un unbearable stutter that was repeating every 3 seconds or so. So I went and cleaned ~50 or so debris pieces that were flying all over the Kerbol system. Doing so increased my GC time to 12 seconds. Not bad. Four times less garbage per second.

But it gets even better. With 2GB heap padding I got something like 24-26 seconds between GC hiccups. After cleaning up the debris it went up to 150. Wow. You made KSP playable again!

[Calvin_Maclure]

On 11/22/2016 at 1:35 PM, Padishar said:

Well, the tech support or general discussion areas would be more appropriate than here, but I'll suggest a couple of things.  First, make sure it works in other programs, Alt and the first letter of a menu should open it, e.g. Alt-F should open the File menu in Notepad (and most other editor programs).  Next, open the settings.cfg file in the KSP folder with notepad and search for MODIFIER_KEY.  This should tell you what the modifier key is configured as in the game.  You should also be able to see this in the in-game settings and change it back to the correct defaults if it is wrong.  Alternatively, you can just correct it in notepad.  This is the section from my settings.cfg:

MODIFIER_KEY
{
	primary = LeftAlt
	secondary = RightAlt
	group = 0
	modeMask = -1
	modeMaskSec = -1
}

 

Thanks for the response. Yep, this is exactly what I have as well and yes the Alt does indeed work ok... But I just dont seem to notice any thing either in game or in the logs when I try to apply the padding... And Ive installed my fair share of mods so, its not like I dont know what im doing there, although mistakes do happen... hmm...  

[Padishar]

1 hour ago, Calvin_Maclure said:

But I just dont seem to notice any thing either in game or in the logs when I try to apply the padding...

Can you try running the game, let it get to the main menu, try Alt-NumPadMultiply, Alt-NumPadPlus and Alt-NumPadMinus (their effects should be obvious from the graph window) and then try Alt-End.  Then quit the game and upload your whole output_log.txt file to somewhere like dropbox and I'll take a look...

[Calvin_Maclure]

3 hours ago, Padishar said:

Can you try running the game, let it get to the main menu, try Alt-NumPadMultiply, Alt-NumPadPlus and Alt-NumPadMinus (their effects should be obvious from the graph window) and then try Alt-End.  Then quit the game and upload your whole output_log.txt file to somewhere like dropbox and I'll take a look...

Sounds good. Just to be clear, ''main menu'', you mean the very first menu upon game startup (Start Game / Settings / ... Quit)?

[Padishar]

Just now, Calvin_Maclure said:

Sounds good. Just to be clear, ''main menu'', you mean the very first menu upon game startup (Start Game / Settings / ... Quit)?

Yes...

[Oshika]

Wow, 

I have create an account only to post my experience of your mod. 

Thank you a lot, actually it works perfectly, i play like 6H to my savegame then suddenly my game makes stutters and it becomes unplayable and so i install your mod and no more stuttering  

i don't know if i have to make the process again at every launch of ksp but for the moment it works ! 

here a screenshot of the graph : 

 

Spoiler

(Sorry for my bad english) 

[Padishar]

1 hour ago, Oshika said:

Thank you a lot, actually it works perfectly, i play like 6H to my savegame then suddenly my game makes stutters and it becomes unplayable and so i install your mod and no more stuttering  

You're welcome.  From the look of your graph you must have one or more mods generating a very large amount of garbage, 130 MB/s is very high.  You should probably have a look to see if any of your mods have updates available.  My first guess would be that you have an older version of the Contract Configurator mod and you have quite a few active contracts.  If this is the case then there is a newer version available that fixes this.  However, their could be other mods that are also causing too much garbage.  If you could post a list of your mods (or upload an output_log.txt file somewhere and link to it) I may be able to suggest which mods are most likely to be causing the problem.

Also, which version of KSP are you running?  Version 1.2.0 had a serious garbage issue when both part highlighting and anti-aliasing were switched on.  This was fixed in 1.2.1 though that introduced a crash issue in the same area of the game code.  This crash has been fixed in version 1.2.2.  If you are running anything earlier than 1.2.2 then you should disable either the part highlighting or the anti-aliasing to avoid either the garbage thrash or the crash.

[Oshika]

On 08/12/2016 at 4:01 PM, Padishar said:

You're welcome.  From the look of your graph you must have one or more mods generating a very large amount of garbage, 130 MB/s is very high.  You should probably have a look to see if any of your mods have updates available.  My first guess would be that you have an older version of the Contract Configurator mod and you have quite a few active contracts.  If this is the case then there is a newer version available that fixes this.  However, their could be other mods that are also causing too much garbage.  If you could post a list of your mods (or upload an output_log.txt file somewhere and link to it) I may be able to suggest which mods are most likely to be causing the problem.

Also, which version of KSP are you running?  Version 1.2.0 had a serious garbage issue when both part highlighting and anti-aliasing were switched on.  This was fixed in 1.2.1 though that introduced a crash issue in the same area of the game code.  This crash has been fixed in version 1.2.2.  If you are running anything earlier than 1.2.2 then you should disable either the part highlighting or the anti-aliasing to avoid either the garbage thrash or the crash.

Thank you for your reply. 

Here my mod list and my game data :  

Spoiler

 

i think there is a few mods that ckan don't display like Kerbal planetary base system and hide empty tech tree nodes. 

You are right with contract configurator, i have a manuel downloaded version and  probably not up to date, i will download the update thank you.   --> see Edit below

My ksp version is the last, 1.2.2

And here my output_log txt : https://dl.dropboxusercontent.com/u/80927159/output_log.txt

Edit : I have already the last version of contract configurator.

 

Edit 2 : 

I finally re-installed my game, no mods, only the memgraphics, and no more stuttering, then i put only a few mods and the stuttering come again (every 2 minutes) but i want to remove it completly, is it possible ? 

Edited December 9, 2016 by Oshika

[Padishar]

On 12/8/2016 at 4:54 PM, Oshika said:

I finally re-installed my game, no mods, only the memgraphics, and no more stuttering, then i put only a few mods and the stuttering come again (every minutes) but i want to remove it completly, is it possible ? 

Sorry for not replying to this sooner, you have a very large list of mods and it's likely that several are each contributing significantly but it's hard to say which ones.  With a mostly stock install it should be possible to use the MemGraph heap padding feature to virtually eliminate the garbage stutters as long as you have enough physical RAM in your machine and the "Last" value at the top of the graph isn't showing a huge number.  A screenshot of the graph window in this new install could be helpful, leave it running in flight for a little while with the stutter and then hit Mod-End, leave it running for some more and then take a screenshot.  If that doesn't improve it enough and you have a reasonable amount of free physical memory while running KSP then you can change the total= line in the padheap.cfg file in the MemGraph/PluginData folder...

[Oshika]

15 hours ago, Padishar said:

Sorry for not replying to this sooner, you have a very large list of mods and it's likely that several are each contributing significantly but it's hard to say which ones.  With a mostly stock install it should be possible to use the MemGraph heap padding feature to virtually eliminate the garbage stutters as long as you have enough physical RAM in your machine and the "Last" value at the top of the graph isn't showing a huge number.  A screenshot of the graph window in this new install could be helpful, leave it running in flight for a little while with the stutter and then hit Mod-End, leave it running for some more and then take a screenshot.  If that doesn't improve it enough and you have a reasonable amount of free physical memory while running KSP then you can change the total= line in the padheap.cfg file in the MemGraph/PluginData folder...

Thank you for your reply,

Firstly a screenshot of the test on my stock install with only a few mods : 

Spoiler

And the screenshot of the memgraph : 

Spoiler

The green arrow represent the processus after doing Mod + end

No more stuttering at all after doing Mod+end combination :

Spoiler

 

 

But i don't really understand why i can't have no stuttering without memgraphic (and patch), because in this case i have only 2 mods  

 

Edit : Ok so even with a fresh install and only memgraph as mod, i have stuttering, this is insane seriously i don't understand : 

so ok maybe it's not all the 5sec but it's not normal for a game who is supposed to be out of early access to have stutters with a config like mine (i7 4790k, 16gb of ram, gtx 980). 

Now i know it don't come essentially from the mods, even a native game, fresh install, can't have no stuttering... 

Spoiler

 

 

 

 

Edited December 10, 2016 by Oshika

[Padishar]

20 hours ago, Oshika said:

Edit : Ok so even with a fresh install and only memgraph as mod, i have stuttering, this is insane seriously i don't understand : 

This is just how the game engine works combined with code that generates too much garbage.  Basically, Unity/Mono keeps a certain amount of free headroom in the heap and once this fills up the garbage collector is triggered which causes a short pause.  The headroom is a certain fraction of the total size of the heap so when less memory in being used there is a lot less headroom and the stutters can happen quite frequently.  The MemGraph heap padding feature both forces a large amount of extra headroom and, due to the increase in total heap size, makes Unity/Mono allocate more headroom itself.

[Oshika]

13 minutes ago, Padishar said:

This is just how the game engine works combined with code that generates too much garbage.  Basically, Unity/Mono keeps a certain amount of free headroom in the heap and once this fills up the garbage collector is triggered which causes a short pause.  The headroom is a certain fraction of the total size of the heap so when less memory in being used there is a lot less headroom and the stutters can happen quite frequently.  The MemGraph heap padding feature both forces a large amount of extra headroom and, due to the increase in total heap size, makes Unity/Mono allocate more headroom itself.

Thank you for your answer, 

If i understand well (because my english is not perfect) i have to use the mod+end function of your memgraph mod to remove the stutter or to reduce it until the devs found a solution/fix that (are they even know this issue ?) ?

I have 16gb of ram, if i put a lot of mods (well, Being careful that the mods are compatible and up to date) can i use the "total" line of padheap.cfg to allocate more ram and be able to remove more stutter ? what value should i use ? 8192 ?

Edited December 10, 2016 by Oshika

[Padishar]

1 hour ago, Oshika said:

If i understand well (because my english is not perfect) i have to use the mod+end function of your memgraph mod to remove the stutter or to reduce it until the devs found a solution/fix that (are they even know this issue ?) ?

Yes, you understand.  The devs do know about the issue, they have been making significant improvements in the amount of garbage the stock game generates in both the 1.1.x and 1.2.x versions.  Mod developers are also, generally, aware of the problem and most have been trying to make similar improvements.

1 hour ago, Oshika said:

I have 16gb of ram, if i put a lot of mods (well, Being careful that the mods are compatible and up to date) can i use the "total" line of padheap.cfg to allocate more ram and be able to remove more stutter ? what value should i use ? 8192 ?

Yes, you can simply adjust the total line to make the stutters further apart.  On a 16 GB machine with a lot of mods I would recommend not going much above 4096 or so, as many mods also use up memory for textures and models which are kept outside the Mono heap.  4 GB of padding should put the stutters pretty far apart unless your game is generating a really huge amount of garbage.  I would actually recommend starting with 2048 and see what effect it has, then increase it if necessary...