Stutter (with mechjeb)

[Kerbital]

I had the worst gameplay today. I capped my framerate at 60, turned off AA and highlighting and the stutter today was unbearable, probably worst so far. It's not even "stutter", it's a solid 1-2 second pause every few seconds when even flying a small craft. It gets progressively worse, like something is building up, and it happens in the hangar, in map mode and even during menus. I'm also noticing extended loading times when switching between ship, map and space center. Alt-End isn't making any difference anymore.

What do I need to adjust in the settings to give it more memory? Thanks!

8        : 1
16        : 1
24        : 1
32        : 1
40        : 1
48        : 1
64        : 1
80        : 1
96        : 1
112        : 1
144        : 1
176        : 1
208        : 1
240        : 1
296        : 1
352        : 1
432        : 1
664        : 0
800        : 0
1008    : 0
1344    : 0
2032    : 0
total    : 1024

Edited November 12, 2016 by Kerbital

[Padishar]

1 hour ago, Kerbital said:

What do I need to adjust in the settings to give it more memory? Thanks!

Just the number on the total line. However, I could do with seeing a screenshot of your graph in a range of situations and how much ram you have and the total memory KSP is using. It sounds like one of your mods is misbehaving and generating loads if garbage.  An output_log.txt file would be handy too...

[Kerbital]

14 hours ago, Padishar said:

Just the number on the total line. However, I could do with seeing a screenshot of your graph in a range of situations and how much ram you have and the total memory KSP is using. It sounds like one of your mods is misbehaving and generating loads if garbage.  An output_log.txt file would be handy too...

I only have MechJeb. Removing and copying a fresh MechJeb folder restores some order. 2-3 hours later stuttering starts creeping in again. So then I deleted all the ships settings files from \Kerbal Space Program\GameData\MechJeb2\Plugins\PluginData\MechJeb2 and things go back to normal again. It looks to me like MechJeb is indeed causing this

Here are some screenshots I took over the last few days (15MB):

https://dl.dropboxusercontent.com/u/14471718/forumstuff/ksp/656425341.zip

Here is the output_log.txt from \KSP_x64_Data

https://dl.dropboxusercontent.com/u/14471718/forumstuff/ksp/6456575.zip

Thanks!

[Padishar]

19 hours ago, Kerbital said:

Here is the output_log.txt from \KSP_x64_Data

This log shows a large amount of spam of this:

[MechJeb2] Warppause : lastAskedIndex=1 CurrentRateIndex=0 WarpMode=HIGH MaxCurrentRate=0

It may only be doing it during timewarp but, even if that is the case, it can't be helping your performance.  This is probably a quirk of the dev build of MJ.

In all of your graphs except for 34 and 35, the Heap Min and Max values are far closer together than I would expect.  This is the main reason your stutters are so frequent.  How much RAM do you have in your machine and how much memory (commit size) does task manager say that KSP is using?

[Kerbital]

I have 32GB. I will need to get back to you later on Task Manager memory usage, because I don't remember, I haven't looked in a while. Funny you mentioned warp as MJ has been very "hesitant" during autolandings and is doing this stop-and-go warp thing as if it couldn't decide what to do: warp or not. I often mess around with ships that are really not designed for proper landings, but this happens a lot even with simple probes and landers. However, this is not the only time stutter happens. It happens during all aspects of the game.

I currently have MechJeb2-2.5.9.0-648 build.

I bumped the total in MemGraph settings to 2048 and it was noticeably better for the rest of the day. This is the same lander I used earlier and it was giving me 5 second stutter. This time it was rather smooth landing. Is it safe to go even higher?  I'll check the RAM usage in Task Manager later and post.

Thank you!

 

[Padishar]

1 minute ago, Kerbital said:

Is it safe to go even higher?

With 32GB of RAM it should be pretty safe to put it way up, 10240 or even higher, especially as you don't have many high memory usage mods installed.  If you do start to see issues, e.g. very slow scene changes or similar, then dial it back a bit...

[Kerbital]

Right now, with 2048 max and few minutes of flying a large ship the commit size is around 5.2GB (5,200,000K) and it's growing.

 

 

 

 

[Kerbital]

I upped the max to 10240. KSP memory usage jumped to 17GB and I don't see much improvement over 2048 so far. It's already much, much better with 2048. At this is with 4x AA. I think we can conclude that:

a) This bug still lingers in KSP.

b) MechJeb is triggering this bug.

c) Increasing the max padding in MemGraph helps to a large degree.

Thank you!

 

[Kerbital]

Yikes, the stutter is back with a vengeance! I counted pauses as long as 3 seconds each! KSP is now using 20GB of RAM On top of that I have insane loading times, switching between  map, ship or space center take over 10 seconds! I think I'm done here. Thanks for all the hep but I lost patience for this game. Cheers!

[Kerbital]

Just popped in here to say that the stutter is still here after 1.2.2 update and the latest official release of MechJeb 2.6

[steve_v]

1 hour ago, Kerbital said:

Just popped in here to say that the stutter is still here after 1.2.2 update and the latest official release of MechJeb 2.6

Yeah, not much has changed. I'm sure SQUADs "garbage reduction" efforts are nice and all, but this BS performance is what has killed my career saves over and over. The best bit is that it gets progressively worse as you have more craft in flight, effectively killing any chance for an enjoyable end-game.
Blaming mods (e.g. mechjeb) is all well and good, and often it is indeed mods creating garbage... but the simple fact is that the runtime sucks. What is the point of a fancy memory-managed modding language if using it as intended (without jumping through hoops, and avoiding half the features) murders performance?
We will remember you, KSP: A beautiful concept, crippled by it's shoddy game engine. Such has been the state of things since day one.

[cantab]

To be fair, "memory managed" languages don't absolve you of the need to write sensible code or to have some understanding of how the memory management works. While it might not always be as obtrusive as in Unity, if your code is spewing garbage like mad it's liable to give the GC more work to do and therefore slow things down.

[steve_v]

1 hour ago, cantab said:

While it might not always be as obtrusive as in Unity, if your code is spewing garbage like mad it's liable to give the GC more work to do and therefore slow things down.

This is true, however my point is that in Unity it is obtrusive. The Boehm-Demers-Weiser algorithm, while simple to implement, is absolutely terrible for real-time performance-critical applications... like games.
Unitys runtime sucks because it's ancient, and its simple GC algorithm is not suitable for games. It's not even multi-threaded, something mainline mono has had for years.
Writing better code is only part of the solution, we also need a better GC... like Sgen, included in any recent version of mono (and experimental in the version Unity uses, before unity messed with it). It's not perfect, but it's a whole lot better than what we have now, and it's liable to dramatically improve this situation.

Edited January 10, 2017 by steve_v

[Padishar]

2 hours ago, steve_v said:

and it's liable to dramatically improve this situation.

[citation needed]  Specifically, what possible reasons do you have to believe that SGen would cope better with stock KSP's garbage creation, let alone a situation where mods are generating over 200 MB/s...?

[steve_v]

31 minutes ago, Padishar said:

what possible reasons do you have to believe that SGen would cope better with stock KSP's garbage creation

As it's not available in Unity, I clearly cannot benchmark it to provide hard numbers - hence "liable to"  rather than "will".
Every benchmark or comparison I can find (for reasonably complex workloads) shows monos sgen significantly outperforming the old boehm implementation, and coming pretty close the performance of .NETs GC.
What makes you think that this won't apply to KSP as well? Surely there is something to be gained with a generational, multithreaded GC - even a minor improvement in the time taken for a collection run is worthwhile here.
If you can keep the allocation rate low enough (and the nursery size large enough), you might get away with very few full collections between scene changes.

31 minutes ago, Padishar said:

mods are generating over 200 MB/s

If mod code is generating 200MB/s, I doubt any GC will deal with it without a significant performance overhead... But I'm not seeing anything like that rate, and I would be surprised if this was the norm.
What I am seeing is noticeable stalls starting at 1/10th of that allocation rate - and it's the time these last that makes it a problem.

Edited January 10, 2017 by steve_v

[Padishar]

3 hours ago, steve_v said:

significantly outperforming

What does this mean?  What were they measuring?  Overall runtime, proportion of time spent doing GC, length of individual GC runs, or something else?

3 hours ago, steve_v said:

Surely there is something to be gained with a generational, multithreaded GC

Yes, I'm not disputing that, there is likely to be some sort of "improvement" when looked at in the "right" way but there's no guarantee this will translate to any improvement in game performance or the gameplay impact of any stutter that does still occur.

The main problem with a generational collector for KSP is that there are quite a lot of objects allocated that live for a considerable period, in some parts of the code, enough is allocated to fill any reasonably sized nursery.  On a nursery collection, any object in the nursery that still has references gets physically copied into the next generation so the nursery has to be kept reasonnably small to avoid the possibility of having to copy too much in one nursery collection.  Large blocks (and I'm not talking that large in KSP terms) skip the nursery altogether, further eroding the theoretical performance benefits.

3 hours ago, steve_v said:

even a minor improvement in the time taken for a collection run is worthwhile here.

This doesn't sound much like:

5 hours ago, steve_v said:

it's liable to dramatically improve this situation

You have to be making quite a chain of assumptions to be able to relate random benchmarks to the perceived seriousness of the stutter in KSP.

3 hours ago, steve_v said:

If mod code is generating 200MB/s, I doubt any GC will deal with it without a significant performance overhead... But I'm not seeing anything like that rate, and I would be surprised if this was the norm.

I'm not saying it's the norm but it does happen.  Just check out the last few posts in the MemGraph thread for an example of what a (presumably) dodgy mod can do...

3 hours ago, steve_v said:

What I am seeing is noticeable stalls starting at 1/10th of that allocation rate - and it's the time these last that makes it a problem.

Even 20 MB/s is a ridiculous allocation rate, what on earth (or Kerbin) is it being used for?  (That's rhetorical, by the way ).  If the heap only has 100 MB of headroom then this will fill up in 5 seconds.  With the current collector, a significant improvement in the length of the stalls would probably be got by multithreading the mark phase but you've seen that doesn't work in Unity and I'd be wary of it in any future version of Unity using SGen too due to the complexity of Unity's threading usage.  With SGen, the frequency of the stalls is likely to improve some due to the nursery reducing the allocation rate in the next generation but exactly how much can't even be ball-park estimated at the moment and I certainly don't expect the length of the stalls on a full collection to improve that much all the time there are several hundred megs of old generation to scan through.

6 hours ago, steve_v said:

Writing better code is only part of the solution, we also need a better GC...

The thing is that writing good code is required whichever GC is used but if the code were good enough then a better GC would give very little benefit at all...

[steve_v]

18 minutes ago, Padishar said:

What were they measuring?

Overall run time / time per operation against object allocation rate.

13 minutes ago, Padishar said:

With the current collector, a significant improvement in the length of the stalls would probably be got by multithreading the mark phase but you've seen that doesn't work in Unity and I'd be wary of it in any future version of Unity using SGen too due to the complexity of Unity's threading usage.

Yes, I have managed to break GC entirely by enabling parallel mark. This working with Sgen would be one of the requirements for "dramatic improvement".

9 minutes ago, Padishar said:

Even 20 MB/s is a ridiculous allocation rate

I know. But what are we going to do, line all the devs up and flog them?

11 minutes ago, Padishar said:

if the code were good enough then a better GC would give very little benefit at all...

But it isn't...

[Kerbital]

It definitely gets worse with time as you have more ships in flight. I played the game again for few days and it eventually gets to the point of being impossible to play. I had 4 relays in space, one large ship, two landers and that's pretty much it. I can't develop any real infrastructure or mount a proper expedition or build bases because the game becomes unplayable. This is really very frustrating as the game could be so much fun otherwise.

I'm limited basically to tinkering with small ships in Kerbal vicinity.

The other game I've been playing a lot is Planetary Annihilation. I play against up to 6 AI opponents across several planets and moons, thousands of units each at times. The game used to slow down a lot, the simulation speed and FPS (but there was never any stuttering), there were lots of complaints, but they managed to improve it so much with the latest update that it's like day and night difference. Although, I believe they're using a custom, in-house engine. No more slowdowns for me. I was hoping for a similar "miracle" with KSP.

Edited January 10, 2017 by Kerbital
Added stuff

[sal_vager]

This particular stutter issue may be due to Mechjeb, so I've split it to Modded support, have you thought to ask @sarbian about this issue yet?

[steve_v]

Poked him some (long) time back, but never really got a quantifiable answer. Without an easy/reliable way to stick it to any particular mod I'm disinclined to hassle more, Sarbian can be a little, er... irritable at times.

[Padishar]

7 minutes ago, sal_vager said:

This particular stutter issue may be due to Mechjeb, so I've split it to Modded support, have you thought to ask @sarbian about this issue yet?

The messages since this one:

...were about the general GC issue rather than anything MechJeb specific and shouldn't really have been moved.

Sarbian was informed about the odd garbage creation profile of MechJeb a few weeks ago but I don't think he's got around to looking at it yet.

Edited January 10, 2017 by Padishar

[sal_vager]

I thought it best to start with Kerbitals issue, as he seems to be using Mechjeb throughout, the GC we already know about and can't change, roll on Unity and a Mono update...

[sarbian]

51 minutes ago, Padishar said:

Sarbian was informed about the odd garbage creation profile of MechJeb a few weeks ago but I don't think he's got around to looking at it yet.

Err, I need to object on the implication of that sentence for those who did not follow the whole mod memory usage in KSP since the start. I know that MJ is a garbage generator monster since a very long time. Case and point : When MJ used KER dV code for a few month (around April 2015)  I implemented a lot of code change to lower the memory use of that part of the code (and AFAIK the current KER dv code is heavily influenced by those changes). I kept improving the memory use of MJ since (Here, here, Landing AP, Core, New Fuel Sim,  and many others).

I would also remind everyone that brought the Unity profiler to modding (It used it long before but since it required a Pro licence before U5 there was not point in sharing it much) and wrote a mod to display the GC usage in 2014...

 

51 minutes ago, steve_v said:

 

Poked him some (long) time back, but never really got a quantifiable answer. Without an easy/reliable way to stick it to any particular mod I'm disinclined to hassle more, Sarbian can be a little, er... irritable at times.

 

I doubt I did not give you a quantifiable answer since as I just explained I have know about the problem and worked on it for nearly 2 years and an half.

 

Now to come back to the original subject : yes, Mechjeb does create a lot of memory Garbage. I improved a lot of the code that does it but the main problem is still here : the UI. MJ uses the old UI system and that system is just plainly terrible in memory use. I plan to change it but I don't want to rush it and end up with a system I don't like. So in the mean time MJ does create some more memory garbage than some other mods. And far less than other.

 

 

Edited January 10, 2017 by sarbian

[Padishar]

3 minutes ago, sarbian said:

Err, I need to object on the implication of that sentence to those who did not follow the whole mod memory usage in KSP since the start.

I certainly wasn't trying to imply that you had done nothing to improve the garbage creation of MechJeb, other mods or KSP itself, just that you haven't yet (as far as I know, the MechJeb thread disappearing hasn't helped ) looked at the specific issue I reported a while ago with the significant spike that happens every 5 seconds...

[sarbian]

I think I fixed that one. The last months are a bit blurry

Edited January 10, 2017 by sarbian