Garbage Collection Stuttering got worse in 1.1.1

[Festivejelly]

Unity seems to handle other things fine. My bet is that something from "the old days" of ksp is causing it. Theres no way the gc should take so long to clean its junk. Its probably how the game objects are created and managed by ksp.

regardless someone should be working on it code can always be optimised. Having the impact of this as "low" on tge bug tracker concerns me.

[Broco]

27 minutes ago, Festivejelly said:

Unity seems to handle other things fine. My bet is that something from "the old days" of ksp is causing it. Theres no way the gc should take so long to clean its junk. Its probably how the game objects are created and managed by ksp.

regardless someone should be working on it code can always be optimised. Having the impact of this as "low" on tge bug tracker concerns me.

The problem isn't how long the GC take to clean out the junk, it's how often it needs to do it. The time it takes is perfectly normal.

[Stoney3K]

3 hours ago, steve_v said:

Don't hold your breath. Others, myself included, have spent considerable time tracking down the cause of this (there's even a mod just for the purpose - kinda indicates it's a widespread issue, no?) and all we've heard from Squad so far is head-in-sand.

People, don't forget, Squad will remain "head-in-sand" (I suspect they have other pieces of work to attend to as well) if we keep bashing Squad for every single problem we have with KSP. If they release too late, people will complain that it takes too long, if they release too early, people will start complaining about bugs being unfixed.

Squad can never do the right thing when the nagging message board users are concerned. Please leave your sense of entitlement at the door and focus on the technical issue that may or may not be present. That is information that is valuable to Squad.

[Broco]

13 minutes ago, Stoney3K said:

People, don't forget, Squad will remain "head-in-sand" (I suspect they have other pieces of work to attend to as well) if we keep bashing Squad for every single problem we have with KSP. If they release too late, people will complain that it takes too long, if they release too early, people will start complaining about bugs being unfixed.

Squad can never do the right thing when the nagging message board users are concerned. Please leave your sense of entitlement at the door and focus on the technical issue that may or may not be present. That is information that is valuable to Squad.

I really didnt try to bash somebody but this is a problem that has been around for ages. The thing is that every person has different preferences or things that are important for them. E.g. some people complain about the wheels, some want more features, some complain about krakens.

I don't really understand what you want to achieve with your post because "focussing on the technical issue" is something we could only do if KSP was open source. As much as details of the problem go, I think Squad has enough information by now to adress this issue.

As a developer you have to decide which things to adress with your limited time and we just want to achieve that they focus more on stability and performance before new stuff is introduced. E.g. I don't care about the wheel problem, you can work around it. But I can't do nothing about my stuttering except deleting half of my flights which I don't want to do.

[Stoney3K]

5 minutes ago, Broco said:

I really didnt try to bash somebody but this is a problem that has been around for ages. The thing is that every person has different preferences or things that are important for them. E.g. some people complain about the wheels, some want more features, some complain about krakens.

I don't really understand what you want to achieve with your post because "focussing on the technical issue" is something we could only do if KSP was open source. As much as details of the problem go, I think Squad has enough information by now to adress this issue.

I'm getting a bit fed up with people that whine about SQUAD doing everything wrong. If they release the 1.1 update too early, people complain about it being buggy and being released too soon. If they decide to delay the release to work out the bugs with the community through the prerelease, people are acting like spoiled brats because they feel entitled to play 1.1 even when it's not even done. Instead of getting support from the community, they're putting SQUAD between a rock and a hard place because they can never make the right decision. They will always alienate 50% of their community of paying customers.

If we want to focus on the technical issue, start by putting a debugger on your KSP process and profile the living excrements out of it. A debugger can provide more information about the call stack than GCMonitor can, and can even trap the processes that generate the most garbage and thus trigger the garbage collector. Clearly there is a memory leak somewhere, but unless someone can positively confirm it is not caused by any mod (and yes, GCMonitor is a mod and it uses the legacy UI, and could therefore also be suspect)

My flights have not been as elaborate so far so I have not hit any performance wall, so I won't be of any help when profiling.

[Galileo]

4 hours ago, Festivejelly said:

@vgamble what spec machine you using?

mines:

windows 10

gtx 970

16gb ram

3.5 ghz i5

I just tested out the career save with absolutely ZERO stuttering. We have nearly the same specs,  except I have 32 gb of ram and I overclock my CPU and GPU. I had graphic settings turned all the way up on everything. Kind of weird that it only effects some and not others. 

[JedTech]

12 minutes ago, Stoney3K said:

If they release the 1.1 update too early, people complain about it being buggy and being released too soon. If they decide to delay the release to work out the bugs with the community through the prerelease, people are acting like spoiled brats because they feel entitled to play 1.1

Squad could complain about the complainers but that wouldn't help their business. A much better way to handle this would have been a long open prerelease available to all users.

[Stoney3K]

6 minutes ago, Berlin said:

I just tested out the career save with absolutely ZERO stuttering. We have nearly the same specs,  except I have 32 gb of ram and I overclock my CPU and GPU. I had graphic settings turned all the way up on everything. Kind of weird that it only effects some and not others. 

Can you check the memory usage in Task Manager for that save? Looks like the KSP process may just be hitting a memory wall and trips the garbage collector to prevent it from swapping.

[HebaruSan]

9 minutes ago, Stoney3K said:

I'm getting a bit fed up with people that whine about SQUAD doing everything wrong.

I hope this doesn't count as back seat moderating, but let's all try to stay focused on technical topics rather than people. Characterizing someone else's posts or defending your own posts is unlikely to go anywhere constructive.

9 minutes ago, Stoney3K said:

They will always alienate 50% of their community of paying customers.

I think this will be less true next time, if there is one. With SQUAD's expressed intent to include store customers in future pre-releases and Jesus bringing the patcher back from the dead recently, 100% of players should have the option to choose between stable and unstable versions (well, except GoG customers, who apparently only get fully stable versions if recent posts are to be believed). That should help to relieve the pressure for a too-early release.

Everything I think I understand about the stuttering problem so far suggests that it is not the kind of thing you can fix all at once; rather, you get one clear report of it happening in a specific case, investigate and fix that case, then repeat for other reports, until eventually things clear up. So I second your suggestion of using profilers to pinpoint where the excess garbage comes from. I'll also repeat my previous suggestion to try to get per-mod measurements so 1) stutter-sensitive players can make more informed decisions about which mods to use, and 2) interested modders can be notified of such problems. We need more granular measurements than "unmodded is fine but modded is stuttery" if things are going to get fixed.

[Broco]

49 minutes ago, Stoney3K said:

I'm getting a bit fed up with people that whine about SQUAD doing everything wrong. If they release the 1.1 update too early, people complain about it being buggy and being released too soon. If they decide to delay the release to work out the bugs with the community through the prerelease, people are acting like spoiled brats because they feel entitled to play 1.1 even when it's not even done. Instead of getting support from the community, they're putting SQUAD between a rock and a hard place because they can never make the right decision. They will always alienate 50% of their community of paying customers.

If we want to focus on the technical issue, start by putting a debugger on your KSP process and profile the living excrements out of it. A debugger can provide more information about the call stack than GCMonitor can, and can even trap the processes that generate the most garbage and thus trigger the garbage collector. Clearly there is a memory leak somewhere, but unless someone can positively confirm it is not caused by any mod (and yes, GCMonitor is a mod and it uses the legacy UI, and could therefore also be suspect)

My flights have not been as elaborate so far so I have not hit any performance wall, so I won't be of any help when profiling.

Dude are you kidding me? There was even a MOD written to investigate this issue, how much more technical help do you expect from the community, Squad know EVERYTHING regarding this bug, they don't need logs, they know exactly what causes it they just don't think it's as urgent as we think it is.

And this threat is not about "whining" about a release coming to early or to late this issue has been around for countless versions now, for lots of people. If you're to lazy to read about a topic, don't comment on it.

Again: This issue is well documented, affects the majority of users (after a certain amount of simultanious flights) and Squad know about it and it is known how to fix it.

33 minutes ago, Stoney3K said:

Can you check the memory usage in Task Manager for that save? Looks like the KSP process may just be hitting a memory wall and trips the garbage collector to prevent it from swapping.

And please stop making comments about it if you don't know what you're talking about. "Memory wall."

 

Oh and before you start a excrementsstorm: I hate being put into a pool with people of any kind. I experienced this issue and found out more people have it and it is known for a very long time. I provided info to the bug tracker, I found out that many mod creators like @Padishar helped investigating it, too and this has nothing to do with "whining". Also you should check who you are replying to before you make comments. I'm not a person that flames in the forum for every little bug. You could've found that one out by checking my profile, see my join date and check my post count.

I've been with KSP since the very beginning and I will not reply to any more comments going the direction of questioning the need to report issues.

Edited April 29, 2016 by Broco

[Stoney3K]

7 minutes ago, Broco said:

Dude are you kidding me? There was even a MOD written to investigate this issue, how much more technical help do you expect from the community, Squad know EVERYTHING regarding this bug, they don't need logs, they know exactly what causes it they just don't think it's as urgent as we think it is.

And this threat is not about "whining" about a release coming to early or to late this issue has been around for countless versions now, for lots of people. If you're to lazy to read about a topic, don't comment on it.

Again: This issue is well documented, affects the majority of users (after a certain amount of simultanious flights) and Squad know about it and it is known how to fix it.

And please stop making comments about it if you don't know what you're talking about. "Memory wall."

I'm not saying that we don't have the tools to provide Squad with the information they need. My point is that, by claiming that SQUAD is ignoring the issue, the chances of this issue getting resolved will diminish rapidly. Squad never denied that the stutter issue exists (just as it never denied any problems with the ocean terrain rendering). At this point, the performance problem is just too specific for user's systems that they can't reproduce the conditions under which it happens.

I'm saying: Stop claiming to know what the response of SQUAD is, and stop being judgmental about SQUAD. If the issue was easy to fix, they would have fixed it already.

The fact that a certain save, in a fully stock game, with exactly identical system specifications, aside from installed memory, produces the issue in one case but not in the other, is a red flag that the program is in fact hitting a "memory wall" -- being notified by the system that memory (or more specifically, allocated address space) is running short and it needs to clean stuff up. In Unity and in any other .NET program, this will trigger the garbage collector. Squad cannot do anything about this happening, and even if they could prevent this enforced garbage collection, it would just mean the program crashes because it runs out of memory.

The garbage collection being triggered is a symptom of being short on memory, and there are a few things Squad can do about that, to begin with, be more efficient with resource management and plug memory leaks, but that is something that affects major parts of the code.

If the issue is such well-documented and reproducable, feel free to make a decent bug report about it. That gets it on the Squad development pipeline.

http://bugs.kerbalspaceprogram.com/projects/ksp/issues

[Broco]

I will no longer reply to your comments because you obviously don't read what I write. Case closed.

[HebaruSan]

(This post removed because I misattributed one of the quotes involved.)

Edited April 29, 2016 by HebaruSan

[Stoney3K]

12 minutes ago, Broco said:

I will no longer reply to your comments because you obviously don't read what I write. Case closed.

I hope you understand that your condescending tone towards other forum users and towards SQUAD (I am talking about sarcastic remarks in your very own thread start) is not really helpful for your case? If someone is wrong on the internet, help them along, Ad-hominems are really not very helpful.

Even though you do have a valid point, such a tone will easily place you in the category of people just bashing SQUAD because they won't listen. I've read up on what @Padishar was investigating in the resource system, and the symptoms do match the issues described here, as well as the issue not appearing on identical systems, save for more memory. Point is, Squad can do nothing about the garbage collection happening, except advise users to install more memory, the only thing they can address is reduce the amount of garbage being created.

The fact that KSP is programmed on .NET is a bit of a strike one, though. As far as I know from the internals of C#.NET, there is no function that allows you to explicitly get rid of a piece of data that you allocated or that was handed to you by another function call. So any data that is being used and is not declared locally on the stack, causes garbage, and that includes function parameters which are called by value. The only "fix" for that is to call as many function parameters as possible by reference, especially if they involve large pieces of data, but that is very invasive for KSP as well as a mod-breaking operation.

[steve_v]

10 hours ago, Stoney3K said:

The fact that a certain save, in a fully stock game, with exactly identical system specifications, aside from installed memory, produces the issue in one case but not in the other, is a red flag that the program is in fact hitting a "memory wall" -- being notified by the system that memory (or more specifically, allocated address space) is running short and it needs to clean stuff up.

I have no knowledge of this "memory wall" term either. Can you point me to the documentation regarding this "notification by the system" mechanism? Don't know about Windows, but AFAIK the Linux kernel won't do anything special unless memory is actually under pressure - I have ~3GB real RAM free and ~5GB disk cache at this point. Process virtual address space on Linux/AMD64 is something like 128TB IIRC, so I doubt that has anything to do with it either. You're welcome to prove me wrong though.

Note that that save does produce the same stuttering for me, on different hardware (only commonality is Intel/Nvidia, someone with AMD, please pipe up) and operating system.

11 hours ago, Stoney3K said:

My flights have not been as elaborate so far so I have not hit any performance wall, so I won't be of any help when profiling.

Have you tried Brocos save yourself? That should be all you need.

Edited April 30, 2016 by steve_v

[Jayad]

Nicely put together Broco!

Posting here to add in my frustration with the stutter too.  I have an uptodate machine: win10, Nvida, 3.5ghz, 16g ram  and the stutter is present with mods stripped out running only 1 flight.  Another thing about mods, i dont buy that its all their fault.  Other games have great frameworks for modders, even if a mod is written very baddly usually only causes fps drops not memory management issues.

I would very much welcome Squad give this issue some attention.  We have a lot of content now and i firmly believe that shifting to quality execution / better visuals strategy would really bring in more players and money.

 

[Festivejelly]

19 hours ago, Broco said:

The problem isn't how long the GC take to clean out the junk, it's how often it needs to do it. The time it takes is perfectly normal.

I guess what I mean is the GC takes longer to perform said "trashing" when there are more objects to clean up. I've timed it between iterations, however as you say the more it needs to run higher the intervals which makes the stuttering even more pronounced.

I found that with stock with just 1 ship I see it run around ever 13 seconds, but when it does run (looking via GC monitor) I cant notice the stutter.

If I make a couple of craft and put them in orbit and observe it, again it takes about 13 seconds between each collection but I do notice the stutter the frames seem to drop lower. My assumption is that it has to clean up more game objects that are being created.

This is stange behaviour because if im not focused on that craft then the other crafts should just be in a pool of objects and wont need to be garbage collected.

Seems to me that in every Update() something is creating a bunch of new objects that need to be cleaned. This just sounds a bit fishy to me. Objects should be created between scenes rather than in the Update() method.

15 hours ago, Stoney3K said:

I hope you understand that your condescending tone towards other forum users and towards SQUAD (I am talking about sarcastic remarks in your very own thread start) is not really helpful for your case? If someone is wrong on the internet, help them along, Ad-hominems are really not very helpful.

Even though you do have a valid point, such a tone will easily place you in the category of people just bashing SQUAD because they won't listen. I've read up on what @Padishar was investigating in the resource system, and the symptoms do match the issues described here, as well as the issue not appearing on identical systems, save for more memory. Point is, Squad can do nothing about the garbage collection happening, except advise users to install more memory, the only thing they can address is reduce the amount of garbage being created.

The fact that KSP is programmed on .NET is a bit of a strike one, though. As far as I know from the internals of C#.NET, there is no function that allows you to explicitly get rid of a piece of data that you allocated or that was handed to you by another function call. So any data that is being used and is not declared locally on the stack, causes garbage, and that includes function parameters which are called by value. The only "fix" for that is to call as many function parameters as possible by reference, especially if they involve large pieces of data, but that is very invasive for KSP as well as a mod-breaking operation.

KSP isnt programmed in .NET.

Its made in Unity which uses Mono which is not as up to date as .NET.
Mono is basically an Open source version of it that was intended to be used for all platforms, such as mac, linux, windows etc.
It can run .NET applications but it certainly isnt .NET

[Festivejelly]

15 hours ago, Stoney3K said:

Point is, Squad can do nothing about the garbage collection happening, except advise users to install more memory, the only thing they can address is reduce the amount of garbage being created.

 

This is wrong in so many ways.
It doesnt matter how much memory you have, the GC will still run. The trick is to limit the creation of garbage so that the GC doesnt need to run as frequently.
Limit the creation of objects to switching between scenes, stop concatenating strings and adding to collections etc.
This kinda stuff needs to be done before a scene loads. They just need to be a bit cleaverer (that a word?) on how they do this. Its an easy trap to fall into with a framework like Mono or .NET in that you're just relying on the GC to take care of stuff for you. But really with things like games it really does need some attention and you know Broco is right, its almost as if its just been cast of as some hocus pocus stuff that doesnt matter, when actually its starting to become a real problem.
Also what compounds the problem is people like you who dont really understand whats going on trying to weigh in on the issue. Its just not helpful at all.

[Stoney3K]

19 minutes ago, Festivejelly said:

Seems to me that in every Update() something is creating a bunch of new objects that need to be cleaned. This just sounds a bit fishy to me. Objects should be created between scenes rather than in the Update() method.

Its made in Unity which uses Mono which is not as up to date as .NET.
Mono is basically an Open source version of it that was intended to be used for all platforms, such as mac, linux, windows etc.
It can run .NET applications but it certainly isnt .NET

For the sake of argument, Mono is compatible with the .NET CLR which is responsible for garbage collection. There is no way around the garbage collection aside from moving to another platform (which is impossible).

The problem is caused by the Update() calls generating a lot of garbage, in particular, as @Padishar pointed out, the part.RequestResource() which is used by all PartModules that interact with the resource system (and that's a lot). Each PartModule will call at least one resource request per physics update, which creates a garbage object. Part.GetConnectedResources() is even worse since it returns a list of available resources, without any means of destroying them except garbage collection.

Multiply the numerous calls to the resource system by a lot of parts, in a lot of vessels that are simulated on rails, and garbage adds up really quickly.

I agree with you, that the action of moving resources through the ship is nothing more than adding and subtracting numbers. It's a completely linear operation, and should not requre any creation or destruction of new data objects which represent a single "slice" of resource that is used in that frame.

Maybe a good solution would be to generate a resource I/O matrix for any craft that goes on rails, and simulate from there. After all, the only thing that will happen is resources being generated (through, for example, solar panels), being stored in fuel tanks or batteries, being processed in fuel cells or ISRUs (which consume one resource and generate another, in any given ratio), and being consumed by engines. One will add a fixed amount to the total available resource, the other will subtract. No need to create or destroy data objects.

I'm not sure if the "original" .NET framework from Microsoft is more efficient when it comes to the garbage collector, but Squad would be unable to move over anyway. That's a decision that the Unity designers will have to make.

Edited April 30, 2016 by Stoney3K

[rkman]

On 29-4-2016 at 9:51 AM, Sirad said:

http://bugs.kerbalspaceprogram.com/issues/4051

This is (i guess) the related bug. The Priority is low, it'l may never get fixed.

Priority is low presumably because it can't be fixed in a reasonable amount of time without the cause in Unity being addressed first.

[Stoney3K]

31 minutes ago, Festivejelly said:

Also what compounds the problem is people like you who dont really understand whats going on trying to weigh in on the issue. Its just not helpful at all.

I think what compounds the problem is that people are under the impression that Squad is closing their eyes on this issue which makes the game unplayable for them. Maybe in your opinion, Squad should put out a statement that this bug is being addressed or something?

For 1.0.5 the biggest problem was high part count ships, now that has been fixed by moving to an entirely new game engine (no easy feat) and instead of applauding, people complain about the newer performance-breaking issue which pops up when they have hundreds of vessels in their save. Don't you agree that that may come off as a bit demanding?

I only play save games with a few dozen missions going at once and I have never seen any garbage collection stutter happening, even with just 4GB of RAM. I've never seen the gameplay need for having more vessels in flight, if a mission served its purpose, I will return or terminate it.

[Padishar]

3 minutes ago, rkman said:

Priority is low presumably because it can't be fixed in a reasonable amount of time without the cause in Unity being addressed first.

The cause is not in Unity. The pauses are caused by the Unity/Mono garbage collection but it only runs as frequently as it does because KSP code generates excessive amounts of garbage. Allocating and discarding 120+ mb every few seconds is ridiculous. If this rate can be significantly reduced then the garbage collections will happen much less often and can also be deliberately triggered at points the user won't notice to reduce them still further.

The truth is that it can't be fixed in a small amount of time regardless of what does or doesn't change in Unity. Some of the low hanging fruit (e.g. The resource system) can be addressed fairly easily but more significant gains would require an audit of the entire code base  and considerable changes to many parts of it.

[Broco]

At least @SQUAD is aware of this issue as you see in the patch notes of 1.1.2:

Quote

* Remove some garbage creation in Part.Update.

 

19 minutes ago, rkman said:

Priority is low presumably because it can't be fixed in a reasonable amount of time without the cause in Unity being addressed first.

Yes and no. I'm with you on the point that I think it can't be done in a short amount of time. I mean every programmer knows this issue: You write your program and when you're finished you start fixing bugs, adding features, etc. and then you find out that you handled something substantial wrong in a class that is referred to by 90% of the rest of the code and you can't just rewrite this without changing how this class (or a function within it) is called by other functions. This mistake causes an error in some special cases you didn't think of in the beginning.

OR: You used the wrong way of addressing something in every single function that addresses something. In both ways you have to put a lot of effort into fixing something that doesn't cause many problems in 95% of the cases but in 5% it has ugly consequences.

So regarding KSP basically there's 2 options:

1) Wait till an engine release comes out that fixes or optimizes it.

2) Rewrite all those garbage creating functions.

If option 1 would be happening in a reasonable and especially fixed amount of time nobody would complain about it. "It's a bug that is gonna get fixed in 4 months." I could live with that. It's a long time, yes but in this case it wouldn't make any sense to put manpower behind a problem that is going to fix itself in the future. But sadly this is not the case here.

[Padishar]

4 minutes ago, Stoney3K said:

which pops up when they have hundreds of vessels in their save.

This issue affects some people so badly  that they do not enjoy playing even in a stock game with a single vessel. They haven't just started complaining about this now, it has been a known issue since I started playing in 0.23 and has generally got worse in every new version. This is not surprising given the nature of the problem and the underlying cause, every new bit of code that gets written has the potential to generate more garbage. 

[Broco]

27 minutes ago, Padishar said:

Allocating and discarding 120+ mb every few seconds is ridiculous.

158 MB of garbage in 6 seconds. And this is just mainly because of resources calculation. But I guess I'm just whining.

 

 

 

Edited April 30, 2016 by Broco