Slight freeze every 5 seconds (or so)

[katateochi]

Some interesting developments in here guys, nice 1! Unfortunately I've not had any time for KSP these last few days (and won't till Sunday now), but when I do get some time I'll see if these suggestions improve my game and report back.

[antilochus]

Some disappointing results as I am testing more.

I am almost positive it is the GC causing the stutter; I got GCMonitor and the stutters coincide with the GC sawtooth. So by fiddling with the framerate, I think I was minimizing the impact of it, because from some quick testing, the frequency of the garbage collection is linked with the framerate. 30fps = 17s stutter, 72fps = 11s stutter, 144fps = 6s stutter.

The section at the end of this is interesting and would suggest there are a few ways of triggering GC: http://docs.unity3d.com/Manual/UnderstandingAutomaticMemoryManagement.html

Someone with more than an arts degree like me should really try to patch something in (not easy to do, from my reading) that slows down GC so that people with x64 and big RAM could both cap frames at 30fps and go for long stretches before getting hit.

edit:

Spent a few more hours digging into this last night, I am quite convinced this is GC related. I found some code on github called "GCThrash" (https://github.com/Gerry1135/GCThrash) that I compiled, that seems to purposely fill up the memory and speed up the rate of GC, if you push it, it will grind KSP to a halt with the same stutter effect in GCMonitor that we are seeing.

From reading a TON of unity developer discussions / blog posts, seems like it is 1) partially Unity's fault by using a very old version of Mono (2.6.5 circa 2010) as compared to newer versions of Mono which have more advanced GC method and 2) partially Squad's fault as there are many workarounds and efficiencies that developers use to avoid this.

When GC occurs, instead of the frame being generated real time at whatever current fps is (e.g. 1/60s, 1/144s) it may take as long as 1/10s (unclear why it takes so long, but general opinion is that the implementation in Mono 2.6.5 is HORRIBLE). This would explain why higher end systems may be impacted more as it is less frequent at lower frame rates AND it is less perceptible if fps is 20-30 etc.

For those who do not have the stutter issue at all, it would be helpful to grab GCMonitor and watch for the "sawtooth" and see what happens to your fps. It may still be occurring but not perceptible to you.

I'm still at a loss for what we can actually do about it as even if Squad implements Unity 5, it seems to still use Mono 2.6.5 which does not have "generational" based GC tools that are more efficient. Until Unity updates Mono Squad will need to optimize their own code. Alternatively, there may be some difference in hardware which would be interesting to see if there is a user-side workaround.

[note -- again, I have zero programming knowledge so apologies if I am incorrect -- don't often find myself the tip of the spear on technical issues like this.]

Edited June 5, 2015 by antilochus

[Padishar]

The reason the GC rate depends on framerate is that the memory is being allocated and released per frame so the more frames that are rendered the more garbage is created that needs to be collected.

The garbage collection takes so long because it has to do a lot of work to calculate which objects are still referenced and which are not and can be collected. Basically (there are some optimisations but this is the general principle), each time it runs it first marks all objects in the heap as being "unreferenced" and then looks through all the reference variables (both global and on the stack) and marks those objects as "referenced". For each object that is marked as referenced it then scans through the whole object and any object references it contains are also marked as referenced and this continues until all the objects that can still be reached by code have been marked. Then it runs through all the objects that are still marked as unreferenced and cleans them up.

KSP could probably be "fixed" so that very little, if any, garbage was created per frame but it would be a difficult job. I believe the current Unity UI framework (GUI/GUILayout) itself is responsible for quite a lot of the current garbage but that the new UI Squad is working on for Unity 5 should be much better in this respect so, even though the same garbage collector is used, the Unity 5 version should improve things as the collector shouldn't need to run as often.

[danfarnsy]

Antilochus,

Thanks for running this down.

I'm having the same issue, even with small part-count craft. The slower rise in the first graph is where I paused the game. The second image is when I switched to the KSC, where the sawtooth was much slower.

http://imgur.com/a/WePDj

I have a large number of mods running in 64-bit Linux, with 5 GB of memory use for KSP. I'm trying to figure out how to force the framerate lower with the NVidia settings in Linux.

[Padishar]

Hmmm, that garbage rate looks very high. Can you try with all the MJ windows (and any other mod-provided UI) closed?

[antilochus]

Antilochus,

Thanks for running this down.

I'm having the same issue, even with small part-count craft. The slower rise in the first graph is where I paused the game. The second image is when I switched to the KSC, where the sawtooth was much slower.

http://imgur.com/a/WePDj

I have a large number of mods running in 64-bit Linux, with 5 GB of memory use for KSP. I'm trying to figure out how to force the framerate lower with the NVidia settings in Linux.

Your FPS in flight is in the mid 30s? I wouldn't limit the framerate too much lower than that or else the FPS might annoy you more than stutters.

I'd also guess with all the mods running you would be creating more garbage than if you were in stock (as Padishar seems to indicate)

Edit: I checked out my linux64 KSP, it appears you can edit the settings.cfg framerate limit (FRAMERATE_LIMIT) to an arbitrary value and it works pretty well. At 30 fps the impact of the stutter is minimal (but it is still present). I don't think the nvidia drivers have a framerater limiter, or at least I wasn't able to find anything online.

I'm going to fiddle with the physics delta time and see if I can get further improvements. I was doing all my testing in Windows as it is the best documented but I actually want to play in linux64, so I'm going to keep working with it.

Edited June 6, 2015 by antilochus

[katateochi]

Turning the frame limit down (by adjusting FRAMERATE_LIMIT in settings.cfg to 30) does make a slight improvement. I've have a small (35 part) rover parked at KSC which I've been using as a test point for this.

With the FPS limited to 30 the spikes are still noticeable but slightly further apart (about 9 seconds apart), without the limit and with an FPS of around 100, they are 6 seconds apart. So it does make a difference, just not very much.

I'm also trying look at KSP's performance with regard to part count so I've been trying various graphics settings. With regard to this issue, graphics settings do not seem to have any impact on the stutter (or on part count performance for that matter). On max graphics settings or with everything turned right down the stutter is about the same.

I've tried the -nolog setting as well and that doesn't seem to have any noticeable difference.

Clearly this is a GC issue. But is that because unity's GC is not optimal or is it because of how KSP is utilizing objects? I have the sinking feeling that while an improvement to unity's GC will make some difference, it isn't the whole answer.

[antilochus]

From the reading I've done, it is a Unity issue that developers have to work around.

There are dozens of posts on the Unity forums as well as a few independent programmer blog posts discussing mitigation strategies for this. Unfortunately, I haven't found any that offer a solution to end users.

[sal_vager]

I guess we're stuck, I really wish I knew a way to fix this, all I can really do is give you my setup in case it helps you.

Hardware:

A10 5800k AMD APU.

Nvidia GeForce GTX650

4gb DDR3 RAM

Software:

Xubuntu Vivid Vervet 15.04 64bit

Proprietary Linux Nvidia driver 346

Hardinfo report on Hastebin

Settings.cfg

.nvidia-settings-rc

My refresh rate is stuck at 60hz thanks to the driver which may be why I don't see the pausing, and Xfce uses at most 9% of the system RAM, and nothing else I run while playing KSP is using much memory or constantly scanning files.

Edited June 8, 2015 by sal_vager

[Padishar]

I've written another little plugin that displays a graph of the time between calls to Update (the frame time) and the garbage collections.

DLL Download

Source link (MIT licence)

Hit Mod-= to display the window. The full height corresponds to 250ms, a constant framerate should show a constant low green bar. Garbage collections during the frame are indicated by a red bar.

Can people that can't see this effect try running this plugin and just spinning the camera with a pod on the launch pad until a glitch happens and then confirm if you can see the glitch?

[antilochus]

Hardware:

A10 5800k AMD APU.

Nvidia GeForce GTX650

4gb DDR3 RAM

Software:

Xubuntu Vivid Vervet 15.04 64bit

Proprietary Linux Nvidia driver 346.

I can't contain my jealously, honestly, I would pay a lot to have stutter-free performance.

Could you post your settings.cfg and xorg.conf?

[sal_vager]

The *buntu's don't make an xorg.conf anymore and I have not made my own, settings.cfg added to post above.

[antilochus]

The *buntu's don't make an xorg.conf anymore and I have not made my own, settings.cfg added to post above.

Thanks, can you post your ~/.nvidia-settings-rc?

[sal_vager]

Yep, I have not changed much from the defaults by the way, KSP does get slow when the hair shader is active (re-entry heat effect) but that's about it.

[antilochus]

Yep, I have not changed much from the defaults by the way, KSP does get slow when the hair shader is active (re-entry heat effect) but that's about it.

Really appreciate it. I wonder if it's something obscure like HPET (which appears to have helped in other cases).

[Carraux]

My system is a i7-2600k running Win 10 x64 with 8GB RAM and a Geforce 970 and a 2x200GB SSD system disk as RAID0 hosting KSP. It still stutters on my system

IF - and only if - the garbage collector is the reason for this stutter, then the question is: Which part of the program (or Unity) is constantly allocating and freeing memory while the rocket/plane is in midair? Stuttering even occurs when there is no staging. Just flying. So we have a static rocket/plane model which need no allocating/freeing. We have a more or less static landscape model which have some LOD logic but I doubt that this needs constant memory allocating.

The only thing left is... sound. Please listen. To have a constant engine sound KSP has to play the engine sound file repeatedly. But you won't hear any sound repitition from the engines without stutter. The stutter occurs at the end of the sound file.

I assume the program logic looks like: REPEAT -> Allocate memory for sound file -> Play sound file -> Free memory -> REPEAT-UNTIL Enging is off

Instead of: Allocate memory for sound file -> REPEAT -> Play sound file -> REPEAT-UNTIL Enging is off -> Free memory

Could this be?

Squad, do you read?

[sal_vager]

Audio is handled by the Unity3D engine, and changes to how Unity handles audio would require access to source code and breach of the Unity license.

Unity stutters even with simple scenes.

This is a real problem with the Unity3D game engine, and is worse for some than others, all we can do is try to mitigate this as much as possible and hope that Unity5 fixes the issue.

Squad cannot fix this, but are moving to Unity5.

[Majorjim!]

Yep, I have not changed much from the defaults by the way, KSP does get slow when the hair shader is active (re-entry heat effect) but that's about it.

So running on linux is a fix? What is your part count limit? Could you try a 600 part ship?

[antilochus]

So running on linux is a fix? What is your part count limit? Could you try a 600 part ship?

Linux is not a fix for me. I have the stutter noted here in both Linux (LMDE Mate) and Windows (Win7) and it occurs with 1 part ships just as often as large ones.

[Padishar]

IF - and only if - the garbage collector is the reason for this stutter, then the question is: Which part of the program (or Unity) is constantly allocating and freeing memory while the rocket/plane is in midair? Stuttering even occurs when there is no staging. Just flying. So we have a static rocket/plane model which need no allocating/freeing. We have a more or less static landscape model which have some LOD logic but I doubt that this needs constant memory allocating.

There is no "if" about it, this effect is definitely caused by the Unity/Mono garbage collector. Unity itself is pretty good about not creating lots of garbage but various features of the Unity API do make it quite awkward (if not, downright impossible) for KSP (and mods) to do some things without creating unnecessary garbage though KSP and quite a lot of mods do also create quite a lot of other unnecessary garbage themselves that it should be quite easy to fix. In an ideal KSP, the turnover of memory would be low enough so that garbage collections could be deliberately triggered at key points where a frame rate glitch will not be noticeable (scene changes, switch to/from map view, pause game, save etc.) and it wouldn't need to run at any other times. This would be tricky to achieve but it should be quite easy to vastly reduce the rate of garbage collections from where it is now.

The GCMonitor mod (linked somewhere earlier in the thread) allows you to easily see the memory turnover and garbage collections happening. The rate of rise of the line should be much lower than it ever is at the moment, ideally it would only ever rise during flight when something does actually need to be allocated.

[sal_vager]

Linux isn't a fix for crazy builds, no.

I keep all my craft to below 200 parts, when I test someone's megawhatsit I have to turn everything down and I still have a low frame rate, but still no obvious pausing.

This stutter issue can affect players with only one part.

[antilochus]

Hardinfo report on Hastebin

Settings.cfg

.nvidia-settings-rc

My refresh rate is stuck at 60hz thanks to the driver which may be why I don't see the pausing, and Xfce uses at most 9% of the system RAM, and nothing else I run while playing KSP is using much memory or constantly scanning files.

Have you noticed any benefits (fps-wise) to running at a lower resolution in windowed mode in Linux? Do you have any performance issues when running full-res and fullscreen?

[Phantasmogasm]

i found this thread after googling this same problem, after reading the thread for a bit i started watching GCM and saw some odd stuff when i turned up the precision on the graph drawing.

http://i.imgur.com/JQ7DORW

Its more easy to see on the higher precision graph:

http://i.imgur.com/TNTgwFu.jpg

but for my game it looks like the system is dumping a boatload of memory into mono, continuously, mono then bricks itself, dumps all the excess memory, and stutters, causing what looks like the entire application to hang, and then the whole process starts again.

whats even more confusing, is if this had anything to do with the memory required to do real time physics processing, why the hell would:

http://i.imgur.com/kAd52vb.png

it be happening at the start menu?

on start up my KSP uses 4.2GB of RAM, rising to about 5.5/6 during intensive gameplay, but the memory graph for KSP remains relatively constant and so does my fps, even during the stutters, which really points the finger at mono.

Though by no means am i an expert or claim to be.

I really hope a fix for this comes about soon, i love KSP and will continue to play the life out of it, but the stuttering is so off putting its almost headache inducing to watch.

Sys Specs:

FX-8320 3.4GHz

GTX-970 4GB (3.5/.5) VRAM

16GB 1600MHz Corsair XMS3

Ubuntu 15.04 x64 / KSP x64

Edited June 10, 2015 by Phantasmogasm

[katateochi]

So running on linux is a fix? What is your part count limit? Could you try a 600 part ship?

Linux is not a fix for me. I have the stutter noted here in both Linux (LMDE Mate) and Windows (Win7) and it occurs with 1 part ships just as often as large ones.

Linux doesn't help with the stuttering (or the part-count related performance) for me either

My system is a i7-2600k running Win 10 x64 with 8GB RAM and a Geforce 970 and a 2x200GB SSD system disk as RAID0 hosting KSP. It still stutters on my system

IF - and only if - the garbage collector is the reason for this stutter, then the question is: Which part of the program (or Unity) is constantly allocating and freeing memory while the rocket/plane is in midair? Stuttering even occurs when there is no staging. Just flying. So we have a static rocket/plane model which need no allocating/freeing. We have a more or less static landscape model which have some LOD logic but I doubt that this needs constant memory allocating.

The only thing left is... sound. Please listen. To have a constant engine sound KSP has to play the engine sound file repeatedly. But you won't hear any sound repitition from the engines without stutter. The stutter occurs at the end of the sound file.

I assume the program logic looks like: REPEAT -> Allocate memory for sound file -> Play sound file -> Free memory -> REPEAT-UNTIL Enging is off

Instead of: Allocate memory for sound file -> REPEAT -> Play sound file -> REPEAT-UNTIL Enging is off -> Free memory

Could this be?

Squad, do you read?

I don't think it's anything to do with engine sounds, I get the stutter with a craft that is just sitting on the runway before any of it's engines are even activated (it happens constantly regardless of what is going on. It happens in the editors too although when looking at the GCmonitor graphs the interval between calls to the CG is greater).

But I think you are right to question why it is that something is constantly allocating and freeing memory.

From what I've read, this stuttering issue IS a unity GC problem, BUT it can be mitigated with more optimal use of resources. One answer on stackoverflow talks about the need to load your resources at the start rather than on the fly and about creating a pool of objects and enabling and disabling them rather than destroying them. It suggests that you should avoid calls to Instantiate, Destroy and Resources.UnloadUnusedAssets while ingame (while the latter can optimize memory usage it could be costly if it's being called too often).

(I'm just summarizing here from what I've understood)

As I understand it the move to unity 5 won't just solve this because the GC is still pretty much the same (I may be wrong about this, I hope I am). If the GC isn't being significantly changed, then this problem won't just go away. SQUAD needs to go over how resources are loaded and unloaded and make improvement to work around this unity limitation.

Linux isn't a fix for crazy builds, no.

I keep all my craft to below 200 parts, when I test someone's megawhatsit I have to turn everything down and I still have a low frame rate, but still no obvious pausing.

This stutter issue can affect players with only one part.

Is more than 200 parts a crazy build!? In the past I used to regularly build craft with 400-500 parts and still have reasonable performance. 200 is such a low part-count in my view and it really prevents detailed designs. Sure you can get anywhere in a 100 part craft, but from a creative building point of view that's a real killer.

I've been experimenting with part-count performance and graphics settings and adjusting graphics settings doesn't make any difference on my FPS for craft above a certain part count. For example my 300 part station will run at 5-10FPS when on the very very lowest graphics settings or when I've turned everything right up to full.

My specs just for completeness sake:

CPU: Intel Core i7 @ 3.07GHz

RAM: 16GB DDR3 (corsair dominator)

MB: Asus Rampage III extreme

GPU: GTX 760

physX: GTX 460

Main HDD: 500GB Samsung SSD (I've also tried having KSP on a different disk to the OS, no difference)

OS: primarily Win7, also tried it in Ubuntu 14.04LTS

[sal_vager]

It's crazy for me, I couldn't think of a better word at the time, "Whackjobian build" maybe?

I just don't make big craft, so I tried to go overboard with fuel tanks but it turns out KSP doesn't like lots of round-8's, it doesn't even like loading

in the VAB.

Oh and guess what, on my max resolution of 1920x1080 and full screen I'm seeing stutter, it's really slight and I have to be looking for it, but it's there, I always played windowed because I am also on the forums and IRC at the same time.

katateochi, the stackoverflow stuff is interesting but I don't know if it'd help, as KSP does load everything at the start and just makes instances of each object when one is needed, it's only those instances that are destroyed, ie a reference to an object with its own variables for position, speed and stuff, the instance doesn't have its own texture or mesh, they aren't loaded from disk each time.

At least that is how Unity should be handling assets, but even the Unity manual page says excessive garbage collection can cause performance issues.

A Unity or Mono dev might be able to tell us more about what controls they have for CG, I don't know if it's fully automatic with no input from a developer or if there's any way to defer collection, also we don't know what efforts Squad have already tried, this issue was reported by HarvesteR years ago so it's fair to assume they have tried to deal with this.

I do know though that Unity uses a modified Mono, so even if they stick to the same version for U5, that doesn't mean it won't have also been modified to work better with the newer engine.