Memory leak?

[xerogravity]

I've just started playing KPS on a new (to me) iMac

The iMac is running 10.9.5 (Mavericks) and has 16Gb RAM, 2.9Ghz i5.

I'm running the latest version of KSP

As my KSP session progresses, if I spend too long on Kerbin (airplane stuff rather than orbital), the ground textures begin to go. To try and explain, there appears to be a corrupted texture leaking over the top of the correct texture, like a grey mash-up grid.

Soon after this any change in area status; moving between SPH and runway, VAB and pad, causes the game to go black screen and hang.

If I go orbital, there isn't quite the same problem but going back to the Space Centre or sometimes even switching ships causes black screen and a hang.

It has all the hallmarks of a memory leak. I tend to just use KSP in short bursts now or quicksave a lot. I can cope with this, irritating as it is, but as we lurch towards a 1.0 release I think t needs to be sorted. First, I need to know if any other Mac users or having any similar problems then I can report this as a bug.

Thanks for your time.

[undercoveryankee]

I've just started playing KPS on a new (to me) iMac

The iMac is running 10.9.5 (Mavericks) and has 16Gb RAM, 2.9Ghz i5.

I'm running the latest version of KSP

As my KSP session progresses, if I spend too long on Kerbin (airplane stuff rather than orbital), the ground textures begin to go. To try and explain, there appears to be a corrupted texture leaking over the top of the correct texture, like a grey mash-up grid.

Soon after this any change in area status; moving between SPH and runway, VAB and pad, causes the game to go black screen and hang.

If I go orbital, there isn't quite the same problem but going back to the Space Centre or sometimes even switching ships causes black screen and a hang.

It has all the hallmarks of a memory leak. I tend to just use KSP in short bursts now or quicksave a lot. I can cope with this, irritating as it is, but as we lurch towards a 1.0 release I think t needs to be sorted. First, I need to know if any other Mac users or having any similar problems then I can report this as a bug.

Thanks for your time.

KSP is a 32-bit process on OS X, so you'll start to see glitches as you approach 4 GB of RAM used. I've seen address-space pressure cause basically the same failure modes you're describing across a couple of different versions of KSP and OS X. There are enough bugs open on the tracker for apparent leaks that opening another one probably won't add any useful information.

The kinds of memory leaks that affect .NET applications like Unity are hard to debug. The system garbage collector doesn't allow unreachable memory to remain allocated (usually; there are a few lower-level bugs where garbage collecting a managed object fails to deallocate an unmanaged allocation that it uses). The more common type of leak is one where an object remains reachable somewhere even after the programmer is no longer interested in it. Automated analysis tools can't tell whether you've leaked that memory or just haven't run the code you were saving it for. And when the crash happens, the stack trace tells you only what was trying to allocate memory when you ran out. You're on your own to figure out what code should have disposed of something to prevent the crash.

1.0 probably won't be the end of memory leaks, but I do hope to see some incremental improvements.

[xerogravity]

I used to run the Boulder Co clouds etc but am now down to just NavyFish Docking Alignment as my only mod. It still bugs out unmodded so I've tried various configs and most mods make it happen sooner. Bit disappointing really.

Prior to v0.24 I ran KSP on an older Mac with no real problems but it started spluttering around 0.25 so moved to the new iMac for 0.9. Even stock won't run cleanly now. Doesn't look good for 1.0 at this rate.

[sal_vager]

Hi xerogravity

It has been noticed that Macs use a lot of RAM even when idle, so there is not much RAM for KSP, this causes KSP assets to be cached and they may not be retrieved properly.

Anything you can do to reduce your RAM use may help, such as shutting down unused applications, also please see the stickies and provide your logs.

[Faster]

It has been noticed that Macs use a lot of RAM even when idle, so there is not much RAM for KSP, this causes KSP assets to be cached and they may not be retrieved properly.[...]

Likely you typed that in a hurry and what was thought and typed turned out different, so no worries as the intent was obviously to be helpful; unfortunately what came across is neither factual nor logical. The closest I could some would be the (made up) "It has been noticed that Mac users usually buy Macs with small amounts of RAM (less than 4GB),and then there's not enough for KSP, causing some of KSP's memory to be swapped to disk, i.e., virtual memory is used, leading to awful performance, but no visual glitches or crashes or bugs like you just described." That's also not true (particularly in this case where 16GB was reported), nor related to symptoms in rendering, but it would make sense if it were.

Anyway, one way for xerogravity to check if s/he is on to something is to observe memory consumption over time especially when returning to known states, e.g., back to the Space Center with the same number of craft/parts in flight or none. If memory is indeed leaking, then consumption will trend up over time even in these steady-state situations. But, I don't think it would be worth the headache: what would you do with that information, tell Squad the OS X build is broken?

Memory leaks don't directly cause problems other than out-of-memory crashes (they can indirectly cause all sorts of other problems, e.g., poor performance with certain GC algorithms that tend to thrash when memory pressure is high.) The symptoms xerogravity saw presenting are indicative of programming errors, bugs in texture handling most likely, not poor reference management. Though, in theory they could be bugs whose symptoms are triggered by other bugs that are technically memory leaks.

xerogravity: the forums seem to say 0.90 on OS X is a very mixed bag and far less stable for many of the folks reporting than previous versions, e.g., the thread I started related to this: http://forum.kerbalspaceprogram.com/threads/113514-Which-Operating-System-for-0-90-on-Apple-Hardware -- though I didn't call out all the symptoms you saw, I have seen them all and much more on OS X with 0.90; the last 0.24.x builds were the Rock of Gibraltar by comparison.

[sal_vager]

Heh, Macs using a lot of Ram is common, apparently this is normal and "unused ram is wasted"

Several Mac users I know are finding 4 to 6 gigs of their Ram are in use even immediately after boot and with no programs open, and on an 8gig system that isn't leaving KSP with as much, especially if you have Safari or other programs running.

Yes KSP and the Unity3D game engine will load assets and keep them in memory, these are freed on scene change but the PQS terrain system is known to allocate a larger cache than there is memory available, but rather than being a "memory leak" this is a bounds checking issue.

There's no fix for this at this time, the only workarounds are to reduce memory usage overall with texture management addons, the 1.0 release should improve the situation though if Squad take note of community feedback.

Also, KSP on OSX has some issues with graphics which may explain the grey grid xerogravity reports, these are usually caused by anti-aliasing and the PPFX edge detection graphics feature, neither of which are working properly in Unity on OSX and are disabled by default on OSX for this reason.

[Jouni]

First, "free memory" is an ambiguous term, especially when we're comparing different operating systems. In many systems, the amount of memory a process uses is not even a well-defined quantity, because memory regions may be shared between processes.

Here is an overview of the memory usage of my two Macs (8 GB MacBook Pro and 16 GB iMac) under a fairly typical workload in OS X 10.10:

[table]

[tr]

[td][/td]

[td]MacBook Pro[/td]

[td]iMac[/td]

[td]iMac with KSP[/td]

[/tr]

[tr]

[td]App Memory[/td]

[td]3.81 GB[/td]

[td]4.17 GB[/td]

[td]6.70 GB[/td]

[/tr]

[tr]

[td]File Cache[/td]

[td]990 MB[/td]

[td]3.24 GB[/td]

[td]5.17 GB[/td]

[/tr]

[tr]

[td]Wired Memory[/td]

[td]1.22 GB[/td]

[td]1.28 GB[/td]

[td]1.29 GB[/td]

[/tr]

[/table]

The workload is the same for both computers. App Memory means memory used by user-level processes, while Wired Memory is used by kernel processes. File Cache is essentially free memory: recently opened files that are kept in memory, because there is no better use for the memory at the moment. These figures are based on the figures reported by Activity Monitor; top reports about 1 GB less. I'm not even sure what causes the difference.

There is definitely something strange going on with the memory usage of KSP in OS X 10.10. The last column is the iMac with a newly started unmodded KSP sandbox in empty VAB. KSP uses 2.45 GB of memory, which is a lot more than it used to be. If I put a Mk1 command pod on the launchpad and revert to VAB, memory usage increases to 2.50 GB. I repeat the same a few times with a few different command pods, and the memory usage increases to 2.57 GB. After doing the same with Kerbal X and Learstar A1, memory usage in empty VAB increases to 2.71 GB. After a few more stock crafts, the memory usage is at 2.76 GB. After doing the same with a newly-built 41-part rocket using parts not in the stock crafts, memory usage is at 2.90 GB. Then I fly a mission with the rocket, recover the pod, and return to VAB, and the memory usage has increased to 2.95 GB. I clear the VAB, and the infamous disco effects start.

The same also occurs with anti-aliasing off (it has been at the default 2x setting since I originally installed KSP 0.22 on this Mac). At the end of a similar sequence, the memory usage in empty VAB is 2.77 GB. After deliberately using as many different parts as possible and reverting to VAB, I finally got the memory usage over 2.95 GB, starting the disco effects.

It doesn't look like a memory leak to me. More likely, the game caches the assets that have been used, in addition to loading them at startup. The more parts there are, the higher the initial memory usage becomes. The more parts you use in a single session, the higher the memory usage grows.

Edited March 22, 2015 by Jouni

[Jouni]

Random observation: 32-bit processes can't use the full 4 GB in Yosemite. I tried allocating memory in 1 MB blocks, and the allocations started to fail soon after 3500 MB (around 3.42 GB). Apparently the rest of address space is reserved for something else. Kernel-related stuff and GPU memory are my best guesses. Incidentally, the GPU of my iMac has 1 GB of memory, and KSP starts behaving badly when memory usage approaches 3 GB. A similar limit may have existed before, or it may have been introduced in Yosemite, or even in some GPU drivers shipped with Yosemite.

It seems that there are a number of reasons why KSP 0.90 and OS X 10.10 are such a bad combination:

1. KSP both preloads all assets on startup, and caches a second copy of the assets that have actually been used. This may either be intentional or a bug, and the bug can be general or Mac-specific.
   
2. A 32-bit program can't use the full 4 GB addess space in Yosemite. The precise limit may depend on the GPU drivers used.
   
3. The number of stock parts increased significantly in 0.25 and 0.90. As a result, the memory limit is now easier to reach than before.
   

[Master Tao]

Ay, memory management on Mac OS X is... complex. I've been trying to track down reproduction steps for the odd memory-related crash off and on since last July.

The best I can say is this: with stock KSP, I've never had a memory crash as long as I free up at least 3.5-4 GB of physical RAM before starting KSP.

This type of crash does happen much more frequently if there's little to no physical RAM available at launch, but I can't get them to happen consistently. Any ideas?

[Jouni]

I investigated some of the more obscure sides of OS X memory management. Apparently OS X always reserves around 0.5 GB of address space for shared libraries, leaving only 3.5 GB writable virtual memory for a 32-bit process. My test program was able to use 3.4 GB of that, so it worked as it should. KSP, on the other hand, allocated only 2.8-2.9 GB of virtual memory, leaving a large unmapped region at the end of the address space. As it uses around 200 MB of shared libraries, KSP ran out of memory when its memory usage approached 3 GB.

This has probably something to do with the fact that KSP runs in a virtual machine. Perhaps Mono sets the heap limit lower in OS X 10.10 than in earlier OS X versions. If someone using an earlier version of OS X could run "vmmap -resident KSP" when KSP is running, and post the end of the output similar to the following, it might help understanding the situation.

==== Summary for process 1586
ReadOnly portion of Libraries: Total=205.3M resident=118.8M(58%) swapped_out_or_unallocated=86.5M(42%)
Writable regions: Total=2.9G written=2.2G(73%) resident=2.3G(80%) swapped_out=261.5M(9%) unallocated=616.6M(20%)

REGION TYPE                      VIRTUAL RESIDENT
===========                      ======= ========
CG backing stores                  7188K    7188K
CG image                             40K      40K
CG raster data                     14.8M    14.8M
CG shared images                    336K      96K
CoreAnimation                        16K      16K
CoreUI image data                    84K      84K
Foundation                            4K       4K
IOKit                             510.3M   358.0M
Kernel Alloc Once                     4K       4K
MALLOC                              1.9G     1.5G        see MALLOC ZONE table below
MALLOC (admin)                       64K      16K
MALLOC freed, no zone              2768K    2736K
MALLOC_LARGE                       3720K    3680K
MALLOC_LARGE (reserved)             912K       0K        reserved VM address space (unallocated)
Memory Tag 242                       12K      12K
OpenCL                               16K      16K
OpenGL GLSL                         128K      76K
STACK GUARD                        56.1M       0K
Stack                              23.8M     560K
VM_ALLOCATE                       549.1M   477.1M
VM_ALLOCATE (freed)                  12K       8K
VM_ALLOCATE (reserved)               40K       0K        reserved VM address space (unallocated)
__DATA                             9204K    5512K
__GLSLBUILTINS                     2588K     412K
__IMAGE                             528K       0K
__LINKEDIT                         49.4M    11.9M
__OBJC                             2488K    2144K
__TEXT                            155.9M   106.9M
__UNICODE                           544K     256K
mapped file                        68.2M    17.9M
shared memory                        68K       8K
===========                      ======= ========
TOTAL                               3.3G     2.5G
TOTAL, minus reserved VM space      3.3G     2.5G


                                   VIRTUAL   RESIDENT ALLOCATION      BYTES
MALLOC ZONE                           SIZE       SIZE      COUNT  ALLOCATED  % FULL
===========                        =======  =========  =========  =========  ======
GFXMallocZone_0x15e9000               1.1G     893.6M      38306       1.1G     94%
DefaultMallocZone_0x15cb000         724.6M     641.3M     427680     668.1M     92%
DispatchContinuations_0x15f8000      8192K       116K        107         7K      0%
CoreGraphics_0x3800c00                244K       232K        665        15K      6%
QuartzCore_0x182dc00                   76K        76K        508        16K     20%
===========                        =======  =========  =========  =========  ======
TOTAL                                 1.9G       1.5G     467266       1.7G     93%

[Jouni]

I continued abusing KSP until it crashed. The game managed to allocate 3.2 GB of writable virtual memory, but over 240 MB of that was still free, when it ran out of memory. This looks like a fragmentation issue. Player.log shows a large number of error messages like the following:

KSP(13161,0xb06a9000) malloc: *** mach_vm_map(size=8294400) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

My interpretation is that the game tries to allocate between 4 MB and 8 MB using malloc(), but the malloc zone (probably DefaultMallocZone) does not have a contiguous region large enough to satisfy the demand. malloc() then tries to allocate 8 MB more virtual memory using mach_vm_map(), but that request also fails. Either none of the remaining regions of contiguous address space is large enough, or OS X just refuses to allocate address space from the remaining regions.

==== Summary for process 13161
ReadOnly portion of Libraries: Total=200.8M resident=163.5M(81%) swapped_out_or_unallocated=37.3M(19%)
Writable regions: Total=3.2G written=56.8M(2%) resident=3.0G(93%) swapped_out=0K(0%) unallocated=242.7M(7%)

REGION TYPE VIRTUAL RESIDENT
=========== ======= ========
CG backing stores 9044K 9044K
CG image 12K 12K
CG shared images 336K 100K
CoreServices 468K 468K
CoreUI image data 64K 64K
IOKit 637.8M 582.4M
Kernel Alloc Once 4K 4K
MALLOC 1.9G 1.8G see MALLOC ZONE table below
MALLOC (admin) 64K 16K
MALLOC freed, no zone 7132K 7060K
MALLOC_LARGE (reserved) 912K 0K reserved VM address space (unallocated)
Memory Tag 242 12K 12K
OpenCL 16K 16K
OpenGL GLSL 128K 76K
STACK GUARD 56.1M 0K
Stack 23.3M 576K
VM_ALLOCATE 763.4M 649.4M
VM_ALLOCATE (freed) 16K 16K
VM_ALLOCATE (reserved) 40K 0K reserved VM address space (unallocated)
__DATA 9120K 6436K
__GLSLBUILTINS 2588K 708K
__IMAGE 528K 124K
__LINKEDIT 48.4M 15.3M
__OBJC 2488K 2452K
__TEXT 152.3M 148.2M
__UNICODE 544K 484K
mapped file 69.1M 27.0M
shared memory 132K 112K
=========== ======= ========
TOTAL 3.7G 3.2G
TOTAL, minus reserved VM space 3.7G 3.2G


VIRTUAL RESIDENT ALLOCATION BYTES
MALLOC ZONE SIZE SIZE COUNT ALLOCATED % FULL
=========== ======= ========= ========= ========= ======
GFXMallocZone_0x15e9000 1.1G 1.1G 56719 1.1G 95%
DefaultMallocZone_0x15cb000 804.3M 724.2M 984987 741.3M 92%
DispatchContinuations_0x15f8000 8192K 108K 112 7K 0%
CoreGraphics_0x400a000 152K 144K 660 15K 9%
QuartzCore_0x182ce00 16K 16K 2 112 0%
=========== ======= ========= ========= ========= ======
TOTAL 1.9G 1.8G 1042480 1.8G 93%

There are three kinds of memory regions that kept growing, as I used different parts and moved between VAB and the launchpad:

1. IOKit, which means stuff related to device drivers. This suggests that there could be a problem in some GPU drivers shipped with Yosemite.
   
2. MALLOC, and specifically DefaultMallocZone. The region does not really grow, but it keeps fragmenting due to a large number of small memory allocations.
   
3. VM_ALLOCATE, which I guess is the memory managed directly by Mono without using malloc(). This could be a result of the game caching the assets that are currently in use, but not releasing them when they're no longer needed.
   

Edit: There is also "GL: Detected 0 MB VRAM" near the beginning of Player.log, which suggests a Unity/GPU driver issue.

Edited March 23, 2015 by Jouni

[Yakky]

Hi xerogravity

It has been noticed that Macs use a lot of RAM even when idle, so there is not much RAM for KSP, this causes KSP assets to be cached and they may not be retrieved properly.

Anything you can do to reduce your RAM use may help, such as shutting down unused applications, also please see the stickies and provide your logs.

FWIW I run KSP on a Mac with 32 gigs of physical RAM and still have these problems, so I don't think the main culprit is a memory paging or swapping issue, although it's possible that it could compound other problems.

[Master Tao]

That's a good start.

What ways did you abuse KSP? It would really help to have a reasonably reliable method of intentionally causing that crash, not just waiting until it does.

[Jouni]

What ways did you abuse KSP? It would really help to have a reasonably reliable method of intentionally causing that crash, not just waiting until it does.

The easiest way is to build a craft with as many different parts as possible, put it on the launchpad, and revert to VAB. After reverting, the game uses more memory in VAB than before. If that doesn't raise memory usage high enough to cause disco lights or other graphics glitches, repeat the same with parts you haven't used already.

Reusing the parts you have already used doesn't seem to affect memory usage that much. I'd guess that either the game caches a second copy of the assets that have been used, or there is a weirder than usual memory management issue. If you're using stock parts with low graphics settings, you may run out of parts before memory usage grows high enough. In that case, you may try observing different planets at various distances to increase memory usage. It's helpful to return to VAB afterwards, as the game seems to use more memory in VAB than during flight.

[Master Tao]

Thanks!

I'll give those approaches a deeper look Soon^TM. Any chance you have a craft file that will cause the crash with load-launch-revert?