How do I debug crashes?

[dlmarti]

I am a long time KSP player, playing exclusively on Linux.

I have watched other players, on the forums, struggle with crashes.  I however have never experienced any, not in years of playing.  At least none do to the game.  I have had crashes because of mods.

I am struck with new territory;   While I love the new 1.1, I am experiencing a lot of crashes and I am unsure of how to debug/report them.

All of my crashes occur in the VAB, just leaving the VAB, or at first stage.  I have experience no other crashes.

The crashes either occur with a mem free error (5% of the time), or with no error at all (95% of the time).

Is there a mode that I could put the game into that would allow me to see the crash errors, so that I can report them?

[Snark]

Moving to Technical Support.

There's a log file that gives exactly what you want.  It's called output_log.txt, and is located in your KSP_data or KSP_x64_data folder, depending on which KSP flavor you're running.  This will include the crash details.

Aside from that, there's also another log file that is somewhat less useful (because it generally doesn't have the crash details), but might be helpful if there are any pre-crash symptoms that could give a clue to the problem.  This is the KSP.log file from the main KSP installation directory.  When you have an actual game-crash-to-desktop, it typically doesn't catch the actual crash, but it may have some interesting items leading up to the crash, so it may sometimes be useful.

(The mod thread for Kerbal Construction Time has a pretty good discussion of crash output and where to find it; skip down to the "Reporting bugs" section.)

[dlmarti]

There is no file called output_log.txt anywhere in my file tree.

I was already aware of the KSP.log, it does not show the errors.

[Padishar]

On linux the file is called player.log and is in an odd place. See the how to get support sticky thread in this subforum for details of where to find it.

[sal_vager]

On Linux it's the Player.log, which can be found in the /home/username/.config/unity3d/Squad/Kerbal Space Program/ folder.

It often has some extra information such as stack traces when KSP crashes, though these are not usually very useful as they show memory addresses, rather than useful error messages, like this.

Spoiler

Native stacktrace:

    /home/user/Games/Kerbal_Space_Program/ksp/KSP_Data/Mono/x86_64/libmono.so(+0x91772) [0x7f35981cf772]
    /home/user/Games/Kerbal_Space_Program/ksp/KSP_Data/Mono/x86_64/libmono.so(+0x348a5) [0x7f35981728a5]
    /lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0) [0x7f359b35f3d0]
    /home/user/Games/Kerbal_Space_Program/ksp/KSP.x86_64() [0xd2b0d8]
    /home/user/Games/Kerbal_Space_Program/ksp/KSP.x86_64() [0x84b2cf]
    /home/user/Games/Kerbal_Space_Program/ksp/KSP.x86_64() [0x84b069]
    /home/user/Games/Kerbal_Space_Program/ksp/KSP.x86_64() [0x84bb19]
    /home/user/Games/Kerbal_Space_Program/ksp/KSP.x86_64() [0x84bcc8]
    /home/user/Games/Kerbal_Space_Program/ksp/KSP.x86_64() [0x84bde5]
    /home/user/Games/Kerbal_Space_Program/ksp/KSP.x86_64() [0x84d983]
    /lib/x86_64-linux-gnu/libpthread.so.0(+0x76fa) [0x7f359b3556fa]
    /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f3599bdcb5d]

Debug info from gdb:

Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.
No threads.

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Most of the crashes in Linux, and I suspect on OSX and Windows, seem to be in the custom mono library included with all Unity games, and this is almost always shown as the problem file in the stack trace.

 

 

[Hodari]

Looking through my KSP.log file, I noticed these errors:

[LOG 16:15:10.101] Load(Model): Squad/Parts/Misc/PotatoRoid/PotatoRoid
[ERR 16:15:10.112] ConvexHullBuilder: convex hull has more than 255 polygons!

[ERR 16:15:10.114] Gu::ConvexMesh::loadConvexHull: convex hull init failed! Try to use the PxConvexFlag::eINFLATE_CONVEX flag. (see PxToolkit::createConvexMeshSafe)

[ERR 16:15:10.125] ConvexHullBuilder: convex hull has more than 255 polygons!

[ERR 16:15:10.126] Gu::ConvexMesh::loadConvexHull: convex hull init failed! Try to use the PxConvexFlag::eINFLATE_CONVEX flag. (see PxToolkit::createConvexMeshSafe)

---

[LOG 16:15:10.911] Load(Model): Squad/Parts/Utility/landingLegLT-5/model
[ERR 16:15:10.917] WheelCollider requires an attached Rigidbody to function.

[ERR 16:15:10.918] WheelCollider requires an attached Rigidbody to function.

and similar ones for all the other wheels and landing legs
---

output_log shows this several times:

The referenced script on this Behaviour is missing!
 
(Filename:  Line: 1649)

----------

Nearly all of my crashes have been while in the VAB, when I try to remove a part from my rocket.  For mods, I started off using just KER, earlier today I added RSS and SMURFF, then removed KER after noticing it apparently hasn't been updated for 1.1.2 yet.  Had the same issues in 1.1.0 and 1.1.1 as well though and with all of the above combinations of mods.  Not sure if any of that helps at all

[sal_vager]

None of those errors you show are a worry.

Quote

[ERR 16:15:10.112] ConvexHullBuilder: convex hull has more than 255 polygons!

This one occurs because the roid doesn't have its own collider, one is made from the roids mesh after loading, but because the roid has more than 255 polygons Unity complains.

Quote

[ERR 16:15:10.917] WheelCollider requires an attached Rigidbody to function.

Until you put a wheel on another part it doesn't have a rigidbody.

Quote

The referenced script on this Behaviour is missing!

Some scripts aren't initialized until you're in the flight scene, so this can be ignored.

 

Bugs were occurring with KER during pre-release, which vanished when KER was removed, I'd be wary of using KER too much.

But I've had crashes too without KER, sometimes in the VAB/SPH, it can be when placing parts or just idling, but most of my crashes are during scene change, with some in the flight scene.

And every time I get a stack trace.

[Hodari]

6 hours ago, sal_vager said:

None of those errors you show are a worry.

This one occurs because the roid doesn't have its own collider, one is made from the roids mesh after loading, but because the roid has more than 255 polygons Unity complains.

Until you put a wheel on another part it doesn't have a rigidbody.

Some scripts aren't initialized until you're in the flight scene, so this can be ignored.

 

Bugs were occurring with KER during pre-release, which vanished when KER was removed, I'd be wary of using KER too much.

But I've had crashes too without KER, sometimes in the VAB/SPH, it can be when placing parts or just idling, but most of my crashes are during scene change, with some in the flight scene.

And every time I get a stack trace.

Ok, thanks.  I'll try to look at those log files again the next time it crashes and see if I notice any other errors.  And guess I'll have to live without KER, at least until an updated version is released.  I'm sure you've heard it before, but it'd be really nice if you can at least get the delta-V/TWR info added into the stock game at some point   For now, are there any other compatible mods which provide this info?

[sal_vager]

2 hours ago, Hodari said:

 For now, are there any other compatible mods which provide this info?

MechJeb offers many readouts that can help, and should be working fine in 1.1.2

[dlmarti]

ERROR: ld.so: object '/home/dlmarti/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

Native stacktrace:

        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP_Data/Mono/x86/libmono.so(+0x8960f) [0xf368060f]
        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP_Data/Mono/x86/libmono.so(+0x21a20) [0xf3618a20]
        [0xf76ef410]
        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP.x86() [0x897faa0]
        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP.x86() [0x848a0c1]
        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP.x86() [0x8489e4c]
        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP.x86() [0x848a97b]
        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP.x86() [0x848ab28]
        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP.x86() [0x848ac84]
        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP.x86() [0x848c7c8]
        /lib/i386-linux-gnu/libpthread.so.0(+0x6f70) [0xf767af70]
        /lib/i386-linux-gnu/libc.so.6(clone+0x5e) [0xf7210bee]

Debug info from gdb:

ERROR: ld.so: object '/home/dlmarti/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.
No threads.

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

_float_vertex GL_ARB_imaging GL_ARB_indirect_parameters GL_ARB_instanced_arrays GL_ARB_internalformat_query GL_ARB_internalformat_query2 GL_NV_internalformat_sample_query GL_ARB_invalidate_subdata GL_ARB_map_buffer_alignment GL_ARB_map_buffer_range GL_ARB_multi_bind GL_ARB_multi_draw_indirect GL_ARB_multisample GL_ARB_multitexture GL_ARB_occlusion_query GL_ARB_occlusion_query2 GL_ARB_pipeline_statistics_query GL_ARB_pixel_buffer_object GL_ARB_point_parameters GL_ARB_point_sprite GL_ARB_program_interface_query GL_ARB_provoking_vertex GL_ARB_robust_buffer_access_behavior GL_ARB_robustness GL_ARB_sample_shading GL_ARB_sampler_objects GL_ARB_seamless_cube_map GL_ARB_seamless_cubemap_per_texture GL_ARB_separate_shader_objects GL_ARB_shader_atomic_counters GL_ARB_shader_bit_encoding GL_ARB_shader_draw_parameters GL_ARB_shader_group_vote GL_ARB_shader_image_load_store GL_ARB_shader_image_size GL_ARB_shader_objects GL_ARB_shader_precision GL_ARB_query_buffer_object GL_ARB_shader_storage_buffer_object GL_ARB_shader_subroutine GL_ARB_shader_texture_image_samples GL_ARB_shader_texture_lod GL_ARB_shading_language_100 GL_ARB_shading_language_420pack GL_ARB_shading_language_include GL_ARB_shading_language_packing GL_ARB_shadow GL_ARB_sparse_buffer GL_ARB_sparse_texture GL_ARB_stencil_texturing GL_ARB_sync GL_ARB_tessellation_shader GL_ARB_texture_barrier GL_ARB_texture_border_clamp GL_ARB_texture_buffer_object GL_ARB_texture_buffer_object_rgb32 GL_ARB_texture_buffer_range GL_ARB_texture_compression GL_ARB_texture_compression_bptc GL_ARB_texture_compression_rgtc GL_ARB_texture_cube_map GL_ARB_texture_cube_map_array GL_ARB_texture_env_add GL_ARB_texture_env_combine GL_ARB_texture_env_crossbar GL_ARB_texture_env_dot3 GL_ARB_texture_float GL_ARB_texture_gather GL_ARB_texture_mirror_clamp_to_edge GL_ARB_texture_mirrored_repeat GL_ARB_texture_multisample GL_ARB_texture_non_power_of_two GL_ARB_texture_query_levels GL_ARB_texture_query_lod GL_ARB_texture_rectangle GL_ARB_texture_rg GL_ARB_texture_rgb10_a2ui GL_ARB_texture_stencil8 GL_ARB_texture_storage GL_ARB_texture_storage_multisample GL_ARB_texture_swizzle GL_ARB_texture_view GL_ARB_timer_query GL_ARB_transform_feedback2 GL_ARB_transform_feedback3 GL_ARB_transform_feedback_instanced GL_ARB_transform_feedback_overflow_query GL_ARB_transpose_matrix GL_ARB_uniform_buffer_object GL_ARB_vertex_array_bgra GL_ARB_vertex_array_object GL_ARB_vertex_attrib_64bit GL_ARB_vertex_attrib_binding GL_ARB_vertex_buffer_object GL_ARB_vertex_program GL_ARB_vertex_shader GL_ARB_vertex_type_10f_11f_11f_rev GL_ARB_vertex_type_2_10_10_10_rev GL_ARB_viewport_array GL_ARB_window_pos GL_ATI_draw_buffers GL_ATI_texture_float GL_ATI_texture_mirror_once GL_S3_s3tc GL_EXT_texture_env_add GL_EXT_abgr GL_EXT_bgra GL_EXT_bindable_uniform GL_EXT_blend_color GL_EXT_blend_equation_separate GL_EXT_blend_func_separate GL_EXT_blend_minmax GL_EXT_blend_subtract GL_EXT_compiled_vertex_array GL_EXT_Cg_shader GL_EXT_depth_bounds_test GL_EXT_direct_state_access GL_EXT_draw_buffers2 GL_EXT_draw_instanced GL_EXT_draw_range_elements GL_EXT_fog_coord GL_EXT_framebuffer_blit GL_EXT_framebuffer_multisample GL_EXTX_framebuffer_mixed_formats GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_framebuffer_object GL_EXT_framebuffer_sRGB GL_EXT_geometry_shader4 GL_EXT_gpu_program_parameters GL_EXT_gpu_shader4 GL_EXT_multi_draw_arrays GL_EXT_packed_depth_stencil GL_EXT_packed_float GL_EXT_packed_pixels GL_EXT_pixel_buffer_object GL_EXT_point_parameters GL_EXT_polygon_offset_clamp GL_EXT_post_depth_coverage GL_EXT_provoking_vertex GL_EXT_raster_multisample GL_EXT_rescale_normal GL_EXT_secondary_color GL_EXT_separate_shader_objects GL_EXT_separate_specular_color GL_EXT_shader_image_load_formatted GL_EXT_shader_image_load_store GL_EXT_shader_integer_mix GL_EXT_shadow_funcs GL_EXT_sparse_texture2 GL_EXT_stencil_two_side GL_EXT_stencil_wrap GL_EXT_texture3D GL_EXT_texture_array GL_EXT_texture_buffer_object GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_latc GL_EXT_texture_compression_rgtc GL_EXT_texture_compression_s3tc GL_EXT_texture_cube_map GL_EXT_texture_edge_clamp GL_EXT_texture_env_combine GL_EXT_texture_env_dot3 GL_EXT_texture_filter_anisotropic GL_EXT_texture_filter_minmax GL_EXT_texture_integer GL_EXT_texture_lod GL_EXT_texture_lod_bias GL_EXT_texture_mirror_clamp GL_EXT_texture_object GL_EXT_texture_shared_exponent GL_EXT_texture_sRGB GL_EXT_texture_sRGB_decode GL_EXT_texture_storage GL_EXT_texture_swizzle GL_EXT_timer_query GL_EXT_transform_feedback2 GL_EXT_vertex_array GL_EXT_vertex_array_bgra GL_EXT_vertex_attrib_64bit GL_EXT_x11_sync_object GL_EXT_import_sync_object GL_IBM_rasterpos_clip GL_IBM_texture_mirrored_repeat GL_KHR_context_flush_control GL_KHR_debug GL_KHR_robust_buffer_access_behavior GL_KHR_robustness GL_KTX_buffer_region GL_NV_bindless_multi_draw_indirect GL_NV_bindless_multi_draw_indirect_count GL_NV_bindless_texture GL_NV_blend_equation_advanced GL_NV_blend_equation_advanced_coherent GL_NV_blend_square GL_NV_command_list GL_NV_compute_program5 GL_NV_conditional_render GL_NV_conservative_raster GL_NV_conservative_raster_dilate GL_NV_copy_depth_to_color GL_NV_copy_image GL_NV_depth_buffer_float GL_NV_depth_clamp GL_NV_draw_texture GL_NV_ES1_1_compatibility GL_NV_ES3_1_compatibility GL_NV_explicit_multisample GL_NV_fence GL_NV_fill_rectangle GL_NV_float_buffer GL_NV_fog_distance GL_NV_fragment_coverage_to_color GL_NV_fragment_program GL_NV_fragment_program_option GL_NV_fragment_program2 GL_NV_fragment_shader_interlock GL_NV_framebuffer_mixed_samples GL_NV_framebuffer_multisample_coverage GL_NV_geometry_shader4 GL_NV_geometry_shader_passthrough GL_NV_gpu_program4 GL_NV_gpu_program4_1 GL_NV_gpu_program5 GL_NV_gpu_program5_mem_extended GL_NV_gpu_program_fp64 GL_NV_gpu_shader5 GL_NV_half_float GL_NV_light_max_exponent GL_NV_multisample_coverage GL_NV_multisample_filter_hint GL_NV_occlusion_query GL_NV_packed_depth_stencil GL_NV_parameter_buffer_object GL_NV_parameter_buffer_object2 GL_NV_path_rendering GL_NV_path_rendering_shared_edge GL_NV_pixel_data_range GL_NV_point_sprite GL_NV_primitive_restart GL_NV_register_combiners GL_NV_register_combiners2 GL_NV_sample_locations GL_NV_sample_mask_override_coverage GL_NV_shader_atomic_counters GL_NV_shader_atomic_float GL_NV_shader_atomic_fp16_vector GL_NV_shader_atomic_int64 GL_NV_shader_buffer_load GL_NV_shader_storage_buffer_object GL_NV_texgen_reflection GL_NV_texture_barrier GL_NV_texture_compression_vtc GL_NV_texture_env_combine4 GL_NV_texture_multisample GL_NV_texture_rectangle GL_NV_texture_shader GL_NV_texture_shader2 GL_NV_texture_shader3 GL_NV_transform_feedback GL_NV_transform_feedback2 GL_NV_uniform_buffer_unified_memory GL_NV_vdpau_interop GL_NV_vertex_array_range GL_NV_vertex_array_range2 GL_NV_vertex_attrib_integer_64bit GL_NV_vertex_buffer_unified_memory GL_NV_vertex_program GL_NV_vertex_program1_1 GL_NV_vertex_program2 GL_NV_vertex_program2_option GL_NV_vertex_program3 GL_NV_viewport_array2 GL_NVX_conditional_render GL_NVX_gpu_memory_info GL_NVX_nvenc_interop GL_NV_shader_thread_group GL_NV_shader_thread_shuffle GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent GL_SGIS_generate_mipmap GL_SGIS_texture_lod GL_SGIX_depth_texture GL_SGIX_shadow GL_SUN_slice_accum 
GLX Extensions: GLX_EXT_visual_info GLX_EXT_visual_rating GLX_SGIX_fbconfig GLX_SGIX_pbuffer GLX_SGI_video_sync GLX_SGI_swap_control GLX_EXT_swap_control GLX_EXT_swap_control_tear GLX_EXT_texture_from_pixmap GLX_EXT_buffer_age GLX_ARB_create_context GLX_ARB_create_context_profile GLX_EXT_create_context_es_profile GLX_EXT_create_context_es2_profile GLX_ARB_create_context_robustness GLX_NV_delay_before_swap GLX_EXT_stereo_tree GLX_ARB_context_flush_control GLX_ARB_multisamp:

 

That was my most recent crash, after about 90 seconds of playing.

excrements, I used to enjoy KSP, but these crashes are trying to make that not a thing.

[sal_vager]

Well this part is easy.

ERROR: ld.so: object '/home/dlmarti/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

There are two Steam gameoverlayrenderer.so files, one for 32bit and one for 64bit, while one will load the other will fail, because it is the wrong executable linkable format.

The next part tells us which file generated the error.

Native stacktrace:

        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP_Data/Mono/x86/libmono.so(+0x8960f) [0xf368060f]
        /home/dlmarti/.local/share/Steam/steamapps/common/Kerbal Space Program/KSP_Data/Mono/x86/libmono.so(+0x21a20) [0xf3618a20]

In this case it was the libmono.so in Kerbal Space Program/KSP_Data/Mono/x86/

After that there's memory addresses, then this again.

ERROR: ld.so: object '/home/dlmarti/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

 

Again, it's the wrong version for your architecture, from this I can deduce that you are using a 64bit OS if you have not already told me.

[steve_v]

6 hours ago, sal_vager said:

Native stacktrace:

I have been getting these too. If I go to the effort of providing a bunch of stack traces from a mono built with debugging symbols and complete gdb output, would these be any help?

Is anyone doing any real investigation into this issue? The hotfixes have changed nothing regarding this crash.

 

[sal_vager]

7 hours ago, steve_v said:

I have been getting these too. If I go to the effort of providing a bunch of stack traces from a mono built with debugging symbols and complete gdb output, would these be any help?

Is anyone doing any real investigation into this issue? The hotfixes have changed nothing regarding this crash.

 

I have over 100 so far, I'd have more but I didn't keep them all. These addresses don't mean anything to me, and Unity lacks tools to work with memory directly so there seems to be no way to find what's going on at this level, unless you're a Unity developer I suppose.

I've been trying to investigate this, my knowledge of programming is limited so if the below is wrong let me know.

I think these are linked to the double calls to free(), in c++ the free() method goes to the area of memory pointed to and deallocates it, but it doesn't null the pointer, so in c++ (and c#) you're able to call free() on the same area twice, and of course if it has been reused you've just wrecked whatever was there on the second call, creating lots of "fun".

Similarly if the pointer was incorrect you've just mangled some perfectly good data, this might be what's causing the segfaults that don't show a double free().

In Unity you can't call free() directly nor do you have access to the pointers, it's all done via the Destroy() method which is a black box, but if you call Destroy() on an object that doesn't exist it should return a null reference exception, just like any other Unity method when used on an object that isn't there.

Anyone involved with the pre-release or encountering a bug in the release will already have seen the nullrefs that are generated when an object is missing, and these can be dealt with by adding an if() statement to check if the object exists.

This is all fine until a valid object has a bad pointer or that pointer points to literally garbage values.

How does Unity know the object exists?  I don't know, it's another black box and it may be using an internal list of object names that reference pointers to memory, there's probably fancier ways I'm unaware of.

But pointers don't know what should be in the area they point to, corrupted memory looks just as valid as any other, it's only when a method is invoked that tries to use that memory and finds garbage values that things break down.

So I think (I can't prove) there's something wrong with how Unity is tracking the pointers, maybe it's not removing them immediately or the order is getting messed up, either way it's intermittent as players can play for days with no trouble then bam!, three memory related crashes in a row.

So mono is working as expected with the calls to free(), call it on the same pointer and you have fun times, it's left to the programmer to remove the pointer.

If you're interested you can read the version of mono that Unity uses here.

Squad can (and has) added checks to see if objects exist before acting on them, but even where this hasn't been done the object should nullref, nullrefs will often show where the issue is.

But that's not happening, the objects seem to be reporting to Unity that they are present, so if() checks return as true and no nullrefs are generated.

You end up trying to process values that are half of one thing and half something else, or the area of memory simply isn't in use at all at that moment in time.

So Unity just dies, says "the memory here is bad", and you're left with the unanswered question "Why?".

[steve_v]

1 hour ago, sal_vager said:

I've been trying to investigate this, my knowledge of programming is limited so if the below is wrong let me know.

Sounds about right to me, to the best of my knowledge. (C = some, mono = sweet FA, Unity = Nil)

I know what the error means in the context of native C/C++ (pretty much exactly what it says, glibc thinks somebody has freed the same block of memory twice, potentially corrupting the heap) If this occurs, by default glibc will abort and output a stack trace on STDERR. This behaviour can be turned off, but it's not a particularly good idea, for fairly obvious reasons.

2 hours ago, sal_vager said:

I think (I can't prove) there's something wrong with how Unity is tracking the pointers, maybe it's not removing them immediately or the order is getting messed up, either way it's intermittent as players can play for days with no trouble then bam!, three memory related crashes in a row.

If your theory is correct, and the breakdown is in one of Unitys own mystery-meat methods, this may well be impossible to pin down without the Unity sources. The intermittent bit I would expect for this kind of error, as when a double free occurs, the behaviour is undefined. If that pointer is NULL at the time, we're all good. If it's not, chaos ensues.
This could also be a thread-safety problem *shudder* which would be even more fun to find. Either way, I'm guessing (I don't read Windows stack traces ) it's the root cause of crashes on all platforms.

Use of Electric Fence or Valgrind might be enlightening, but that will require a debug build (of the Unity player too, not just mono), and to be realistic, a copy of the source code to make sense of it. I have neither, hence asking if someone with better skills and tools is looking into this.

45 minutes ago, sal_vager said:

If you're interested you can read the version of mono that Unity uses here.

Indeed, I already have (several copies of) that repo. I have been mucking about with it, but my foo is limited as is my time, so finding this (if it's even in mono), without source or a debug build of the rest of Unity is not really going to be a thing for me. I'll probably waste time on it anyway though.

[dlmarti]

I started a new issue after capturing the crash on a brand new PC with a clean install.

http://bugs.kerbalspaceprogram.com/issues/9824