Jump to content

[1.2.x/1.3] MemGraph 1.1.0.3 - with Stutter Reduction

Padishar

By Padishar
in KSP1 Mod Releases

 Share

Recommended Posts

Oshika

Oshika

Posted
Posted
18 minutes ago, Padishar said:

Yes, you understand.  The devs do know about the issue, they have been making significant improvements in the amount of garbage the stock game generates in both the 1.1.x and 1.2.x versions.  Mod developers are also, generally, aware of the problem and most have been trying to make similar improvements.

Yes, you can simply adjust the total line to make the stutters further apart.  On a 16 GB machine with a lot of mods I would recommend not going much above 4096 or so, as many mods also use up memory for textures and models which are kept outside the Mono heap.  4 GB of padding should put the stutters pretty far apart unless your game is generating a really huge amount of garbage.  I would actually recommend starting with 2048 and see what effect it has, then increase it if necessary...

Thank you i will try that and keep you informed on the situation. 

Link to comment
Share on other sites
Jebs_SY

Jebs_SY

Posted
Posted (edited)

@Padishar

I just debug my 100-mod 1.2.2 setup, cause allocation rates around 30-40MB/s are not nice... and I wonder, would it be possible/useful for memgraph to show/calculate the average allocation rate between the 2nd last and last garbage collection? Maybe only when the memgraph-GUI is shown on screen. Do you think this is useful?

EDIT: Another interesting thing to compare different setups would be the interval. However, interval alone doesn't help, cause the heap can be different big. So maybe Heap-Delta (Max-Min) would be useful, too. Or maybe even Interval / Heap-Delta. This should result in a comparable performance number, or?

Edited by Jebs_SY
Link to comment
Share on other sites
Padishar

Padishar

Posted
  • Author
Posted
5 hours ago, Jebs_SY said:

I just debug my 100-mod 1.2.2 setup, cause allocation rates around 30-40MB/s are not nice... and I wonder, would it be possible/useful for memgraph to show/calculate the average allocation rate between the 2nd last and last garbage collection? Maybe only when the memgraph-GUI is shown on screen. Do you think this is useful?

Well, it wouldn't be difficult to total up the memory allocated as it goes along so it needn't be slow.  However, because of the inaccuracy inherent in the readings that MemGraph takes due to how the Mono heap and the padding mechanism works, I'm not sure it would be useful enough to justify the effort.  The real allocation rate is actually what the graph displays in the time immediately before a collection, the average for the whole interval would often come out a lot lower than the real rate.

5 hours ago, Jebs_SY said:

EDIT: Another interesting thing to compare different setups would be the interval. However, interval alone doesn't help, cause the heap can be different big. So maybe Heap-Delta (Max-Min) would be useful, too. Or maybe even Interval / Heap-Delta. This should result in a comparable performance number, or?

Again, all the values can be rather inaccurate and combining two such values will make the result even worse, so I can't see it being much more useful than just making a judgement from the graph and values that are currently displayed.

Link to comment
Share on other sites
NeuroticGamer

NeuroticGamer

Posted
Posted

@Padishar

Thanks for making this mod.  It's helped tremendously with stuttering.  Of course I'm fighting the behemoth of 140 mods in 1.2.1 seemingly generating garbage at warp speed :).

Is there any value in me installing (I have a legit license via my employer but never had reason to use it) JetBrains' dotMemory profiler?  https://www.jetbrains.com/dotmemory/features/

Would it help in identifying the heavy hitters in terms of garbage generation?

I've dabbled in .NET but my day job is mostly MATLAB with a smattering of C, C++, and Python.

 

Link to comment
Share on other sites
Padishar

Padishar

Posted
  • Author
Posted
9 minutes ago, NeuroticGamer said:

Is there any value in me installing (I have a legit license via my employer but never had reason to use it) JetBrains' dotMemory profiler?  https://www.jetbrains.com/dotmemory/features/

Would it help in identifying the heavy hitters in terms of garbage generation?

No, KSP doesn't use .NET, it uses an old version of Mono built in to the KSP executable.  However, the Unity profiler can be used to do the job instead, see this thread for details on how to set it up...

 

Link to comment
Share on other sites
Padishar

Padishar

Posted
  • Author
Posted
29 minutes ago, kball said:

Would love to be able to try this out on my mac laptop. Has anyone figured out a configuration to enable the hotkeys?

What is the problem?  You can change the key (used with the standard KSP "modifier" key, which I believe is LeftCommand on a Mac) for each of the functions in the GameData/MemGraph/PluginData/MemGraph/settings.cfg file.  Just use whatever values from this page that you want (if you set something, such as the keyRunTests one which is only really useful for developers, to None then it will be disabled):

https://docs.unity3d.com/ScriptReference/KeyCode.html

Link to comment
Share on other sites
kball

kball

Posted
Posted

I finally got it to work, but I only had success with using standard letter keys like 'P' and 'O'. Punctuation didn't work, though I was testing keys that required Shift. So the full key combination was:

opt-shift-1 for '!'

but 'Exclaim' didn't work for that. Maybe because option-shift-1 maps to another unicode character? Not sure how low level Unity's keyboard input is. Also tried Quote, Hash, At.

Anyway, just a heads up. Thanks for this. I've had some luck getting MemGraph to improve stutter. Doesn't seem to always work, but when it does, it makes a world of difference.

Link to comment
Share on other sites
Moofrog

Moofrog

Posted
Posted

Thank you very much for this mod.

Even in Stock KSP or only few amount of small mods I get stutter lag spikes every couple second for a duration of a second. The Game literally freezes.;.;:mad: Somehow turning off the GUI by pressing F2 reduces the stutter to about once a couple of minutes. I runy 8GB of RAM at the Moment but soon it will be 16GB Ram. So first 2048 Padding / Heap Size. Later 4096. Got it. :)

Hopefully one day soon they manage to make the transition away from Mono and with no garbage production at all. Especially the GUI should not generate so much Garbage.

 

Anways keep up the good work! :)

Link to comment
Share on other sites
NeuroticGamer

NeuroticGamer

Posted
Posted
On 12/13/2016 at 6:51 PM, Padishar said:

No, KSP doesn't use .NET, it uses an old version of Mono built in to the KSP executable.  However, the Unity profiler can be used to do the job instead, see this thread for details on how to set it up...

 

I'll have to play with this over the holidays.  I started the install process and it ran for 5 hours on a 6th gen i7, with the current fastest M.2 SSD, and a 125 Mbps internet connection.  Looks like it was mostly the Microsoft tools taking FOREVER :0.0:

I still have to resolve the variation in the Unity download vs the patch level used by KSP.

Link to comment
Share on other sites
HebaruSan

HebaruSan

Posted
Posted

m7uvGvn.png

Are there any known SPH memory leakers in this mod list? I assume I'm going to have to uninstall something to be able to play.

Spoiler

AirplanePlus 8.1
AntaresCygnus 1:v1.1
B9PartSwitch v1.5.3
BDArmory v0.11.0.1
CameraFocusChanger v1.0.1.0
Chatterer 0.9.91
ChopShop 0.10.1.1
CommunityCategoryKit 1.2.1.0
CommunityResourcePack 0.6.4.0
CraftImport 0.8.5.1
DecalStickers 1.1.1
DeployableEngines 0.3.3
DistantObject v1.8.1
DistantObject-default v1.8.1
EngineerLevelFixer 0.3.2.0
EngineLighting 1.5
EnvironmentalVisualEnhancements 2:EVE-1.2-2
FilterExtensions 2.8.0.2
FilterExtensionsDefaultConfig 2.8.0.2
Firespitter v7.5.0
FirespitterCore v7.5.0
FirespitterResourcesConfig v7.5.0
HullcamVDSContinued 0.1.6
Impact v1.5.0
IndicatorLights 1.2.7
IndicatorLightsCommunityExtensions 1.2
InfernalRobotics v2.0.5
InterstellarFuelSwitch 2.3.2
InterstellarFuelSwitch-Core 2.3.2
JettisonFuel 1.0.3
JSIAdvancedTransparentPods V0.1.13.0
KAS 0.6.2.0
KerbalEngineerRedux 1.1.2.8
KerbalHacksDroptankWrapper 1.0
KerbalKrashSystem 1:0.3.8
KerbalPlanetaryBaseSystems v1.3.10
KerbinRoverOffRoadVehicles 1.0.5
KerbnetController 2.1
KHWearableKISProps 1.0
KIS 1.4.0.0
Konstruction 0.1.8.0
Kopernicus 2:release-1.2.2-2
kOS 1.0.3
kOSforAll 0.0.5
KustomKerbals 1.1
LanderControl 814
LandingHeight 2.0
LaserDist v0.9.4
LCDLaunchCountDown 1.7.8
MalemuteRover 0.2.4.0
MarkIVSpaceplaneSystem 2.3.3
MasterTechAerospace 1:1
MemGraph Release_1.1.0.3
Mk1CabinHatch 0.2.0
MK1StkOpenCockpit b0.9
Mk2Expansion 1:1.7.26
Mk3Expansion 1.3
ModularFlightIntegrator 1.2.3.0
ModularRocketSystem 1.13.1
ModuleManager 2.7.5
NavHud 1.3.3
NearFutureProps 0.5.3
NEBULADecalsContinued 0.1.1.1
OPTSpacePlaneMain 1.9.2
OPTSpacePlaneParts 1.0.3.5
PlanetShine 0.2.5.2
PlanetShine-Config-Default 0.2.5.2
PoodsMilkyWaySkybox 1.0.0
PortraitStats 12.0
QuickBrake v1.31
QuickCursorHider v1.05
Races! autodetected dll
RasterPropMonitor 1:v0.28.0
RasterPropMonitor-Core 1:v0.28.0
RCSBuildAid v0.9.1
REPOSoftTech-Agencies V1.4.4.0
RetractableLiftingSurface 0.1.1
RLAContinued 14.0.1a
Scatterer 2:v0.0256
Service-Compartments-6S 1.3
ShipEffects 1.0.7
ShowAllFuelsContinued 1.2.1
SimpleRefuel 0.2
SmokeScreen 2.7.0.0
SolverEngines v3.0
SpaceY-Expanded 1.3.1
SpaceY-Lifters 1.15
SurfaceExperimentPack v2.1.3
surfacelights 1.3.1.0
SVE-HighResolution 2:1.1.4
SVE-Scatterer-Config 2:1.1.4
SVE-Sunflare 2:1.1.4
SVT-HighResolution 2.0
TakeCommandContinued 1.4.5
TankLockRedux 0.2.1.6
TextureReplacer v2.5.4
ThroughTheEyesOfaKerbal 1.1.0
ToadicusToolsContinued 0.22.0
ToughLove 0.0.1
TrainPartsPack 1.1
TweakableEverythingCont 0.1.18
TweakScale v2.3.3
USITools 0.8.7.0
VesselView 1:0.8.3
WaypointManager 2.6.0
WheelsCollection 1.2
WheelSounds autodetected dll

 

Link to comment
Share on other sites
Cheesecake

Cheesecake

Posted
Posted

For me Hullcam VDS writes many issues to the log and caused many lags.

Quote

[EXC 23:43:42.785] NullReferenceException: Object reference not set to an instance of an object
    HullcamVDS.CameraFilter.RenderTitlePage (Boolean title, UnityEngine.Texture2D titleTex)
    HullcamVDS.MovieTimeFilter.OnRenderImage (UnityEngine.RenderTexture source, UnityEngine.RenderTexture target)

 

Link to comment
Share on other sites
HebaruSan

HebaruSan

Posted
Posted
59 minutes ago, Cheesecake said:

For me Hullcam VDS writes many issues to the log and caused many lags.

Thanks! I didn't have those errors, and uninstalling Hullcam VDS didn't affect the symptoms, but checking the log did lead me to some possible clues. (Yes, I forgot about KSP.log; oh, the shame.)

I won't divert this thread any further with detailed debugging, except to say that I'm still in the market for suggestions in case anyone spots another mod that might cause problems in my earlier list.

Link to comment
Share on other sites
  • 2 weeks later...
SpaceFoon

SpaceFoon

Posted
Posted
On 12/30/2016 at 10:30 PM, HebaruSan said:

m7uvGvn.png

Are there any known SPH memory leakers in this mod list? I assume I'm going to have to uninstall something to be able to play.

  Reveal hidden contents

AirplanePlus 8.1
AntaresCygnus 1:v1.1
B9PartSwitch v1.5.3
BDArmory v0.11.0.1
CameraFocusChanger v1.0.1.0
Chatterer 0.9.91
ChopShop 0.10.1.1
CommunityCategoryKit 1.2.1.0
CommunityResourcePack 0.6.4.0
CraftImport 0.8.5.1
DecalStickers 1.1.1
DeployableEngines 0.3.3
DistantObject v1.8.1
DistantObject-default v1.8.1
EngineerLevelFixer 0.3.2.0
EngineLighting 1.5
EnvironmentalVisualEnhancements 2:EVE-1.2-2
FilterExtensions 2.8.0.2
FilterExtensionsDefaultConfig 2.8.0.2
Firespitter v7.5.0
FirespitterCore v7.5.0
FirespitterResourcesConfig v7.5.0
HullcamVDSContinued 0.1.6
Impact v1.5.0
IndicatorLights 1.2.7
IndicatorLightsCommunityExtensions 1.2
InfernalRobotics v2.0.5
InterstellarFuelSwitch 2.3.2
InterstellarFuelSwitch-Core 2.3.2
JettisonFuel 1.0.3
JSIAdvancedTransparentPods V0.1.13.0
KAS 0.6.2.0
KerbalEngineerRedux 1.1.2.8
KerbalHacksDroptankWrapper 1.0
KerbalKrashSystem 1:0.3.8
KerbalPlanetaryBaseSystems v1.3.10
KerbinRoverOffRoadVehicles 1.0.5
KerbnetController 2.1
KHWearableKISProps 1.0
KIS 1.4.0.0
Konstruction 0.1.8.0
Kopernicus 2:release-1.2.2-2
kOS 1.0.3
kOSforAll 0.0.5
KustomKerbals 1.1
LanderControl 814
LandingHeight 2.0
LaserDist v0.9.4
LCDLaunchCountDown 1.7.8
MalemuteRover 0.2.4.0
MarkIVSpaceplaneSystem 2.3.3
MasterTechAerospace 1:1
MemGraph Release_1.1.0.3
Mk1CabinHatch 0.2.0
MK1StkOpenCockpit b0.9
Mk2Expansion 1:1.7.26
Mk3Expansion 1.3
ModularFlightIntegrator 1.2.3.0
ModularRocketSystem 1.13.1
ModuleManager 2.7.5
NavHud 1.3.3
NearFutureProps 0.5.3
NEBULADecalsContinued 0.1.1.1
OPTSpacePlaneMain 1.9.2
OPTSpacePlaneParts 1.0.3.5
PlanetShine 0.2.5.2
PlanetShine-Config-Default 0.2.5.2
PoodsMilkyWaySkybox 1.0.0
PortraitStats 12.0
QuickBrake v1.31
QuickCursorHider v1.05
Races! autodetected dll
RasterPropMonitor 1:v0.28.0
RasterPropMonitor-Core 1:v0.28.0
RCSBuildAid v0.9.1
REPOSoftTech-Agencies V1.4.4.0
RetractableLiftingSurface 0.1.1
RLAContinued 14.0.1a
Scatterer 2:v0.0256
Service-Compartments-6S 1.3
ShipEffects 1.0.7
ShowAllFuelsContinued 1.2.1
SimpleRefuel 0.2
SmokeScreen 2.7.0.0
SolverEngines v3.0
SpaceY-Expanded 1.3.1
SpaceY-Lifters 1.15
SurfaceExperimentPack v2.1.3
surfacelights 1.3.1.0
SVE-HighResolution 2:1.1.4
SVE-Scatterer-Config 2:1.1.4
SVE-Sunflare 2:1.1.4
SVT-HighResolution 2.0
TakeCommandContinued 1.4.5
TankLockRedux 0.2.1.6
TextureReplacer v2.5.4
ThroughTheEyesOfaKerbal 1.1.0
ToadicusToolsContinued 0.22.0
ToughLove 0.0.1
TrainPartsPack 1.1
TweakableEverythingCont 0.1.18
TweakScale v2.3.3
USITools 0.8.7.0
VesselView 1:0.8.3
WaypointManager 2.6.0
WheelsCollection 1.2
WheelSounds autodetected dll

 

 

My graph looks the same way is SBH/VAB. You can make the stutter almost stop by using all the heap you can but when the green bar reaches the top it can stutter literally half a second every second.

Lot of similar mod on the list too.

Link to comment
Share on other sites
Padishar

Padishar

Posted
  • Author
Posted (edited)
19 hours ago, SpaceFoon said:

My graph looks the same way is SBH/VAB. You can make the stutter almost stop by using all the heap you can but when the green bar reaches the top it can stutter literally half a second every second.

Lot of similar mod on the list too.

The graph really shouldn't look like this.  It appears that some mod is not only generating a lot of garbage, but tthe amount it generates increases rapidly over time.  Were you doing anything in the editor (i.e. did you load/build something) or was it just sat there empty?  I think the best bet is probably to do a "simple" binary chop to locate the problem mod.  E.g. on a separate copy of your KSP installation remove half of the mods (move them to a temp folder) and test.  If it still happens then empty the temp folder and repeat removing half the remaining mods (move the into the temp folder again).  If it doesn't happen, then delete all the mods remaining in GameData and move half of the ones in the temp folder back in.  Test again and repeat until you've narrowed it down to the mod at fault.  Obviously, some mods rely on others to function correctly so these will need to be kept together.

However, the most likely cause I can think of is one of the mods that messes with the part categories and/or crew assignment.  You should probably try removing those first.  If that doesn't fix it then continue as described above.

If you have no luck then I would be willing to test a smaller subset of your mods to try to work it out.  If you can eliminate all the very large mods (filesize) then you could zip up your whole GameData folder (except the Squad folder) and PM me a link to it and I'll give it a go with the Unity memory  profiler...

Edit: and @HebaruSan too... :wink:

Edited by Padishar
Link to comment
Share on other sites
HebaruSan

HebaruSan

Posted
Posted
9 hours ago, Padishar said:

The graph really shouldn't look like this.  It appears that some mod is not only generating a lot of garbage, but tthe amount it generates increases rapidly over time.

Is it even garbage? It looked to me like memory that was being allocated linearly every cycle and then not even going through the garbage collector, i.e. a traditional leak; but if you tell me that's not how your mod works, I'll believe it.

9 hours ago, Padishar said:

Were you doing anything in the editor (i.e. did you load/build something) or was it just sat there empty?

When I took my screenshot, I had maybe 6 stock parts assembled, but the graph looks like that in the SPH even with no craft.

9 hours ago, Padishar said:

I think the best bet is probably to do a "simple" binary chop to locate the problem mod.  E.g. on a separate copy of your KSP installation remove half of the mods (move them to a temp folder) and test.  If it still happens then empty the temp folder and repeat removing half the remaining mods (move the into the temp folder again).  If it doesn't happen, then delete all the mods remaining in GameData and move half of the ones in the temp folder back in.  Test again and repeat until you've narrowed it down to the mod at fault.  Obviously, some mods rely on others to function correctly so these will need to be kept together.

Yeah, that's clearly the way to proceed. The trick is mustering the motivation to go through all of that rather than starting over with some other modpack theme or muddling through with 3-minute SPH sessions. I'll post my results if I come back to this.

9 hours ago, Padishar said:

If you have no luck then I would be willing to test a smaller subset of your mods to try to work it out.  If you can eliminate all the very large mods (filesize) then you could zip up your whole GameData folder (except the Squad folder) and PM me a link to it and I'll give it a go with the Unity memory  profiler...

That's a very generous offer; I'll take you up on it if I finish my own binary search.

Link to comment
Share on other sites
Padishar

Padishar

Posted
  • Author
Posted
2 minutes ago, HebaruSan said:

Is it even garbage? It looked to me like memory that was being allocated linearly every cycle and then not even going through the garbage collector, i.e. a traditional leak; but if you tell me that's not how your mod works, I'll believe it.

Each column of the graph is basically the "Last" value from the top of the window.  This is the amount that the allocated size of the heap increased by in the last second (approx).  The columns with a red background indicate that at least one garbage collection ran during that second.  If it wasn't garbage being collected then that rate of memory allocation would run you out of memory very quickly.  The Min, Cur and Max values show the allocated size of the heap itself, Min is after the most recent garbage collection, Cur is the current value and Max is before the most recent garbage collection.  So, the difference between Min and Max is approx. the amount of garbage collected by the last collection (actually more than this is collected because that counts only 4k pages that are completely freed up during the collection).

"Traditional" leaks aren't really possible in C#, that's the whole point of garbage collection based systems.  These happen when the programmer throws away their reference to a bit of memory, or overwrites it with a reference to a newly allocated bit of memory, so they can not then manually delete it.  In a garbage collection system, the collector will detect that the original block no longer has any references to it and it will be collected.  However, due to the hybrid nature of Unity, combining the Mono runtime with a "normal" C++ program, it is possible for the different memory allocation systems to clash which can cause what look a bit like "traditional" leaks but this requires a mod that makes use of Unity C# objects that have a corresponding object in the C++ world.  I don't think this is happening in this case, instead I suspect that some mod is doing some operation on a list (in a way that generates excessive garbage, e.g. some complex bit of LINQ code) on every frame but is also adding items to that list regularly so the amount of garbage generated each time increases fairly linearly.  I wouldn't be surprised if it turns out that there is simply a missing call to List<T>.Clear() somewhere...

Link to comment
Share on other sites
SpaceFoon

SpaceFoon

Posted
Posted (edited)

Ok narrowed it down to Craft Import. Assuming its the only one doing this. Got a stock game with just mem graph and craft import running and the graph was identical to one just posted.  

Thanks for the great explanation I am actually trying to learn a bit about this stuff and that helps.

 

Edit: 100% confirmed its CraftImport. I'm running 100 mods and getting a collection every 160 seconds and the heap is not growing. Memgraph also saves my experience with all those mods. I just let it use all of my 16 gigs and stutter is reduced a lot.

Edited by SpaceFoon
Link to comment
Share on other sites
Padishar

Padishar

Posted
  • Author
Posted
39 minutes ago, SpaceFoon said:

Edit: 100% confirmed its CraftImport. I'm running 100 mods and getting a collection every 160 seconds and the heap is not growing. Memgraph also saves my experience with all those mods. I just let it use all of my 16 gigs and stutter is reduced a lot.

Good job on tracking it down, @HebaruSan also has that installed so, hopefully, removing it will fix his issue too...

Have you reported it to the CraftImport author?

Link to comment
Share on other sites
DeadOnDuna

DeadOnDuna

Posted
Posted

What a great mod :0.0: Made KSP playable again on my potato pc ( 80+ mods with 4GB RAM ). I went up from 5 – 7secs to 30+ secs intervals. Thanx @Padishar

And know my stupid question: Is there a way to use SSD space to gain extra memory benefits ? (ok I’m 99.9% sure that it’s a silly question )

I play at 95% of ram (10 -15 FPS (good enough for me)) and when I do a manual “clear RAM” (with a third party program )more than 1,5 GB gets released and I get 20 – 30 FPS… is there any way to improve my GC intervals??? 

(I don’t care about FPS,, 15ish is good enough for me )

Link to comment
Share on other sites
damerell

damerell

Posted
Posted (edited)
21 hours ago, DeadOnDuna said:

Is there a way to use SSD space to gain extra memory benefits ?

The best you can do is to ensure your swap space (Linux) / page file (Windows) is on the SSD. That's used when physical memory is exhausted.

Edited to insert correct quote.

Edited by damerell
Link to comment
Share on other sites
Guest
This topic is now closed to further replies.
 Share
Go to topic listing
×
×
  • Create New...