ANOTHER how to load KSP faster (1.4.3)

[Corona688]

37 minutes ago, Lisias said:

I don't know what the memory management does, and I don't care.

Clearly.

Quote

None of them never are released again.

For the most part memory isn't returned by a program, it gets taken by the OS.  It can tell what's recently used - an important hardware CPU feature - letting it do this without you or your programs having to worry about it much or explicitly call the OS very often.  Again, there are exceptions for large blocks.

Quote

On top of that, Unity simply does not know when stop. And since it never returns memory neither, you have the offense added to the injury.

Some patterns of memory use are really bad for the strategy the OS polices RAM with.  Fragmented memory is especially a problem.  You can waste 60% of your RAM and still have every page "recently used".

Quote

It's not a problem for "modern" garbage collectors from almost 10 years now.

Memory fragmentation, like disk fragmentation, isn't solved - it's mitigated and policed.  And like certain patterns of access are really bad for disk fragmentation, certain patterns of memory access are really bad for memory fragmentation.

Quote

And once you have some way to control and predict the GC behaviour, you can choose the times when a full defragmentation can happens without too much FPS impairing. Switching to Map, Tracking Station or any other KSC "building" is an excellent window of opportunity for a more laborious and comprehensive memory defragmentation.

That would be nice, yes.  Don't mistake my explaining a problem for excusing it.

Quote

Not to mention that, well, there're heap use approaches that minimizes fragmentation itself - at expense of some memory waste, but hell - it's still better than what we get now.

By any angle I look, Mono's GC is not yet top notch and Unity's approach to the problem is worse than the problem. And yes, KSP is the "victim" here.

Garbage collection encourages the kind of programming that only a garbage collector could tolerate, I think.

Edited May 17, 2018 by Corona688

[Lisias]

9 hours ago, Corona688 said:

Clearly.

I will kindly ignore this. This is not the place for a system programming lesson, neither about teaching the minimal courtesy while discussing technical matters on a public forum.

 

9 hours ago, Corona688 said:

For the most part memory isn't returned by a program, it gets taken by the OS.  It can tell what's recently used - an important hardware CPU feature - letting it do this without you or your programs having to worry about it much or explicitly call the OS very often.  Again, there are exceptions for large blocks.

For most part, programmers don't handle the problem and doesn't care. Some of them, however, have to write daemons for servers that should work cooperatively with another daemons, i.e., the guy must return back to system the memory it get once it finishes its job. Don't worry, I already had read Tannenbaum. I know how systems work, I wrote low level system utils for some of them.

And please note that in any moment I said that GLIBC returns heap memory to the system.

But please note, also, that Linux is not the only O.S. around here. Take a look on this page and be aware that there are people working (or with experience) with FreeBSD, Solaris and other UNIX variants where the norm is memory allocation using mmap. Hint: Take a look on FreeBSD and the Hoard. Not a FreeBSD fan (or just don't know what it really is)? Try Google then.

The UNIX world is far bigger that you think. Don't reduce it to Linux and Gnu.

9 hours ago, Corona688 said:

Memory fragmentation, like disk fragmentation, isn't solved - it's mitigated and policed.  And like certain patterns of access are really bad for disk fragmentation, certain patterns of memory access are really bad for memory fragmentation.

I have a problem: I need to run concurrently programs that needs, exporadically, big hunks of memory. Some of them run now and then, and when they does a huge load of memory is taken from the system. Other ones "pulses": some times they need lots of memory, sometimes they don't.

Well... Some programs take the memory, use it and then give it back to the system. Others, don't - and this ones are a problem.

I don't care if the first ones mitigated, policed, avoided or just plain solved the god damned problem - all what matters to me is that they solved my problem: I can run them on cheaper appliances and, then, can spend the money on redundancy or in more hardware to be used on yet more clients.

Or, in this specific case... My problem is that I can't run KSP satisfactorily on a hardware that is twice the hardware my son uses to run another huge space game - but granted, my son doesn't mods heavily the game as I do, so... :-)

9 hours ago, Corona688 said:

That would be nice, yes.  Don't mistake my explaining a problem for excusing it.

Yeah... I think I made that mistake. My apologies.

9 hours ago, Corona688 said:

Garbage collection encourages the kind of programming that only a garbage collector could tolerate, I think.

There're programming patterns and yet more programming patterns.

Some patterns are good for low-tech, commercial, office automation products. Others, suits real time requirements. There're a different GC approach for all of them, and the damned things work besides taking a toll (there's no free lunch).

Don't use office automation solutions for real time problems, and everything will be fine.

Edited May 18, 2018 by Lisias
better phrasing

[Capt. Hunt]

On 5/14/2018 at 8:27 AM, [INDO]dimas_1502 said:

1.80 GHz AMD A6 APU with Radeon R4
6 GB RAM
not so potato
 

never used that
btw it gone slow after the loading bar is on the halfway to the end

not so potato man?  No offense, but that's a glorified paperweight, I'm surprised it's running windows 8.1 like you mentioned further down.  That box has to be 20 years old.

BTW, It gets slow around 50% because that's about when KSP starts loading MM.

Edited May 18, 2018 by Capt. Hunt

[Lisias]

8 hours ago, Capt. Hunt said:

not so potato man?  No offense, but that's a glorified paperweight, I'm surprised it's running windows 8.1 like you mentioned further down.  That box has to be 20 years old.

Nonsense. The A6 is a quad core CPU - REAL quad core (no hyperthreading), and the Beema GPU is approximately equivalent to the HD 7470 and it runs X-Plane from 21 to ~5 FPS (depending of the settings). These chips hit the market around 2014, more or less.

It's a modest powered CPU/GPU for use on notebooks, so you will not get that latest FPS - but it's a honest and capable machine. This should thing runs Elite Dangerous very fine - my son''s machine is older, just a bit beefier (2.13GHz, but it's a Xeon so a very decent cache) and it does fine with graphical settings at max even by that graphic card being pretty old (something from nvidia with 512Mb VRAM).

No offense, but you must give a update on your tech knowledge.

[Corona688]

10 hours ago, Lisias said:

I will kindly ignore this.

I will kindly pretend you ignored it.

I will also kindly ignore your notes on optimization as if you read my post closely you'll see I brought up both things you mention first.

 

 

10 hours ago, Lisias said:

Yeah... I think I made that mistake. My apologies.

Fair enough.

 

[Lisias]

On 5/14/2018 at 11:04 AM, [INDO]dimas_1502 said:

MY KSP LOADS IN 4 HOURS
yeah my modlist is pretty high, but in 1.3.1 i have much more mods and it's just 30-50 minutes long
 

Well... It appears that we neglected the OP problem. Sorry, sometimes in the "heat of the battle" I tend to oversee the real issue. =D

Well, back to the real issue. I don't thing we will manage to "fix" the problem, but we can try to identify it and, then, perhaps we manage to get "insider" hints about how to mitigate the thing.

I think that would be a good idea to use GC Monitor and take a peek on what Unity is doing with the memory as KSP loads. Since your setup is taking 4 hours to complete, this can be somewhat a challenge - AFAIK, there's no persistent log features on this tool. Anyway, let's give this a try.

While loading, I identified roughly three "phases" on modded KSP loading:

1. Pre Module Manager
2. During Module Manager
3. Post Module Manager

The GC behaviour changes on each on of these phases, being relatively easy to identify them over the time line using GCMonitor:

Pre-MM Screenshots ("Phase 1")

From startup to a few minutes:

 

From startup until the Module Manager kicks in (transition from "phase 1" to "phase 2"):

 

Module Manager ("phase 2"):

 

"Phase 3" kicks in:

 

"Phase 3" on the middle if the process (almost finishing, to tell the true - GCM is overwriting the "Phase 2" at the right):

(sigh). Forum doesn't me allow to add more pictures on this post. I will try another one.
 

Edited May 18, 2018 by Lisias

[Lisias]

What I learnt until now:

• GC is very active on "Phase 1"
• On "Phase 2" (Module Manager), the GC is heavily active (see the red bars on the bottom)
• The memory use really gets high on the 'Phase 3". Interestingly, the GC is not very active on this phase.

Since Garbage Collecting is a somewhat CPU intensive task, my best guess at the moment is that the OP's issue is bitting him hard here, on that 'phase 2'.

Since we are already here, the following screenshots are from the "KSP Loaded" "phase", when the Main Menu is being shown.

This "cliff" to the right appears to be the Main Menu being loaded:

 

And these slopes appears to be how Unity handles memory. And I really don't think this is a remotely acceptable approach.

From this one, if I leave KSP alone on the Main Screen, apparently on each GC run, a bit of memory is not cleaned (probable memory leak), so the next "cliff" is a bit higher then the previous, and this repeats until KSP crashes.

Edited May 18, 2018 by Lisias
typos. as usual.

[Dimas152]

okay i got it
module manager patches applied to parts slows it down
i removed a "colorchange" MM config i got from somewhere in the forum and now it loads in 1 hour