A mod to bind all mods - the ultimate modder's challenge

[Moach]

Hello all,

lately I've been playing KSP as usual, in the only way I know how: With as many mods as I can cram into the limits of RAM overuse...

...needless to digress, the game crashes more than not - just as I pile on a mod heap I can begin to take proud in, I'm assailed by relentless memory allocation failures, brought on as the game haplessly shoves every single asset it finds under GameData into an unwitting <4GB swath of RAM

sub-optimal? tell me about it....

problem here is -- a certain twin brother of mine is seemingly of the belief (or was at the time he decided this years ago) that "premature optimization is the root of all evil" -- a notion that as a fellow game programmer (not affiliated with Squad), I largely disagree to.

whereas I'm sure by that he implied that optimization would come at a later point, it does seem our thirst for new shiny rocket parts has outraced development on that aspect

which leads us to our problem/opportunity/challenge at hand:

the problem:

KSP loads every single bit of "this may be needed at some point" data it finds under the folder GameData - this is widely known, for it is also the sole principle of the game's exemplary moddability

Alas, it does not account for the fact that a lot of people making a lot of awesome mods just may require more memory than one can write an address to in a mere 32 bits of binary digits -- and this is why the whole thing crashes like a house of cards in an earthquake as we earnestly satisfy our individual afflictions of the modder's addiction, and do so by uncompromisingly heaping great many unmetric tonnes of "MoAr AwEzUuhM" into our less-than-fully-prepared copies of the game (sorry, Simulator)

The (potential) Solution:

what if - KSP did NOT load every single part, internal space and cockpit decor (ok: flight instrumentation) prop it has stocked in PRIOR to actually needing them?

what if, bear* with me here - It loaded only simple, easily allocatable THUMBNAILS of each part, along with it's relevant CFG information to display in the editor? -- then EXPANDED those stand-in definitions of such assets only as they are placed on your newly built deathtrap ground-breakingly designed vessel and/or upon encountering a previously launched vessel that contains them?

* yes, I'll have a BEAR with me, a big, mean BEAR - All big and brown and covered with hair! - so you better listen to my point, since I have a BEAR, and it's WITH ME... - I think that's what the expression is meant to imply, no?

well, then -- first noticeable thing: a heavily modded game would load in not much longer than a vanilla version... why - that's pretty good, no?

and well - it quite simply, would NOT crash from running out of memory... not unless you somehow manage to heave such an unreasonably vast array of junk creative inventions up there that would collectively feature the sum of ALL the parts in your GameData catalog

and that's not just unlikely -- it's also inadvisable, and a grandly foolish, fruitless endeavor that indeed, nobody could hope to benefit from (specially not your Kermans)

so, crazy people aside -- the most remaining half of us would greatly gain from any one mod that be geared towards addressing the memory issue directly at it's core in such a way

The challenge: (any prizes beyond "thanks and optional worship" purely imaginary, bragging rights: irrevocable and ever-lasting)

To create a mod that overrides the KSP loading sequence -- allowing it to load "low res" versions of everything under GameData -- then substitutes it "on the fly" (quite literally) for the real thing, just as (and only if) it becomes needed

how to accomplish that is up to The Challenger -- one may preprocess all assets into low-res format with an external app, storing the full versions elsewhere (or even compressed) for the plugin to pickup as needed... one could also try and see if it's viable to bypass the loading scene altogether, mayhaps even with an alternative KSP.exe file

however one chooses to approach it - that is the mission

now -- who dares do this most deftly deed?

Cheers

Edited September 15, 2015 by Moach

[Elthy]

This would have been great months ago. But with 1.1 around the corner (and 64 bit for windows) the RAM saving isnt important anymore. Reducing the Startuptime would of course be great, but instead we would propably have a small lag everytime we place a new part, i would rather have long startup times...

[More Boosters]

This would have been great months ago. But with 1.1 around the corner (and 64 bit for windows) the RAM saving isnt important anymore. Reducing the Startuptime would of course be great, but instead we would propably have a small lag everytime we place a new part, i would rather have long startup times...

We need all the RAM for mods!

[11of10]

There used to be a mod for exactly that: Load on Demand

[zeropositivo]

This would have been great months ago. But with 1.1 around the corner (and 64 bit for windows) the RAM saving isnt important anymore. Reducing the Startuptime would of course be great, but instead we would propably have a small lag everytime we place a new part, i would rather have long startup times...

Not everyone has the benefit of a 64bit OS, or the RAM avaliable to benefit from 64bit in the first place. Having only 4gb of RAM on a PC is a lot more common than you might believe

[nothingSpecial]

It was funny to read the post. And yes, Load on Demand was doing exactly this, but is is abandoned unfortunately.

[Moach]

well, it's licence terms do read:

LICENSE Do whatever you want as long as you mention that your stuff is based on this and ofc do whatever is inferred by licenses of 3rd party stuff used by this project.

which is pretty neat, I guess - allowing pretty much anyone to take over it and continue on, as long as one mentions where it started and credits Faark (the original author) for it...

I'd gladly take up this project myself, in case anyone's wondering... but I fear the upcoming months of stuff I got to do (work, you know, that annoying "real life" thing) wouldn't allow me give it the dedication needed to actually see it through completion

which is why I thought of tossing the idea out for grabs...

and anyone who makes this happen will rightfully be hailed "The Great Modbinder" -- The long sung hero of memory allocation for eons cometh! ...or somesuchwise

there is the upcoming 64bit support to consider though... yet, many a mod has overlapped official new developments and continued on to expand in various ways in graceful cooperation (FAR, DeadlyReentry, to name a few)

So for now, the challenge still stands, and here we have already a good starting point to go with!

Cheers

Edited September 15, 2015 by Moach

[NathanKell]

You might also want to look at the OnDemand code in Kopernicus (since biome maps and MapSOs generally are even worse sinks for memory than textures, since you can't upload them to the GPU and you need to keep them uncompressed).

[MrHappyFace]

Does the OnDemand stuff in Kopernicus apply to the stock planets by default? If so, couldn't you install Kopernicus to possibly save some memory?

[NathanKell]

It does not, because the maps used by stock planets aren't files in GameData, they're part of the assets that are loaded automatically at KSP startup (before it processes GameData).

[TheMoonRover]

Not everyone has the benefit of a 64bit OS, or the RAM avaliable to benefit from 64bit in the first place. Having only 4gb of RAM on a PC is a lot more common than you might believe

In fact I'd go so far as to say having more than 4GB is relatively uncommon, at least for laptops.

So such a mod would still be useful after 1.1 hits.