GameData/Squad/Spaces duplicate asset memory reduction

[somnambulist]

Now in testing -- see this post!

Hi all -

I decided to look into reducing KSPs memory footprint by removing most of the duplicate textures in GameData/Squad/Spaces The approach is similar to what I did with KWRocketry+ -- load the assets with model nodes and then use much smaller placeholder textures to keep the part loader happy (see my sig). This would reduce KSP's overall memory footprint by about 100MB.

I would, however, like to get some community and, hopefully, mod and dev feedback before I release the files. No Squad assets would be redistributed, but Squad CFGs and assets would be overwritten. My thinking is that it's no worse than a rescale of stock parts or even TouhouTorpedo's MK3 internals.

Thoughts?

Edited September 1, 2013 by somnambulist

[The Destroyer]

I would love this. I want MOAR MODS but stupid RAM limit is hindering.

[Faark]

Personally, i would prefer to not alter files but try to fix things in memory, so you could easily uninstall it without having to reinstall everything from scratch. Have you ruled out this approach? Ofc i don't know what exactly you are going to do and if it is as easy as messing around with loaded PartModules.

[somnambulist]

Personally, i would prefer to not alter files but try to fix things in memory, so you could easily uninstall it without having to reinstall everything from scratch. Have you ruled out this approach? Ofc i don't know what exactly you are going to do and if it is as easy as messing around with loaded PartModules.

There's really no way to do that outside of a plugin which unloads the assets (*well* beyond my abilities). The part loader loads every file type it recognizes in GameData so just removing references in the cfg isn't enough.

[kahlzun]

i just tend to delete the items in spaces that i don't use.

The iva for the Mk2 cockpit is worse than useless.

[somnambulist]

Update on this project. I stumbled across a probable bug where trying to change the textures on certain internals prevents that internal from loading and also corrupts any other internal using model nodes to reference that texture (http://bugs.kerbalspaceprogram.com/issues/1256 in case anyone wants to take a look). Even with that setback, I've been able to carve about 90MB out of Squad's internals without affecting the actual in-game appearance. I should have something releasable by the end of the weekend.

As part of this process I've uncovered A LOT of textures, normal maps, etc. that could easily be reduced by 25% without affecting apparent in-game quality. I won't modify and distribute Squad's IP, but I can write-up a tutorial on how to do it yourself using Photoshop and your favorite text editor and provide a list of the suggested files to edit.

[Greys]

That's not really a bug.

MODEL{} is a config node that only exists in the context of PART{}, that would be to say, it's a function of PART{} not a concept that just exists.

When you go to INTERNAL{}, MODEL{} isn't a thing, RESOURCE{} isn't either and I wouldn't expect MODULE{} to work either.

Until Squad includes MODEL{} in INTERNAL{}, or creates a similar feature in that context, you're not going to be able to simplify IVAs in that way.

Instead you may be able to rename the .mu files and textures, dump them all into one folder, and do things that way.

[somnambulist]

MODEL{} is a config node that only exists in the context of PART{}, that would be to say, it's a function of PART{} not a concept that just exists.

When you go to INTERNAL{}, MODEL{} isn't a thing, RESOURCE{} isn't either and I wouldn't expect MODULE{} to work either.

I thought that, too, but then I reread mu's post

Parts, Props and InternalSpaces can all use MODEL nodes instead of the traditional same-driectory loading system.

Works exactly like mu describes; I've got 'Squad/Spaces/' slimmed down to about 184 MB using model nodes and overwriting duplicate textures with small placeholders.

Just a teaser of the Mark 1-2 Pod running the model node "optimized" IVA. Overall memory footprint is down ~~110 MB compared to stock with full res textures.

[somnambulist]

v.1 test release

Between thunderstorms and power outages I managed to cobble together an initial test release.

In the above screenshot the first instance of KSP is stock, the second instance is after installing the memory optimization. Both were launched at the same time and left to sit idle in the VAB screen for 30 minutes. The savings in that screenshot is about 108 MB -- KSP's memory use fluctuates a bit, but 110 MB is the average savings. Your savings may vary with lower texture resolution settings.

Installation

WARNING:

• The Stock Memory Optimization pack overwrites Squad-supplied textures and CFGs. Uninstallation requires reinstalling the contents of 'GameData/Squad/Spaces/'. Making a backup of your game folder before proceeding is strongly encouraged.
  
• If you've deleted anything from 'GameData/Squad/Spaces/' you will be missing internal textures.
  
• Additional third-party internals *may* need to be reinstalled after you install the Stock Memory Optimization.
  

To install: Merge the contents of the zip with your GameData folder. Overwrite existing files.

Mac users: Download the freeware MacMerger and use it to merge the folders. It's far more reliable than doing it in the Finder.

Download v.1

Outstanding issue

• The Hitchhiker and the mk1 Pod are not happy with model nodes. They'll have to remain unoptimized for now.
  

[Torminator]

This makes me quite happy. I fully approve of a series of optimization mods.

[The Destroyer]

So this reduces it by 100 MB, nice!

[Frogbull]

Good work!

But one of the most strange thing is that the game load all in one time; why it don't stream the files ?

[ahappydude]

Good work!

But one of the most strange thing is that the game load all in one time; why it don't stream the files ?

Its magic This sounds like a great idea and plugin so i will try it today and report back =) Thanks for this, always get a high use of ksp.exe even with few mods, oh i remeber 0.13 i think or 0.12 run flawless

[somnambulist]

But one of the most strange thing is that the game load all in one time; why it don't stream the files ?

Having all the assets loaded makes switching between craft quicker.

[Greys]

You still don't need all the assets loaded. At any given time the crafts that exist in a flight state are only making use of some fraction of the parts that the instance of KSP has access to, and there's no reason to have full resolution textures for the unused parts on hand when you only need to be able to render the VAB icon; then load better textures when the player actually grabs that part to be used. If you saw my post in that one thread about load times, it took about a second and a half to load 30 1024x textures, so loading single parts in step with a player building things in the VAB/SPH, shouldn't be any sort of problem.

[Galane]

KSP runs into the same problem old CP/M and DOS programs did with their limited amount of available RAM. The common solution was loading "overlays" or modules with a small core program always loaded to manage it all.

For example a word processor would load the core part and the editor module. Then when you went to print it'd kick the editor out and load the printing module. Spell check, word count and other functions all separate modules and if the document being worked on was too large for the remaining RAM (remember, 64 kilobytes for CP/M, the word processor program and the document!) it'd get loaded in chunks from floppy disk then changes, if any, written back before loading the next chunk.

Speed wasn't too bad. If the computer had a hard drive it could be pretty darn quick since it was only displaying and working with plain text.

Programmers have gotten a bit soft and spoiled with gigahertz of speed and gigabytes of RAM and storage. The first hard drive I owned was five megabytes and with DOS and all the software I had installed it was only half full. I work with single images and documents every day that wouldn't fit on that hard drive.

This PC is a 2.5 Ghz AMD 7550 dual core with 6 gig RAM and a nVidia 9800 GT video card with a gig of RAM, running 64 bit Windows 7 - yet to get tolerable performance out of KSP I have to disable anti aliasing, disable the VAB/SPH crew and terrain scatter, use Active Texture Management and run at a lower resolution AND turn off the logging.

But as has been shown by various mods like ATM it is possible to help the performance. Would be nice if Squad would take a break from adding more features and do an optimization/performance release to clean up loose ends, shrink over-large textures and whatever else they can find to speed it up. I'd bet they'd find a lot of things to prune and tune.

[Justin Kerbice]

But as has been shown by various mods like ATM it is possible to help the performance. Would be nice if Squad would take a break from adding more features and do an optimization/performance release to clean up loose ends, shrink over-large textures and whatever else they can find to speed it up. I'd bet they'd find a lot of things to prune and tune.

Don't want to be kind of "rude" or something but it's not very relevant to the thread you've just unburied .

I already wrote about this, big issue: mostly noisy morons wants more (see madness around 0.23.5 release !), they don't care about better, improved, optimized KSP .

I'm sure they prefer 30 new parts all buggy, badly integrated in career, 30 new bugs than 2 new parts only, 10 bugs fixed for good and better memory management.

One thing which could be try is adding one or more "dummy"/holder process(es) which will just keep textures in their own memory space (~4 GB each maybe), which could be at least nice for people who have a computer with more than 4 GB ram.

[Porkjet]

Wait a minute... It's possible to use MODEL {} in an internal CFG??? That's awesome! Will help optimze my IVA's aswell.

[somnambulist]

Guessing this got necro'd because I mentioned I might be reviving the work as a replacement for PolecatEZ's reduction pack. Just deduplification, no size reductions. Any work is going to have to wait a week as I'm out of of town.

Wait a minute... It's possible to use MODEL {} in an internal CFG??? That's awesome! Will help optimze my IVA's aswell.

Jah. It works just like it does on a part.

[nanowinner]

Keep it up!

[sharpspoonful]

Guessing this got necro'd because I mentioned I might be reviving the work as a replacement for PolecatEZ's reduction pack. Just deduplification, no size reductions. Any work is going to have to wait a week as I'm out of of town.

Jah. It works just like it does on a part.

I've always been curious as to why the KSP preloaded all the texture files to RAM, when most games nowadays have placeholder textures that get swapped to full textures when needed.

It seems, and is, extremely inefficient.