Compressing .mbm texture files pre-run to speed up load times, have some questions.

[Toasttify]

So as most know kerbal space program has a ridiculous long load time, especially when one has many mods and extra parts added. So far it seems the main culprit is that .mbm files store the textures uncompressed in the form of .pbm images, and every time the game runs it maxes out cpu and compresses every single texture for that load only and discarding them after ksp is quit.

To this point my program scans the part folder, locates the .mbm textures and edits their hexadecimal to turn the file into a normal bitmap image, now here is some questions that hopefully someone can answer.

1.When ksp compresses the texture, which format does it use, simple dxt or another.

2.How could I go about disabling the compression and force it to use the pre-compressed textures?

3.Would you guys even want a utility like this?

[Devo]

So as most know kerbal space program has a ridiculous long load time, especially when one has many mods and extra parts added. So far it seems the main culprit is that .mbm files store the textures uncompressed in the form of .pbm images, and every time the game runs it maxes out cpu and compresses every single texture for that load only and discarding them after ksp is quit.

To this point my program scans the part folder, locates the .mbm textures and edits their hexadecimal to turn the file into a normal bitmap image, now here is some questions that hopefully someone can answer.

1.When ksp compresses the texture, which format does it use, simple dxt or another.

2.How could I go about disabling the compression and force it to use the pre-compressed textures?

3.Would you guys even want a utility like this?

Any utility that lowers the loading time of KSP I would almost pay money for. It's frustrating having to keep multiple copies of KSP, ones for modding, ones for playing ... if you had to load every part up every time you need to test a part you have made, it would drive you insane, so you basically take all parts out except the ones you are working on for a fast load.

If you can give me a utility that lets me load all my parts in less time (and I mean considerably less) I will send you a cheeseburger.

[FatherSarge]

Where is this magical program you speak of???

[Bluegobln]

I believe the intent is to move to using user editable PNG textures for every part in the future. When this happens the compression makes more sense, I think. It begs the question: can't we use PNG's now? Then why the HELL don't people use them?

However somehow having the option to enable pre-compressed textures for parts that wish to use them in the CFG file would be INCREDIBLE. My guess is it would take only a few simple changes to the code in the game, and an addition to the code for part CFG's, and bam, no problem! Course then we'd have to compress things ourselves when making parts, but that's not really difficult to set up.

[iamaphazael]

What do you mean that the game compresses files on load? That doesn't make any sense. Do you mean that it uncompresses them?

[BlazingAngel665]

What do you mean that the game compresses files on load? That doesn't make any sense. Do you mean that it uncompresses them?

No, no he doesn't.

[electronicfox]

Faster loading times would be a godsend, I dont really know the innards of KSP so I cant help with the info you need, but you have embarked upon a noble and mighty quest that would help many.

[iamaphazael]

No, no he doesn't.

Thanks. That was helpful

[TracerFox]

What do you mean that the game compresses files on load? That doesn't make any sense. Do you mean that it uncompresses them?

Okay, here's how current KSP loading works. Right now most(all?) parts use uncompressed textures in their part folders. When starting KSP, these textures and models are loaded into memory (every single one), and the textures are then compressed into an internal format to cut down on memory usage. Problem is, once you exit the game, these compressed textures are dumped from memory and lost. This means that KSP has to re-load and re-compress those textures *every single time* the game is started. This is why the game takes so long to load if you have a lot of parts.

What the OP is proposing is an addon which will *save* those compressed textures to a file. When KSP is launched, the plugin will search for pre-compressed textures, and if found will load those and skip the normal KSP loading process.

Make more sense now?

[iamaphazael]

Okay, here's how current KSP loading works. ...<useful information>... Make more sense now?

Thank you. Yes, that does make more sense, at least in that it helps me to understand what's going on. It still doesn't make much sense to me in terms of why.

I understand that the game is a memory hog, and so I understand the motivation behind them wanting to reduce the amount of RAM that's sitting around not being used, and as far as that goes, it certainly makes sense to only do that work once and not have to repeat it each time the game loads. Although I do wonder how much of a savings it would really amount to, since the amount of work to do is still linear in the number of parts. In other words, even with saving a precompressed version, that file still has to be read from disk, so the time savings amounts to (disk access time for the difference between the compressed and uncompressed versions) + (the time it takes to perform the compression). I'd assume that disk access is going to outweigh compression time here, so how much savings we'd get out of precomputing it depends largely on what kind of compression ratio we're talking about.

But the thing that doesn't make sense to me is why the game would do this compression thing at all. I mean, the game is also a big CPU hog, and it seems to me that spending processor cycles on de-compressing something that they already had as a raster is a poor trade-off. This depends somewhat on how often this happens (whether it's just when the flight scene loads up? each time a new instance of a part gets loaded into the flight scene? continuously as the view angle on each part changes?). I guess that this concern is dependent on whether we're talking about proper compression, or whether it's some sort or internal representation that is a necessary step in mapping textures onto their 3d models, and having this "compressed" version actually speeds up the rendering process versus doing it straight from a raster. I don't really know anything about game engines; I'm just an algorithms guy.

So I'm just going to get off this thread because there are obviously people who know more about the subject than I do. But in any case, thanks for actually taking the time to answer my question

[Bluegobln]

As of version 0.11, all part textures are compressed to DXT format on load, to enable mip-maps and improve performance.

I assume this is still the case. I have hopes for your ideas about keeping compressed versions of the textures to hugely improve the game's load times. Basically I see a few simple needs here:

1. Need to be able to keep pre-compressed textures and load them automatically.

2. Need a way to manually tell KSP to re-compress the textures again, such as when you install new parts, or update old parts.

3. As an improvement on #2, a way to automatically detect changes to the parts folder, and thus automatically know to compress files again. (Check to see if new parts installed? Check to see if files' date modified has changed?)

How much of this can be done outside KSP itself is well beyond my understanding of software. I'd guess a lot of it would have to be implemented by the devs themselves, though they sure would have an interest in improving their game in this way I think. Maybe if we rattle some pots and pans they'll hear us over the sound of the rocket engines.

[JeBuSBrian]

As a game developer myself, I can only say that this scheme of theirs makes absolutely no sense. It would serve them well to look at games like The Elder Scrolls or even Paradox's strategy games, where texture caches are generated just once, then used repeatedly. The other thing Squad has done which baffles the mind, is that they've completely borked Unity's own batching mechanisms by forcing every part to use its own texture file. This means that all geometry is forced to use its own draw call, rather than being combined for any parts that may share texture resources. For modders who use common textures across multiple parts, this could save dozens of draw calls per frame.

[ghost010]

they probably have a reason for their own coding(having it more in their own hand,and maybe the upgrade from 3 to 4[and maybe even from free to pro]) but than.. i dont understand why to convert the textures to DXT on load,and not just make them DXT from the start(when creating the parts)

Edited May 16, 2013 by ghost010

[JeBuSBrian]

they probably have a reason for their own coding(having it more in their own hand,and maybe the upgrade from 3 to 4[and maybe even from free to pro]) but than.. i dont understand why to convert the textures to DXT on load,and not just make them DXT from the start(when creating the parts)

This makes me wonder whether or not pre-compressed DDS files could be packed into the MBMs rather than PBM, RAW, or PNG. I know that Squad's silly mechanism strips the headers off those file formats, so I wonder whether or not the engine would recognize a DDS even without its header included.

I think I shall test this now.

Edit:

Initial tests with DXT1, DXT3, and DXT5 have failed. The texture becomes corrupted in game. I guess pre-compressing them isn't going to work.

Edited May 16, 2013 by JeBuSBrian

[Zesty Donut]

Don't worry guys! I heard the devs have/are going to resolved it in .20! I'm not sure what they will do but they said reduced loading times by a huge amount!

[b0s]

Hi

Are you still working on a mbm batch convetor thingie? Because this could also lower memory usage significantly and allow more mods without crashing.

[sidfu]

if u crashing due to memory usage remove parts from mods that u dont use.

[Charise]

I highly suspect this is something that Squad is going to have to implement. It is going to require tight integration with how resources are loaded, which I kind of doubt we can mod effectively. If you will recall from Harv's rant on the subject , the amount of work to improve a feature begins to increase the more complete the feature is. In this particular case, in order to to go from "we can load parts now!" to "we can load parts faster!" is probably significantly more work than it required to load parts in the first place. A significant amount of time that the developers could be using to work on things that don't work at all (like career mode, apparently).

There are significant reasons why any major game now-a-days caches game resources, or uses a single really big file to store everything (or both). Reasons that will eventually compel Squad to do the same.

Now, for the not so faint of heart, I am going to address some technical aspects, sort of in reply to these.

Okay, here's how current KSP loading works. Right now most(all?) parts use uncompressed textures in their part folders. When starting KSP, these textures and models are loaded into memory (every single one), and the textures are then compressed into an internal format to cut down on memory usage. {{Emphasis added}} Problem is, once you exit the game, these compressed textures are dumped from memory and lost. This means that KSP has to re-load and re-compress those textures *every single time* the game is started. This is why the game takes so long to load if you have a lot of parts. {{snip}}

Thank you. Yes, that does make more sense, at least in that it helps me to understand what's going on. It still doesn't make much sense to me in terms of why.

I understand that the game is a memory hog, and so I understand the motivation behind them wanting to reduce the amount of RAM that's sitting around not being used, and as far as that goes, it certainly makes sense to only do that work once and not have to repeat it each time the game loads. Although I do wonder how much of a savings it would really amount to, since the amount of work to do is still linear in the number of parts. In other words, even with saving a precompressed version, that file still has to be read from disk, so the time savings amounts to (disk access time for the difference between the compressed and uncompressed versions) + (the time it takes to perform the compression). I'd assume that disk access is going to outweigh compression time here, so how much savings we'd get out of precomputing it depends largely on what kind of compression ratio we're talking about.

But the thing that doesn't make sense to me is why the game would do this compression thing at all. I mean, the game is also a big CPU hog, and it seems to me that spending processor cycles on de-compressing something that they already had as a raster is a poor trade-off. This depends somewhat on how often this happens (whether it's just when the flight scene loads up? each time a new instance of a part gets loaded into the flight scene? continuously as the view angle on each part changes?). I guess that this concern is dependent on whether we're talking about proper compression, or whether it's some sort or internal representation that is a necessary step in mapping textures onto their 3d models, and having this "compressed" version actually speeds up the rendering process versus doing it straight from a raster. I don't really know anything about game engines; I'm just an algorithms guy.

So I'm just going to get off this thread because there are obviously people who know more about the subject than I do. But in any case, thanks for actually taking the time to answer my question

While reducing the amount of memory used is nice, it isn't really the point. I am going to introduce a Golden Rule: It takes longer to find something to do than to do it. That is, the amount of time it takes to get the appropriate data to the appropriate chip is a significant part of the time it takes for that chip to do the things you want it to. Little details like the cache miss penalty don't phase algorithm guys, it is O(1), dang it! (For sufficiently large values of 1.)

The great thing about these particular compression algorithms is that they are supported in hardware by virtually any video device worth talking about made in the last decade. Because they (usually) operate on a fixed 4:1 compression ratio, and the video card supports the format, when you transfer textures from main memory to video memory along the relatively slow bus in between, you only need to send 1/4 of the data as you would for an uncompressed image. This means that the video card can spend more time doing the things you want it to, and less time waiting for you to tell it what things you want it to do. This is a good thing.

But it gets better. The video card actual stores the textures compressed in video memory. Because video memory is generally pretty limited, this means that you can fit more textures and whatnot into video memory. In addition, if you have more stuff to draw than you have video memory available--which is generally the case--you have to resend textures that the video card dropped because it didn't have room for them "occasionally." But you are resending 1/4 of the data... less often. Fantastic.

But wait, there is more. The GPU can actually operate on these compressed images more or less in situ; that is, basically any operation that it can perform on a texture, it can perform directly on a compressed texture, in hardware, without having to decompress the image first. And because the compressed texture is 1/4 the size of a uncompressed texture, these operations require 1/4 of the operands to get loaded into registers across the relatively slow bus between the video memory and the GPU. The GPU can probably work on them as fast or faster than an uncompressed texture.

Is this sounding too good to be true? It is. There are a number of limitations.

The first (and only one that is really relevant) is that they are patent encumbered. A portion of the cost of your video card goes to pay for the license fee that the manufacturer must have to support these algorithms. A portion of the price of Windows goes to pay the license fee that Microsoft must pay to support them in DirectX. This is all well and good, unless you want your program to run reliably on non-Windows platforms. (Especially the ones where people don't like to pay license fees.) There are alternatives that, AFAIK, carry more or less all of the same benefits at a slight image quality penalty without being patented. The upshot of this is that a contentious developer cannot distribute the textures pre-compressed. They need to ascertain the best supported format on a particular system, and cache the compressed results to be subsequently reloaded as needed.

This caching business has pretty significant benefits in addition to avoiding re-compression on every load. The method in which all the data is currently stored is... suboptimal from a performance perspective. And it goes back to our Golden Rule.

When reading a whole bunch of relatively small files, the cost of FINDING those files is a significant portion of the cost of READING them. Let us suppose that I want to read the file "C:\Users\me\Desktop\KSP_win\GameData\Squad\Parts\Aero\CanardController\model000.mbm". Let us further suppose that the guy at Microsoft that wrote the filesystem was a little naive... mostly so we don't have to go over all of the provisos. (Generally, if you used it recently, it is probably cached; if it was adjacent on disk with something you recently read, it is probably cached.) Finding a file (naively) goes something like:

1. Read the Master File Table (MFT) to find the entry for the root directory of "C:"

2. Open the file $Secure, read the list of Access Control Lists (ACLs) to find the ACL that controls access to this directory.

3. Determine if the current user has the appropriate permissions. (This probably involves reading stuff from somewhere... you know, enumerating group membership and whatnot, but I wouldn't know where offhand.)

4. If so, use the disk address provided in the MFT entry for the directory to find the listing of the contents of the directory and search it for the entry for "Users"

5. Using the information we just obtained, find the entry in the MFT for the directory "Users"

6. Open the file $Secure, blah blah blah, the current user has permission to open the directory

7. Use the address provided in the MFT to find the listing on disk, read it, and find the directory "me"

...

N. Using the information we just obtained, find the entry in the MFT for the file model000.mbm

N+1. Ascertain if the current user has the appropriate privilages.

N+2. Read the file.

N+3. Oh yeah, by the way, update the Last Accessed Time for all of the files and directories we just touched. (Wait, we have to write to disk even though we just wanted to read? Whose bright idea was THAT?) (This operation is actually deferred for potentially an hour, but I mention it because we are increasing the disk load when we would rather our computer was focusing on playing KSP.)

From the naive approach, you can see just what a travesty trying to find a small file could be. Even if you factor in all of the provisos it is still a significant penalty.

But wait, it gets worse. When you defrag your hard drive, like everyone tells you you should, it is just making sure that individual files are in one contiguous piece. It doesn't do anything about related files. Even though model000.mbm and model001.mbm are in the same directory, there is a pretty stiff chance they are nowhere near each other on disk. This is important because, for those of you that are still tortured by spinning platters instead of a solid state drive, the time it takes to move the read head around (and for the disk to rotate to the correct orientation) are pretty significant--overwhelming even--to the cost of reading a few sectors. (There is our Golden Rule again.) For good performance, you want all your data in one nice continuous piece so you never have to move the read head around.

These two reason are why most fully-developed games (which KSP is by no means) go for a humongous file with everything in it instead of a humongous pile of little files. Now, I highly doubt Squad is going to move away from the humongous pile of little files because that is how to mod the game and whatnot. But by doing any necessary pre-processing and then storing all/most/some of the results in a cache file of some sort they could potentially realize some pretty significant savings on the load times. They could even, if they did it right, start to drop resources from memory that are not being used at the moment (because it is now (hopefully way) cheaper to load them from disk again when they are needed) so they don't smack their face against the 32bit address limit as much... but that is getting a little more off topic.

The trick is that doing it right is not really a walk in the park and, because of the mods and whatnot, detecting cache invalidation reliably is a pretty important to get right. The naive ways of detecting cache invalidation--"just MD5 the file!" doesn't really help you any, and I don't trust file timestamps for some reason--may or may not be robust.

In order to get it Right , this is something Squad is going to have to implement.

You might be able to Fudge It , though. My spur of the moment idea is make a mod which:

After all the resources have been loaded, for each part:
- if the part has a placeholder texture, load the real pre-compressed texture(s) for that part from the cache to replace the placeholder
- otherwise, store the compressed real texture(s) in the cache and replace (with a backup of course) the texture file with a placeholder

The chances of realizing any appreciable savings with this method, however, don't seem compelling enough to outweigh the effort it would take (for me) to try it to see. If someone has this modding thing in the bag and has nothing better to do, go for it!

So the bottom line is that texture compression is a good thing, and I can't wait for Squad to run out of other things on the TODO list. Mostly because the other things on the TODO list are probably pretty cool, and then I wouldn't have to wait as long to get to play with those cool things. The status quo does give me an excuse to go make a sandwich before I start crashing things, though...

I am actually kind of scared at how long this post probably is. **Averts eyes from screen in horror while click "Submit" button**

[Charise]

Wait... I think I just double-posted. I painstakingly typed out a novel, posted it, went "AAAARG, something went wrong and lost everything!", painstakingly retyped it, and posted it again and this time managed to catch a glimpse of the fine print on the screen that flashed up briefly. And by this time I don't even remember if there even WAS a screen with fine print the first time around or not. Soooo... if your face just exploded from five novels hitting it all at once... Ooops. I'm sorry.