[Testing] Compressing Textures in Memory

[AncientGammoner]

I found an interesting command while working on a plugin, basically it compresses textures in memory and I'm curious to see what it does for other people. The code is really simple:

[KSPAddon(KSPAddon.Startup.MainMenu, false)]
public class CompressRAM : MonoBehaviour
{
    void Start()
    {
        foreach (GameDatabase.TextureInfo t in GameDatabase.Instance.databaseTexture)
            t.texture.Compress(true);
    }
}

It simply tries the compress command for all the loaded textures, most of them it doesn't work because the texture is 'not readable', but it obviously manages to compress some since task manager shows:

Working Set (Memory), Memory (Private Working Set), Commit Size

Without plugin: 3554 mb, 3530 mb, 3706 mb

With plugin: 3164 mb, 3139 mb, 3306 mb

~ 400mb reduction

If you notice I run with a huge number of mods (pretty much as many as I can without crashing the game), so every little bit helps. If you want to test it out you can download the dll here: http://www./download/xd4atz8ci9b4269/CompressRAM.dll Just drop it in the Plugins directory, then post the before/after RAM usage which I'm curious to see.

[Majiir]

I experimented with this a while back (compressing immediately as they're loaded instead of at the main menu) and didn't find any textures compressed at all. Can you dump a list of textures that don't throw an exception, i.e. what kind of textures are loaded as readable?

I still think the solution here (other than Squad fixing their system) is to write a custom .mbm loader, but if .png works then we might see a major shift in what mods use.

[Gristle]

I'll give it a go. I too run as many mods as I can before KSP takes a dump. I've found that KSP crashes with a working set somewhere between 3.6Gb and 3.7Gb. Are there any logs that would be helpful in determining if this is working or what it is doing?

[Starwaster]

before (working, peak, private, commit)

2924 mb, 3008mb, 2895mb, 3157mb

after

2889, 2967, 2864, 2963

percentage drop

1.2%, 1.35%, 1.07%, 6.145%

All in all, the actual drop percentage-wise isn't very impressive except perhaps for commit. (which in some ways is more significant as it translates to lessened 'paged' or 'swap' file usage)

However a little harder to quantify is that load times between scene changes 'felt' more responsive. The numbers above might actually have been a little worse for 'after' because I pushed things a little more than 'before' to see if the superior load performance was placebo or if it was real.

My testing procedure in each case was 'start ksp, load saved game, enter tracking station, exit, enter VAB, load each page in each parts category, exit VAB, enter tracking station, visit ship. 'before' this last took ~30-45 seconds. After it seemed more like 15 seconds. Or less.

So, actual memory usage, little improvement except for commit. in-game it just feels faster and it's harder to place a numerical value on that.

Edited September 6, 2013 by Starwaster

[Moon Goddess]

I'm trying to figure out what happened with mine, I was running along fine with 4 GB of ram and having no problems, I upgraded my computer to 8 GB and suddenly I had to uninstall big mods to keep Kerbal from crashing, as it was sitting at 3.9gb when loaded. I have no idea how it worked before.

[AncientGammoner]

I experimented with this a while back (compressing immediately as they're loaded instead of at the main menu) and didn't find any textures compressed at all. Can you dump a list of textures that don't throw an exception, i.e. what kind of textures are loaded as readable?

How did you compress the textures while they loaded? Also, I was trying earlier to figure out more information about what it was compressing, but the whole thing is a mess. I mean GameDataBase.TextureInfo has bools like isReadable and isCompress but I couldn't find a coordination between those and what was being compressed. I also tried storing the names of all the textures that threw an exception then removing those from the complete list of texture names to find the ones it compressed but it gave me back like everything so I probably did it wrong (I'm pretty new to C#), also it looked like some texture names were blank. If you have a piece of code that would show which textures are actually being compressed I'd be happy to run it and figure this out.

[MOARdV]

It simply tries the compress command for all the loaded textures, most of them it doesn't work because the texture is 'not readable', but it obviously manages to compress some since task manager shows:

I wonder if this converting RGBA8 textures into DXT1 or DXT5 format? Do you see a visual difference in-game between a session with compressed vs. not compressed? What texture quality settings do you use in the game? Does changing the texture quality affect the number of textures it compressed? KSP or Unity converts RGBA to DXT on at least some PNG textures if texture quality is not at max, which can lead to artifacts on UI elements.

[AncientGammoner]

Yes I believe its converting it to DXT, I haven't noticed much of a difference but I have a huge number of parts and I don't know which textures were compressed. If you want to see what it's like, try it once, you can always delete the dll and reload the game to get it back to the way it was.

[Gristle]

My results using a similar system a Starwaster. Entered tracking station loaded Space Station in orbit, Let KSP simulate for approximately 30 seconds, then quit to Space Center, VAB, and cycle through all parts.

No Compression:

Working Set: 3,485,036

Peak Working Set: 3,487,244

Private Working set: 3,458,456

Commit: 3,648,540

With Compression:

Working set: 2,961,108

Peak Working set: 3,018,088

Private Working Set: 2,934,488

Commit: 3,114,288

So on my system I'm seeing about a 500MB reduction in memory usage.

[sirkut]

I noticed that compression option when I was working on the Reflaginator mod. Nice that you tested it out. I'm going to give it a shot.

[NathanKell]

Bump for an interesting find: At offset 0C of an mbm, if the byte is 00, it will compress (to DXT1/5 depending on alpha); if the byte is 01, it will leave as RGB24/RGBA32.

[BoZo]

Wow, I got about ~900MB less memory usage with this. I use a LOT of mods.

[AdmiralTigerclaw]

I'm trying to figure out what happened with mine, I was running along fine with 4 GB of ram and having no problems, I upgraded my computer to 8 GB and suddenly I had to uninstall big mods to keep Kerbal from crashing, as it was sitting at 3.9gb when loaded. I have no idea how it worked before.

What was probably happening is that you physically could not exceed the 4GB allocation limit. The issue with the 32 bit limitation is that it's a 32 bits of memory addressing. If you think of memory as a grid, with each memory location containing a unique, 32 bit address number. Then you can only have up to 2^32 address spaces for ram.

Since you only had 4 gigs of RAM, it was physically impossible for your computer to allocate the address numbers that were higher than 2^32. Because they didn't exist. And that's not even counting that your OS was probably taking about half of that anyway. Instead, your computer uses virtual memory (hard drive page swap) to store anything else in order to keep going. That doesn't require the software/hardware to perform memory address allocation on your RAM itself and the file allocation tables for huge hard drives have long since been adjusted to account for absurdly large drive sizes, so it never encountered the 32 bit problem with virtual ram.

Thus, when you added 4 more gigs of RAM to your system, making it 8 gig, you added an additional 2^32 memory cell addresses for your computer to access. Now the OS can go 'here, have all the RAM you need without having to swap with the hard disk drive.

Since KSP only works 32 bit, it encounters a problem when the computer tries to allocate more than 2^32 RAM addresses to it. The Unity Engine literally gets the value and since its addressing can't handle more than 32 bits, it overflows* and loses the address and/or data byte. Once that occurs, the program has no earthly idea what to do with itself. If left alone KSP would immediately start acting weird and doing things that make no sense. And I'm not talking in the sense of the Krakken Glitches the program has, but the program literally derping with all forms of weird behaviors... As it would try to take whatever garbage value the overflow caused and return that instead, resulting in chaos.

Instead either the Unity Engine or the OS detects the overflow flag and saves your data from the program by CTDing it.

*Just in case some people don't know what an overflow is. I expect most do, but I can't assume: Imagine that you can count to 1000, but not 1001 because any number above 1000 doesn't exist. You start counting apples and have 1000 or more. When you hit 1000 you have to stop because you don't have a way to even say 1001. Or alternately, you wrap around and start from 1 while making a tick on something. That would be an overflow. And the tick would technically be an overflow flag. For humans, overflow isn't an issue. But a computer has limited resources and doesn't have the ability to generate a new numbering system to account for exceeding a current limit. If it overflows, usually the result is that the data value gets pushed into another RAM cell, which may be occupied by an important value. (As in the case with Stack Overflows). So effectively, Overflows are when data values exceed the number that the computer can see. Usually these are expressed in bit lengths which tells you how many combinations there are. 2^X in terms of the bits being X.

2^32 = 4,294,967,296 bytes. Or... 4.2 gigabytes.

2^64 = 1.84467440737 xe^19 or... 18 MILLION Terabytes.

[sarbian]

@AdmiralTigerclaw you need to read about how windows allocate memory for a 32bit program under a 64bit OS, because it does not work like you think it does.

[AdmiralTigerclaw]

I'm not a windows programmer. So I just gave the electronics engineering answer to the best of my knowledge.

I know it's not the address allocation in and of itself that's causing the problem. Because that would have caused a crash the instant KSP tried to load... Because it invariably would have accessed RAM addresses across the entire memory range from zero to 8 gigs. The OS is handling that for the program.

Point I should have clarified is that the OS didn't HAVE more than 4 gigs to allocate to begin with. So no matter how much KSP wanted to use it, it never GOT allocated enough RAM to cause an overflow in the ram values within' the unity engine. With eight gigs, the OS can keep throwing RAM at KSP until it uses more RAM than the Unity Engine can 'count'.

Splitting it between physical and virtual memory likely just makes it balance the load between them, so that neither actually hits 4 gigs. Unintended memory segmenting.

But since physical memory is a higher priority than page files/virtual memory, when you actually have enough RAM, the balancing doesn't happen because physical memory gets allocated first due to superior performance. Virtual Memory is supposed to be effectively the OS's backup memory.

[NathanKell]

Yes, Unity will page if it can't allocate to its entire potential address space. If it can, however, it (buggily) decides not to page.

So yes, it's best to make sure (somehow) that KSP can only access ~3.2GB memory

[lincourtl]

Interesting. Here are the results I got.

AMD Athlon Dual Core 4850e 2.5 GHz

2 GB memory

Windows Vista Business SP2 32 bit

Testing procedure: Load KSP, go to VAB, load Kerbal X, launch to 1,500 meters, pause, record data, kill KSP process, repeat

..........................Before...............After......

Commit.............1763068............1605156

Working.............1614036............1440162

Physical.............1503144............1395468

So I'm only seeing between a 7%-9% change, which isn't much, but it did feel a little snappier. However, I am using PolecatEZ's Squad texture reduction pack. I also have a lot of non-part mods like MechJeb2, Protractor, Chatterer (just the voices and beeps), Editor Extensions, RCS Build Aid, etc.

[mrluckypants96]

Looks helpful, but my anti-virus freaks out whenever I try to dl.

[Synthesis]

Okay, so...

At the menu after launching, without the DLL--3.4 GB. I like to live dangerously.

With the DLL at the same point--2.45 GB....?

I'm not sure how long it'll hold, it but seems to work!