[1.0][Release-5-0][April 28, 2015] Active Texture Management - Save RAM!

[rbray89]

Just realized: Is the mod BoulderCo because you're located in Boulder, Co?

I was wondering when people might notice this.

[rbray89]

The newest update doesn't go well with AlternisKerbol mod. The loading gets stuck in newduna_nrm.TGA. A file with just 3MBs from the texture folder of Alternis. I waited for 10minutes and no progress. The .Exe was running, though. It didn't freeze or anything.

The previous version, 2-14, had no issues with Alternis

If I remove just BoulderCo from the game folder, the game loads normally

did you create a config for the mod or is it just there?

[rbray89]

Another issue here with 2-15 (agressive or basic). The game freeze at launch on Boulder/Clouds/Textures/Kerbin1 (several minutes). I have no crash folder or message, it just stop loading.

KSP.log is here.

EDIT : without ATM, KSP loads fine.

Hmmm... I really should've double checked with my own mod. I'll do this.

[rbray89]

Thanks for all your hard work!

Loading time with the cached version is much improved -- in my case it's halved from about 8 minutes to about 4 minutes.

I'm still only seeing 714MB saved in 2-15 Aggressive compared to the 1.5GB savings I saw in 2-12 Aggressive. I'll happily take the 714MB, but I'm wondering why the huge difference between what you were doing in 2-12, and what you've been doing in 2-14 and 2-15 (I missed the 2-13 release).

Also, the amount of memory saved reported in the log decreased from loading 2-15 the first time to loading the cached version -- 714MB down to 701MB. That's not significant, but thought I'd mention it.

You can check your actual memory usage in system processes. There were many, many bugs in the previous calculation. It is finally fixed and reporting correctly now.

[rbray89]

Thanks for the info. I did wait for 4 minutes before giving up (windows console report KSP.exe not responding) but I'll try again. Is the very very long loading time only on the first run, or is it still long when caching is done?

once caching is complete, loading times will be about half of what you get on the first run.

[rbray89]

I'm loading KSP as I type and saw the same thing - KSP appeared frozen at that point for about 2 minutes. Eventually it did resume loading, however. It doesn't appear to be doing so any faster than in prior versions.

EDITED: Yeah, loading times don't seem changed much at all for me. Similarly, with 2-15 Aggressive, the log is showing only 657MB saved, compared to prior versions saving over a gigabyte. Any savings is very welcome, but just wanted you to know.

There were many bugs with the memory reporting in the logs before. I like to use the system resources on my computer as a check. Saves the same ammount, but it is actually reported correctly now. As for loading times, the first load should be slow, and subsequent loads should be about 1/2 the time of the first load.

[lincourtl]

There were many bugs with the memory reporting in the logs before. I like to use the system resources on my computer as a check. Saves the same ammount, but it is actually reported correctly now. As for loading times, the first load should be slow, and subsequent loads should be about 1/2 the time of the first load.

So 2-12 and earlier were reporting about twice the actual memory saved? That doesn't seem quite right either because I'm running up against the wall with fewer parts mods since 2-14. In any case, I'll try loading KSP in windowed mode with Performance Monitor running both with and without Active Texture Management.

Once again, please don't mistake my comments for complaining. I'm totally chuffed with what your mod allows me to do in KSP with my poor memory starved machine.

[rbray89]

Another issue here with 2-15 (agressive or basic). The game freeze at launch on Boulder/Clouds/Textures/Kerbin1 (several minutes). I have no crash folder or message, it just stop loading.

KSP.log is here.

EDIT : without ATM, KSP loads fine.

Yeah, just tired it out on my machine and it stopped for a bit. This is because it is a massive texture. Just let it keep going. It will load.

[team.leit]

If the memory saved was exaggerated in all the versions before 2.15, what would 389 MB of saving in 2.14 translate to in 2.15?

[rbray89]

So 2-12 and earlier were reporting about twice the actual memory saved? That doesn't seem quite right either because I'm running up against the wall with fewer parts mods since 2-14. In any case, I'll try loading KSP in windowed mode with Performance Monitor running both with and without Active Texture Management.

Once again, please don't mistake my comments for complaining. I'm totally chuffed with what your mod allows me to do in KSP with my poor memory starved machine.

Oh, it is fine. If there is some sort of issue, I want to nail it down. The older versions had issues with config interpretation, made some textures unreadable that shouldn't have been (specifically Interstellar resource maps and rasterprops images), resized images that shouldn't have been, and had major bugs in memory use due to copy paste issues. Due to these, there is some actual extra memory use in the new version in some cases, but it is necessary for some mods to work. I'm pretty darn confident in this iteration though. You can take a look at the Log to review the textures to make sure all of the rules have been followed, but I am quite certain they are.

I think you said that you didn't have any new config files, but if you did add any, they will have to be updated between 4-12 and the current version.

[rbray89]

If the memory saved was exaggerated in all the versions before 2.15, what would 389 MB of saving in 2.14 translate to in 2.15?

Between 2.14 and 2.15 it wasn't much. Due to mipmap generation, some PNG heavy packs actually took up MORE memory, which wasn't added into the calculation before. Maybe a few MB difference.

I also added more rules to protect props and stuff that displays data from being shrunk, so there is also that that adds to the memory usage.

[team.leit]

Between 2.14 and 2.15 it wasn't much. Due to mipmap generation, some PNG heavy packs actually took up MORE memory, which wasn't added into the calculation before. Maybe a few MB difference.

I also added more rules to protect props and stuff that displays data from being shrunk, so there is also that that adds to the memory usage.

Okay, thanks. Although I am not sure why the most I have been getting savings wise has been the 389 MB

[Argelle]

Thanks rbray89, you're right, after the first (long) load all is fine and nearly as fast as without your mod BUT with memory reduction. In my config, ratio first/second run times is not 2 but close to 3.5

[rbray89]

Okay, thanks. Although I am not sure why the most I have been getting savings wise has been the 389 MB

It depends on the mods you use. If you don't use a LOT of mods, you won't see much in terms of improvement.

[rbray89]

Thanks rbray89, you're right, after the first (long) load all is fine and nearly as fast as without your mod BUT with memory reduction. In my config, ratio first/second run times is not 2 but close to 3.5

Ah, Very good then! Yeah, ecat deserves major props for the idea. The caching mechanism works beautifully. Another neat side effect, is that people can go into the caching tree, and rename the cache files to include the .png extension, and they can view the newly re-sized texture! Note that normal maps have been converted to the Graphics normal map to process faster on load.

[ecat]

Yeah, just tired it out on my machine and it stopped for a bit. This is because it is a massive texture. Just let it keep going. It will load.

I have a possible improvement for these extra slow files too, in fact it speeds up all of the initial file loads. By how much? It is system dependant but a simple implementation I'd guess would be good for a 30% improvement on compression time. Still, it is extra complication for something which the cache renders redundant on all subsequent loads and so may not be worth the effort.

Thanks again rbray, it is wonderful to be able to play the game with all the mods and none of the crashing

Boulder Co. I was there once on a short tour of the US. Of all the places I visited Boulder was the one that felt most like home. Beautiful countryside too

[Castun]

So, I notice that Toolbar is listed as affected by this mod, anyway to change that? I just notice that the buttons are awfully pixelated like they're being overcompressed (particularly the icon for notifying of available Toolbar updates.)

I was wondering when people might notice this.

Didn't think anything of it until I noticed your flag avatar hah. I'll actually be in the area again in a couple weeks to visit family in Denver. Drove to Boulder a couple years ago via the scenic route, visited the brewery for the tour and some beers, was a good time.

[rbray89]

I have a possible improvement for these extra slow files too, in fact it speeds up all of the initial file loads. By how much? It is system dependant but a simple implementation I'd guess would be good for a 30% improvement on compression time. Still, it is extra complication for something which the cache renders redundant on all subsequent loads and so may not be worth the effort.

Thanks again rbray, it is wonderful to be able to play the game with all the mods and none of the crashing

Boulder Co. I was there once on a short tour of the US. Of all the places I visited Boulder was the one that felt most like home. Beautiful countryside too

Hmmm... Might be interested. I'm assuming it has something to do with threading? Right now everything is single threaded, but I could probably create a "cacher" thread to write all the newly generated files to disk.

So, I notice that Toolbar is listed as affected by this mod, anyway to change that? I just notice that the buttons are awfully pixelated like they're being overcompressed (particularly the icon for notifying of available Toolbar updates.)

Didn't think anything of it until I noticed your flag avatar hah. I'll actually be in the area again in a couple weeks to visit family in Denver. Drove to Boulder a couple years ago via the scenic route, visited the brewery for the tour and some beers, was a good time.

Hmmm... I'm not sure about why they would be pixelated. They should be in the exclusion list if all the icons are in 000_Toolbar.

Yeah, I fell in love with and in Boulder. Between the hiking, climbing, running, skiing and cycling it doesn't leave me with much free time.

[Castun]

Hmmm... I'm not sure about why they would be pixelated. They should be in the exclusion list if all the icons are in 000_Toolbar.

Not sure, but it seems the small arrow at the bottom of Toolbar to move it, as well as the Update Available button are pixelated. I think I still have an older version I can reinstall to get a screenshot of it.

Yeah, I fell in love with and in Boulder. Between the hiking, climbing, running, skiing and cycling it doesn't leave me with much free time.

Yeah, this will be our 4th time to Denver, and we've also fallen in love and started looking for jobs. Tough to find anything when you're out of state though. Might be time to just pick up and move when our lease is up.

[NoMrBond]

Hrm, upgraded from 2-14 (basic) to 2-15 (basic) and KSP won't load, hangs on the Kerbin1 cloud texture (Astronomers HD Clouds v2) with VE Clouds and City Lights mod

Reverted back to 2-14 and now it loads again

[rbray89]

Hrm, upgraded from 2-14 (basic) to 2-15 (basic) and KSP won't load, hangs on the Kerbin1 cloud texture (Astronomers HD Clouds v2) with VE Clouds and City Lights mod

Reverted back to 2-14 and now it loads again

It just looks like it hangs. It just takes a long time to process. After that, you won't have to worry about it.

[likke_A_boss]

Is there suppose to be a 200MB folder that appeared? Or did I do something wrong?

[NoMrBond]

It just looks like it hangs. It just takes a long time to process. After that, you won't have to worry about it.

Are we talking 5+ minutes just for that one texture?

[Edit] ok it did finish loading, 9 minutes for that one texture though, jebers.

Edited January 18, 2014 by NoMrBond

[rbray89]

Is there suppose to be a 200MB folder that appeared? Or did I do something wrong? http://i.imgur.com/gvdJl3m.png

Yes, it caches all the resized/normalized textures to load them faster on the next run.

[ecat]

Hmmm... Might be interested. I'm assuming it has something to do with threading? Right now everything is single threaded, but I could probably create a "cacher" thread to write all the newly generated files to disk.

I'm not sure if threading the IO would would help much, it's worth trying but the OS may be threading in background?

For my quick and very dirty test, I just chopped each texture in two and worked on the two halves in parallel...

        static int t1Height;
        static int t1Width;
        static int t1h;
        static int t1Index;

        static Color32[] newPixels;
        static Color32[] pixels;
        static Texture2D tTexture;

        static System.Object thisLock = new System.Object();

        public static void ThreadCalcHelper1( )
        {
            for (; t1h < t1Height; t1h++)
            {
                for (int w = 0; w < t1Width; w++)
                {
                    Color32 c32 = GetPixel(pixels, tTexture, ((float)w) / t1Width, ((float)t1h) / t1Height, t1Width, t1Height);
//                    lock (thisLock)
                    newPixels[t1Index++] = c32;
                }
            }
        }



        public static void Resize(Texture2D texture, int width, int height, TextureFormat format, bool mipmaps)
        {
            //Color32[] 
            pixels = texture.GetPixels32();
            int origWidth = texture.width;
            int origHeight = texture.height;
            //Color32[] 
            newPixels = new Color32[width * height];
            int index = 0;

            tTexture = texture;

                //for (int h = 0; h < height; h++)
                //{
                //    for (int w = 0; w < width; w++)
                //    {
                //        newPixels[index++] = GetPixel(pixels, texture, ((float)w) / width, ((float)h) / height, width, height);
                //    }
                //}

            t1Height = height;
            t1Width = width;
            t1h = (height / 2);
            t1Index = width * (t1h);

            Thread t1 = new Thread(new ThreadStart(ThreadCalcHelper1));
            t1.Start();

            for (int h = 0; h < (height / 2); h++)
            {
                for (int w = 0; w < width; w++)
                {
                    Color32 c32 = GetPixel(pixels, texture, ((float)w) / width, ((float)h) / height, width, height);
                    //                    lock (thisLock)
                    newPixels[index++] = c32;
                }
            }

            while (t1.IsAlive)
                    Thread.Sleep(0);

            texture.Resize(width, height, format, mipmaps);
            texture.SetPixels32(newPixels);
            texture.Apply(mipmaps);
        }

Having two or more threads process each texture may be inefficient when processing small textures, other than that the workload is guaranteed to be more or less evenly split, so not a bad option and easy to implement. The biggest drawback, with caching in place it's of marginal benefit.