[1.0][Release-5-0][April 28, 2015] Active Texture Management - Save RAM!

[johnsonwax]

Question for the experts.

If I want to keep a part from loading, and the part has a .cfg, a .mu, and a .tga or .png I can compress the .cfg and it'll disappear from the editor, but is KSP still loading the .mu and/or the .tga/.png? My assumption was no, that KSP parses .cfg files and then loads assets referenced by the .cfg, but I'm wondering if that's accurate - do I also need to compress the .mu/.tga/.png?

[Black-Talon]

With the 4.0 release this went from awesome to wanting it on every stock install - nice job and thank you!!

I will say I'm now struggling to configure this quite the way I'd like, my issues range from just being a texture n00b to confusion on how the config file functions.

I'm going to dive into exploration of each of these points and questions today - but maybe you already have answers or guidance?

1. If I was setting up a config for a mod/parts pack that you haven't already made a config for, how would I know what textures are normal maps? The naming convention doesn't seem to be consistently followed so there must be another way to tell - open the file and see if it's grayscale?
   
2. Given how fast the DXT loads at game start, I want this for functionality for all textures - I'm unclear though if my config settings are accidentally excluding some textures when I only wish to not scale those textures? How do I convert to DXT without rescaling?
   
3. I also want to compress all textures, right? So if even if I exclude textures from scaling, am I also excluding them from compression?
   
4. I wish there was a way to know if KSP needed to read a texture throughout runtime - maybe the log will give me clues which allow me to see if unloading textures from memory causes issues or not?
   
5. I'm expecting to make heavy use of the "min" config setting but I'm unclear (see above) what this will exclude textures from being processed with which features. I'll probably explain this better when I explore it.
   
6. I love the runtime execution of this conversion/compression/scaling - idiot proof! But given the leaks and memory issues that exist and evidently experienced by so many, what would it take to also distribute a separate executable which can create the cache? This may even allow for other optimizations, and warnings about total texture size, etc.

Anyway, that's what I'm curious about and going to look at today. Thanks again for the nice enhancement to a great game!

-Talon

Edited December 13, 2014 by Black-Talon

[Liarra Sniffles]

First off, good work so far with this!

I would highly recommend creating a simple program or .bat file to run this outside of the ksp game-loader itself, to get around the memory limit of the actual game, but other than that it's worked wonders.

[NathanKell]

johnsonwax your assumption is incorrect. KSP loads all recognized file types, and only later compiles them. So any image under GameData and not under a PluginData folder will be loaded, no matter what.

[rbray89]

Question for the experts.

If I want to keep a part from loading, and the part has a .cfg, a .mu, and a .tga or .png I can compress the .cfg and it'll disappear from the editor, but is KSP still loading the .mu and/or the .tga/.png? My assumption was no, that KSP parses .cfg files and then loads assets referenced by the .cfg, but I'm wondering if that's accurate - do I also need to compress the .mu/.tga/.png?

No, KSP and this mod load ALL textures automatically. Do disable loading, they must be removed from GameData.

With the 4.0 release this went from awesome to wanting it on every stock install - nice job and thank you!!

I will say I'm now struggling to configure this quite the way I'd like, my issues range from just being a texture n00b to confusion on how the config file functions.

I'm going to dive into exploration of each of these points and questions today - but maybe you already have answers or guidance?

1 - If I was setting up a config for a mod/parts pack that you haven't already made a config for, how would I know what textures are normal maps? The naming convention doesn't seem to be consistently followed so there must be another way to tell - open the file and see if it's grayscale?

2 - Given how fast the DXT loads at game start, I want this for functionality for all textures - I'm unclear though if my config settings are accidentally excluding some textures when I only wish to not scale those textures? How do I convert to DXT without rescaling?

3 - I also want to compress all textures, right? So if even if I exclude textures from scaling, am I also excluding them from compression?

4 - I wish there was a way to know if KSP needed to read a texture throughout runtime - maybe the log will give me clues which allow me to see if unloading textures from memory causes issues or not?

5 - I'm expecting to make heavy use of the "min" config setting but I'm unclear (see above) what this will exclude textures from being processed with which features. I'll probably explain this better when I explore it.

6 - I love the runtime execution of this conversion/compression/scaling - idiot proof! But given the leaks and memory issues that exist and evidently experienced by so many, what would it take to also distribute a separate executable which can create the cache? This may even allow for other optimizations, and warnings about total texture size, etc.

Anyway, that's what I'm curious about and going to look at today. Thanks again for the nice enhancement to a great game!

-Talon

1) There are two ways, Most normal maps use NRM at the end of the file. These are handled automatically. After loading, you can pop open the logs and search for something like "Texture is not a normal map and texture is not readable" Those have to be added to the normal map list.

2) All you have to do is add a config for each mod you have, or one giant config with a bunch of folders listed. Some mods will be leaving textures alone on purpose as they aren't rendered, but are used as lookup maps.

3) Nope. Compression and rescalling are mutually exclusive.

4) I tried doing this, but unfortunately, unity sealed the class, so there was no way for me to do that.

5) Min is just the lower bounds of resizing. Basically it was introduced to make sure icon textures didn't get shrunk to oblivion.

6) See below.

First off, good work so far with this!

I would highly recommend creating a simple program or .bat file to run this outside of the ksp game-loader itself, to get around the memory limit of the actual game, but other than that it's worked wonders.

I chose to avoid this so that everyone on every platform would be able to run it, and to reduce complexity as much as possible.

[Black-Talon]

You replied so quickly! Thank you! Which means my edit was missed...

EDIT - I should have shared my goal -

I currently have a thought that the best way to run KSP would be...

• Compresses all textures to DXT format
  
• Generates mipmaps for all textures
  
• ONLY IF texture is above YxY dimensions OR above X KB in size: Resize texture to 1/2 it's current size
  

Reasoning: Some textures are giant for good reason, don't scale them down to 512x512, just cut them in half. Small textures are already small enough, beyond compression/optimization I don't really need to scale them down further, especially since the quality impact will be higher when cutting a 512x512 down to 256x256 than a 2048x4096 to 1024x2048.

Clearly Active Texture Management can do this for me ... but my best guess at a config setup is below and I'm not certain it's working in the way I desired:

ACTIVE_TEXTURE_MANAGER
{
	filter_mode = Trilinear
	make_not_readable = true

	compress = true
	mipmaps = true
	scale = 2
	max_size = 0
	min_size = 512

	compress_normals = true
	mipmaps_normals = false
	scale_normals = 2
	max_size_normals = 512
	min_size_normals = 128
}

PS - Oh hey, these things are also questions (minor bugs?) on my mind:

1. Comparing Basic to Aggressive configs, DBG is set to TRUE in Aggressive but commented out in Basic?
   
2. Aggressive is set to ignore Agencies textures but Basic isn't?
   
3. I don't think I understand the choice to put all the configs for ActiveTextureManager in the \BoulderCo\ folder instead of the \ActiveTextureManagement\ folder - with the craziness of EVE and EVE Texture packs putting lots of EVE configs and additional textures in \BoulderCo\ it is a crazy place that should probably be wiped clean regularly when mods are updated or re-installed. Why add these configs to \BoulderCo\ when a perfect place away from the fray already exists?

[TheRomantics]

I'm trying to find out if this mod is actually working for me or not. I'm pretty new to Kerbal and just don't have any experience with how long these mods take to load. I'm running a highly modded Kerbal right now and live streaming it on Twitch and any time I have a crash, I have about a 17 minute load time to start the game back up. Here's my mod list. https://docs.google.com/spreadsheets/d/1fQH9Lh-CvVctWAij6PoLz5LhGiy9ZvmuCLY7tk7uwZw/edit?usp=sharing

I had first installed the x64 aggressive version and then realized that I wasn't using the 64bit Kerbal. I was getting about a 17 minute load time with the x64 version installed. So I thought, EUREKA! that must be what it is. I then just merged the x86 aggressive version into the folder thinking it would help. My load time is still 17 minutes. Am I doing something wrong or is this expected with those mods even using this mod to compress? Did I mess it up by just installing the x86 version on top of the x64 version when I should have been deleting something first?

Thanks for your time.

[Saint-Stanislas]

I was wondering what files RaspterPropMonitor is using as his police ? I have difficulties setting up an exception for it so the text doesn't get all giberrish when I use scaling, the .cfg in the package doesn't seem to do it, and in this config files, if I set enabled to false, would it prevent atm of running a particular cfg file or would it prevent atm from treating the specific folder ?

OH and, what is DGB = true for?

Great Mod !!!!!

@Romantic: It's normal! It can easily take an hour, dont' rush thinking it's bugged , just go do something else. It'S only for the first load though as the mods reprocess your textures. If it's actually crashing (quitting on it'S own) just restart it as many time as necessary as the conversion will restart where it left. If it's unresponsive, and window suggest you to close an unresponsive task, it's actually progressing, leave him be.

Edited December 14, 2014 by Saint-Stanislas

[Black-Talon]

OH and, what is DGB = true for?

It turns out the debug output in the output_log ... I'm afraid I'm no help on the rest of your questions though, I'm similarly experimenting to understand a couple things about config exceptions/overrides for textures/folders.

[LucidPixels]

Having put aside this mod in the past in favor of just pruning unused parts and forcing OpenGL, I must say the latest version is fantastic. Though the initial conversion took several hours, loading times are now an order of magnitude faster.

[johnsonwax]

johnsonwax your assumption is incorrect. KSP loads all recognized file types, and only later compiles them. So any image under GameData and not under a PluginData folder will be loaded, no matter what.

Aha! That's why we should ask these questions. Would explain why I'm not getting the kind of memory savings I was expecting. Thanks to all for the clarification. Time to find a new approach here.

[Kerbas_ad_astra]

I've had an issue with the interior normal textures from the Taurus HCV pack -- they were correctly identified as normals, but weren't compressed ("compressed = false" in their tcache files), resulting in the cache becoming bigger than the originals (because of the mipmaps, I assume), even though I've got Aggressive and it should be compressing everything by default (ATM seems to be generally a little flaky about obeying default parameters, especially for normals -- do I need to delete the cache after changing the master config?). I resolved it by creating a new config:

ACTIVE_TEXTURE_MANAGER_CONFIG
{
    folder = R&SCapsuledyne
    enabled  = true
    OVERRIDES
    {
        R&SCapsuledyne/Spaces/.*
        {
            compress = true
            mipmaps = false
            scale = 2
            max_size = 0
            make_not_readable = true
        }
    }
}

Edited December 15, 2014 by Kerbas_ad_astra

[rbray89]

I've had an issue with the interior normal textures from the Taurus HCV pack -- they were correctly identified as normals, but weren't compressed ("compressed = false" in their tcache files), resulting in the cache becoming bigger than the originals (because of the mipmaps, I assume), even though I've got Aggressive and it should be compressing everything by default (ATM seems to be generally a little flaky about obeying default parameters, especially for normals -- do I need to delete the cache after changing the master config?). I resolved it by creating a new config:

ACTIVE_TEXTURE_MANAGER_CONFIG
{
    folder = R&SCapsuledyne
    enabled  = true
    OVERRIDES
    {
        R&SCapsuledyne/Spaces/.*
        {
            compress = true
            mipmaps = false
            scale = 2
            max_size = 0
            make_not_readable = true
        }
    }
}

If you want to specify params, you also need to specify NRM params if you don't want the defaults to be used. An easy fix is to delete the OVERIDES entry. This is only needed when specifying settings that are different from the global config.

[rbray89]

You replied so quickly! Thank you! Which means my edit was missed...

EDIT - I should have shared my goal -

I currently have a thought that the best way to run KSP would be...

• Compresses all textures to DXT format
  
• Generates mipmaps for all textures
  
• ONLY IF texture is above YxY dimensions OR above X KB in size: Resize texture to 1/2 it's current size
  

Reasoning: Some textures are giant for good reason, don't scale them down to 512x512, just cut them in half. Small textures are already small enough, beyond compression/optimization I don't really need to scale them down further, especially since the quality impact will be higher when cutting a 512x512 down to 256x256 than a 2048x4096 to 1024x2048.

Clearly Active Texture Management can do this for me ... but my best guess at a config setup is below and I'm not certain it's working in the way I desired:

ACTIVE_TEXTURE_MANAGER
{
	filter_mode = Trilinear
	make_not_readable = true

	compress = true
	mipmaps = true
	scale = 2
	max_size = 0
	min_size = 512

	compress_normals = true
	mipmaps_normals = false
	scale_normals = 2
	max_size_normals = 512
	min_size_normals = 128
}

PS - Oh hey, these things are also questions (minor bugs?) on my mind:

1. Comparing Basic to Aggressive configs, DBG is set to TRUE in Aggressive but commented out in Basic?
   
2. Aggressive is set to ignore Agencies textures but Basic isn't?
   
3. I don't think I understand the choice to put all the configs for ActiveTextureManager in the \BoulderCo\ folder instead of the \ActiveTextureManagement\ folder - with the craziness of EVE and EVE Texture packs putting lots of EVE configs and additional textures in \BoulderCo\ it is a crazy place that should probably be wiped clean regularly when mods are updated or re-installed. Why add these configs to \BoulderCo\ when a perfect place away from the fray already exists?

1& 2 are oversights on my part... 3 was me thinking I'd be clever, but thinking that I'm now regretting it :/

[Aivoh]

Thanks to Neutrinovores post #2579 (and BD Armory) I was able to use the BD Armory ATM config as an example to make individual configs for my other icon textures in their mod folders to keep them sharp.

Such as: 000_Toolbar (Blizzy's Toolbar?), ConnectedLivingSpace, KerbalEngineer, MagicSmokeIndustries (InfernalRobotics), RCSBuildAid, Quick Exit, VOID

Of course I had to delete the previously cached icon folders from ATM.

Edit: This is with Aggressive and 1/4 Resolution in the game settings.

Edited December 15, 2014 by Aivoh

[Kerbas_ad_astra]

If you want to specify params, you also need to specify NRM params if you don't want the defaults to be used. An easy fix is to delete the OVERIDES entry. This is only needed when specifying settings that are different from the global config.

But ATM wasn't applying the default parameters from ActiveTextureManagement.cfg, which includes compressing normals and rescaling them by 2.

I actually just noticed that those parameters in my R&SCapsuledyne.cfg don't apply to normals at all, but adding that file still resolved the issue that ATM wasn't compressing the normal texture -- was it just forcing ATM to look more closely at the folder and jostling it into action? Where and why would ATM be making judgement calls to not compress certain textures or normals? I'm seeing other normal textures in e.g. Nertea's Mark IV System also going uncompressed. Does ATM leave a log of what it's doing to the textures? If so, I'll clear those folders from the cache, recompress them, and see what ATM has to say for itself.

[rbray89]

But ATM wasn't applying the default parameters from ActiveTextureManagement.cfg, which includes compressing normals and rescaling them by 2.

I actually just noticed that those parameters in my R&SCapsuledyne.cfg don't apply to normals at all, but adding that file still resolved the issue that ATM wasn't compressing the normal texture -- was it just forcing ATM to look more closely at the folder and jostling it into action? Where and why would ATM be making judgement calls to not compress certain textures or normals? I'm seeing other normal textures in e.g. Nertea's Mark IV System also going uncompressed. Does ATM leave a log of what it's doing to the textures? If so, I'll clear those folders from the cache, recompress them, and see what ATM has to say for itself.

The override section should only be used if you need to go against the defaults, and the normalmap settings have to be specified as well if there are normalmaps in the location pointed to.

Look at some of the default configs. Most are VERY short, pointing to a folder, saying the config is ready, and that is it.

If there are any normal maps not being compressed, it means you don't have a config for them, or the mod author improperly setup the config.

[willow]

That last update was epic. You've cut my initial startup time down from an hour into something like 10 minutes... AND it no longer crashes on a few of those pesky textures. Thank you!

[Kerbas_ad_astra]

Alright, I think I understand: before it compresses normals, ATM needs a config file saying that it's OK. Thanks for the heads-up -- it seems I've got a lot of memory savings to be made by defining a few more configs. I can put them here, or figure out how to put them on GitHub and make a pull request to you. Can I define multiple "folder = " in one config the way that the normal list can include multiple "texture = "? It would be handy for mods like Near Future Technologies that come in multiple folders.

[rbray89]

Alright, I think I understand: before it compresses normals, ATM needs a config file saying that it's OK. Thanks for the heads-up -- it seems I've got a lot of memory savings to be made by defining a few more configs. I can put them here, or figure out how to put them on GitHub and make a pull request to you. Can I define multiple "folder = " in one config the way that the normal list can include multiple "texture = "? It would be handy for mods like Near Future Technologies that come in multiple folders.

most certainly. That is fine.

[hvacengi]

I feel like I must be doing something wrong. I've tried installing the basic option, and then the aggressive option (after deleting all components of basic of course) but I don't seem to see any appreciable reduction in RAM usage or loading time. And if I check the log file, if find this:

[LOG 13:08:45.915] ActiveTextureManagement: Memory Saved : 0B
[LOG 13:08:45.916] ActiveTextureManagement: Memory Saved : 0kB
[LOG 13:08:45.917] ActiveTextureManagement: Memory Saved : 0MB

As near as I can tell, all that I should have to do is copy the ActiveTextureManagement and BoulderCo folders into my GameData folder, but I feel like I'm missing something. Do I need to be doing something else to enable the mod to work? I have a copy of my output_log.txt that I can PM a drop box link as needed.

[rbray89]

I feel like I must be doing something wrong. I've tried installing the basic option, and then the aggressive option (after deleting all components of basic of course) but I don't seem to see any appreciable reduction in RAM usage or loading time. And if I check the log file, if find this:

[LOG 13:08:45.915] ActiveTextureManagement: Memory Saved : 0B
[LOG 13:08:45.916] ActiveTextureManagement: Memory Saved : 0kB
[LOG 13:08:45.917] ActiveTextureManagement: Memory Saved : 0MB

As near as I can tell, all that I should have to do is copy the ActiveTextureManagement and BoulderCo folders into my GameData folder, but I feel like I'm missing something. Do I need to be doing something else to enable the mod to work? I have a copy of my output_log.txt that I can PM a drop box link as needed.

Ignore the logs. The Memory count is not accurate at all.

[Robotengineer]

Any word on .90 compatibility?

[lincourtl]

Any word on .90 compatibility?

I was just going to try it, because as of 0.90 stock KSP will no longer run in 2GB.

On edit: Nope. ATM keeps running out of memory trying to do the conversions. The cache directory never even gets created. KSP.log here: https://www.dropbox.com/s/ti9xqtt8i2majls/KSP.90.ATM.Aggressive.log?dl=0

[LOG 20:10:38.102] Load(Audio): Squad/Sounds/sound_tab_retreat
[LOG 20:10:38.127] Load(Texture): NASAmission/Flags/NASA
[LOG 20:10:38.172] ActiveTextureManagement: Loading texture...
[EXC 20:10:38.200] OutOfMemoryException: Out of memory
    ActiveTextureManagement.TextureConverter.InitImageBuffer ()
    ActiveTextureManagement.TextureConverter.IMGToTexture (ActiveTextureManagement.TexInfo Texture, Boolean mipmaps, Boolean isNormalFormat)
    ActiveTextureManagement.TextureConverter.GetReadable (ActiveTextureManagement.TexInfo Texture, Boolean mipmaps)
    ActiveTextureManagement.CacheController.RebuildCache (ActiveTextureManagement.TexInfo Texture, Boolean compress, Boolean mipmaps)
    ActiveTextureManagement.CacheController.FetchCacheTexture (ActiveTextureManagement.TexInfo Texture, Boolean compress, Boolean mipmaps)
    ActiveTextureManagement.DatabaseLoaderTexture_ATM.UpdateTexture (ActiveTextureManagement.TexInfo texture)
    ActiveTextureManagement.DatabaseLoaderTexture_ATM+<Load>d__4.MoveNext ()
    UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
    :MoveNext()
[LOG 20:10:38.213] Load(Texture): NASAmission/Parts/GrapplingDevice/grabberDiffuse
[LOG 20:10:38.232] ActiveTextureManagement: Loading texture...
[EXC 20:10:38.236] OutOfMemoryException: Out of memory
    ActiveTextureManagement.TextureConverter.InitImageBuffer ()
    ActiveTextureManagement.TextureConverter.IMGToTexture (ActiveTextureManagement.TexInfo Texture, Boolean mipmaps, Boolean isNormalFormat)
    ActiveTextureManagement.TextureConverter.GetReadable (ActiveTextureManagement.TexInfo Texture, Boolean mipmaps)
    ActiveTextureManagement.CacheController.RebuildCache (ActiveTextureManagement.TexInfo Texture, Boolean compress, Boolean mipmaps)
    ActiveTextureManagement.CacheController.FetchCacheTexture (ActiveTextureManagement.TexInfo Texture, Boolean compress, Boolean mipmaps)
    ActiveTextureManagement.DatabaseLoaderTexture_ATM.UpdateTexture (ActiveTextureManagement.TexInfo texture)
    ActiveTextureManagement.DatabaseLoaderTexture_ATM+<Load>d__4.MoveNext ()
    UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
    :MoveNext()
[LOG 20:10:38.249] Load(Texture): NASAmission/Parts/GrapplingDevice/window
[LOG 20:10:38.254] ActiveTextureManagement: Loading texture...
[EXC 20:10:38.272] OutOfMemoryException: Out of memory
    ActiveTextureManagement.TextureConverter.InitImageBuffer ()
    ActiveTextureManagement.TextureConverter.IMGToTexture (ActiveTextureManagement.TexInfo Texture, Boolean mipmaps, Boolean isNormalFormat)
    ActiveTextureManagement.TextureConverter.GetReadable (ActiveTextureManagement.TexInfo Texture, Boolean mipmaps)
    ActiveTextureManagement.CacheController.RebuildCache (ActiveTextureManagement.TexInfo Texture, Boolean compress, Boolean mipmaps)
    ActiveTextureManagement.CacheController.FetchCacheTexture (ActiveTextureManagement.TexInfo Texture, Boolean compress, Boolean mipmaps)
    ActiveTextureManagement.DatabaseLoaderTexture_ATM.UpdateTexture (ActiveTextureManagement.TexInfo texture)
    ActiveTextureManagement.DatabaseLoaderTexture_ATM+<Load>d__4.MoveNext ()
    UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
    :MoveNext()

Edited December 16, 2014 by lincourtl

[biohazard15]

Any word on .90 compatibility?

It does not work for me.

Actually, it increases my memory consuption - from 1.66 to almost 1.8 Gb.

It might be due to OpenGL, though (but in 0.25, the two worked together without any problem)