[1.0][Release-5-0][April 28, 2015] Active Texture Management - Save RAM!

[rbray89]

The quick and dirty method of experimenting with these things:

1) Create a wrapper function for each and every Unity function you wish to use

2) inside each wrapper create a critical section around the the Unity function call

3) Write the threaded code, it should work fine since everything is protected.

4) Start to selectively remove the CS protection.

5) profit Or lose a great deal of hair. Multi-threadding is great fun but can be tricky

I may have some time at the weekend if you hit any problems.

I suspect we should be fine.

I once wrote a multi-producer, muti-consumer queue (every element had to be processed by all consumers) where everything existed on their own threads... THAT was fun

[rbray89]

Re caching the Clouds folder did the trick! It works!! Thank you!

Awesome! Now we can tell people to do this before installing I think. Tried to design it so it wouldn't need to be this way, but I guess that didn't work out.

I saw that the Nvidia one is full of static while trying

I ll have a go at libsquish on my side too.

Sounds good! Yeah, I think I'll drop the Nvidia one. Between that, and the fact that the Nvidia port isn't *quite* operational yet, I think libsquish is the way to go.

speaking of \BoulderCo\Clouds, caching the latest version of Astronomer's Visual Pack with its 10 or so 8K textures is truly a herculean effort unto itself. i can practically hear the wailing and gnashing of teeth coming from within my GameData folder, but the libsquish algorithm seems to be slowly and painfully getting along with it.

that aside, i'd just like to extend my thanks to you for developing ATM all this time. i feel like you essentially saved the majority of the KSP community twice in a row now; first when the proliferation of good mods resulted in everyone running into severe memory limitations, and then when those people installed so many mods that loading the game was getting to be a ridiculously drawn-out process (especially when troubleshooting; i will not miss the days when i would make a 30-second emendation to GameData, load the game for 6-10 minutes, see that the same problem was still there, and then repeating the same process for a whole evening).

I could see that. Those textures are massive. I think the suggestion of multi threaded compression is a good one. Especially if it works

No worries Though part of me would wish that squad would implement some of this on their end The big reason I choose MIT license is so that others can use my code in commercial products (Squad, if you are reading this *HINT* *HINT*)

[Lechuza]

Awesome! Now we can tell people to do this before installing I think. Tried to design it so it wouldn't need to be this way, but I guess that didn't work out.

Ehmmm, not really. I installed a new mod (active struts) that needed some chaching, and for some reason when I restarted the clouds disappeared. I will try to re-cache the clouds folder again and see if that fixes it again. This must be the weirdest thing, since one thing has nothing to do with the other. I will keep you posted.

Edit: Figured out whats happening, just not how to fix it. It works when I cache the clouds folder and the game enters, but the moment I quit and enter the game again with a pre-cached Clouds folder the clouds are gone again. So the only way right now pretty much is to cache the clouds folder everytime, which is quite time consuming. This thing get weirder and weirder every time.

Edited December 9, 2014 by Lechuza

[rbray89]

Ehmmm, not really. I installed a new mod (active struts) that needed some chaching, and for some reason when I restarted the clouds disappeared. I will try to re-cache the clouds folder again and see if that fixes it again. This must be the weirdest thing, since one thing has nothing to do with the other. I will keep you posted.

Edit: Figured out whats happening, just not how to fix it. It works when I cache the clouds folder and the game enters, but the moment I quit and enter the game again with a pre-cached Clouds folder the clouds are gone again. So the only way right now pretty much is to cache the clouds folder everytime, which is quite time consuming. This thing get weirder and weirder every time.

Ah... hmmm... Then I'm not sure... Are you using aggressive or basic?

[Lechuza]

x64 Basic on Linux

[rodion]

No worries Though part of me would wish that squad would implement some of this on their end The big reason I choose MIT license is so that others can use my code in commercial products (Squad, if you are reading this *HINT* *HINT*)

enthusiastically agree. honestly i don't even know why Squad doesn't offer you a job. a long time ago when Squad polled forum users on what they'd most like to see in future updates i chose greater optimization and extended capabilities over everything else, and several versions later i am still wishing for the same thing. in a sandbox game of KSP's scale the player should feel like space, time and imagination are the limiting factors one has to contend with, not game software.

[sp1989]

I have been trying to install the new version of ATM for a few hours now and I fully aware of how long we have to wait but it gets hung up in the same place. Now I was used to it seeming frozen and then starting. However it seems to get officially frozen on Squad "unpaidresearch" item and it just doesn't move. I play on a Mac and the rainbow wheel just doesn't go away. I waited for almost a half hour and it just stuck there. I reloaded and the loading screen zoomed to the same spot and got stuck. Just wondering if this is a common issue or something I am doing wrong? I will keep trying but I just don't want to waste much more time especially if its an easy fix.

[Robotengineer]

I have been trying to install the new version of ATM for a few hours now and I fully aware of how long we have to wait but it gets hung up in the same place. Now I was used to it seeming frozen and then starting. However it seems to get officially frozen on Squad "unpaidresearch" item and it just doesn't move. I play on a Mac and the rainbow wheel just doesn't go away. I waited for almost a half hour and it just stuck there. I reloaded and the loading screen zoomed to the same spot and got stuck. Just wondering if this is a common issue or something I am doing wrong? I will keep trying but I just don't want to waste much more time especially if its an easy fix. http://i.imgur.com/QoiTefB.png

If you can, run console alongside it and watch the Unity Player.log, as well as Activity Monitor, if that is what it is actually stuck on, try removing it from the cache.

[stardestroyer]

I have been trying to install the new version of ATM for a few hours now and I fully aware of how long we have to wait but it gets hung up in the same place. Now I was used to it seeming frozen and then starting. However it seems to get officially frozen on Squad "unpaidresearch" item and it just doesn't move. I play on a Mac and the rainbow wheel just doesn't go away. I waited for almost a half hour and it just stuck there. I reloaded and the loading screen zoomed to the same spot and got stuck. Just wondering if this is a common issue or something I am doing wrong? I will keep trying but I just don't want to waste much more time especially if its an easy fix. http://i.imgur.com/QoiTefB.png

I had the same issue but it seems to be normal, you are probably using texture replacer (should be the next folder in the cueue) and it has very big files to convert, so it takes a very long time. After that point I'm actually converting a file named KerbinScaledSpace300.png of 25MB. 3 hours so far and I haven't finished so I can't tell if it's the last problem

[Nereid]

Is the first startup of ATM in its latest release really that slow? KSP is loading for over 30 minutes and not even 50% are done...

[madlemur]

Is the first startup of ATM in its latest release really that slow? KSP is loading for over 30 minutes and not even 50% are done...

I do believe that it is. There have been reports of multi-hour delays as ATM converts/compresses the textures to a DXT format, which will make subsequent loads MUCH faster.

[sarbian]

rbray89 : I submitted my patch. I get a nice 98% CPU usage with it. I'll see if I can get some kinds of bench to see how faster it is.

Edit : about 5 time faster on my system (an i7-2600K)

I see that I left a debug line in the commit : UnityEngine.Debug.Log("Pcount" + Environment.ProcessorCount);

Edited December 9, 2014 by sarbian

[Saint-Stanislas]

HEY everyone !!!! The new ATM is taking eons to load the first time, it is all normal.

It also seem to be a bit longer for mac users! It is all normal !

I think that the first load is kind of not the point here, but memory saving on long term is and it's working damn well!!

I just don't know how much reading people do before posting an ''issue'', since this thread is getting loaded with ''frozen screen report'' even though the first page is quite clear about what you should expect at first load. Anyway, people that don't even read the first post are probably neither reading the last ones so... THANK YOU rbray89 !!!

For those who think it is bugged but are able to read go back a few pages, somebody was suggesting to load heavy mods one by one, convert them, and then load another one. This method prevent the memory crash in case were your gamedata was already at the edge of what 32bit could do and the conversion is pumping more.

As an example, let's say your game crashe normally at 2,5Gb,and you're installation ask 2gb, then atm would try to convert all this .... using another block of memory and might break the limit. You can just split your gamedata in two different ones, load the game one time with each, and merge them back together. Atm cache will merge as well and you should then be able to boot without crash.

The only advantage of this is that you don'T have to worry about restarting the process everytime it crashes. As an example, instead of waiting in front of the computer for 2 hours to be sure it is still processing, boot 2 KSP.exe with half the Gamedata in each (but both ATM) and go to bed.

ps: my gamedata is pretty small (1.2gb) and therefore I didn'T even try these solution. But I don'T see any reason they would not work, worst case is that ATM still convert 2-3 files when you merge everything together because you didnt use the same settings or something

[RandulfJ]

Will it be OK with the use of Unity's functionality? (Mathf, Vector, Color32, etc...)

4-1: should fix the issues seen with EVE and other mods.

https://github.com/rbray89/ActiveTextureManagement/releases/tag/4-1

So I re-cached everything with this version.. >.> . Worked good on first load, had clouds. Shut down and restart KSP, no clouds and same errors in output_log. Bunch of NREs from Clouds.Clouds.Update in EVE.

NullReferenceException: Object reference not set to an instance of an object

at Clouds.Clouds.Update () [0x00000] in <filename unknown>:0

[Beetlecat]

A brief note about the new ATM and a weird issue when compressing RetroFuture:

I'm *still* in the process of building my new cache (started around mid-day and have been re-starting the game off and on and making good progress) and noticed it would choke on every third item or so in the RetroFuture pack. So much so, that I took out everything not needed by the process from the gamedata folder just leaving RetroFuture (and things after it) and it would still crash the game after every few components. After a dozen+ restarts, it eventually did finish, and movd on to the Squad folder, and it's been processing that non-stop for while.

Just figured I would throw this out in case others are noticing that same thing, or if there's something that nli2work should be aware of.

Fingers crossed!

[Chezburgar7300]

I assume there are several others using Astronomer's visual pack with the 4.1 update and running into a load issue? It gets stuck on detaileve1.png which is a 14.1kb file and then the game crashes. It's loaded up most of my heavy-load mods (KW, B9 etc) but always gets stuck on this. I have read through this thread that people have had issues similar and kept restarting and it gets there eventually, but I've tried many, many times and KSP just crashes every time. Thanks for the RAM savings though, and from what it loads before it crashes it is a lot faster than 3-8!

[Saint-Stanislas]

Maybe you can try making an exceptions in atm configs for that particular file ?

ACTIVE_TEXTURE_MANAGER_CONFIG

{

folder = Astrosomething

enabled = true

OVERRIDES

{

Astrosomething/pathtothefile/filename

{

compress = false

mipmaps = false

scale = 1

max_size = 0

}

}

[rodion]

update:

unfortunately i'm also getting the no clouds in EVE/AVP bug too right now. however, to ease the burden of recaching the ludicrous textures, i resized the astronomer's 8K textures to 4K (which is enough for me anyway, personally) and crushed them more aggressively than the originals (at no small cost; my CPU hit 65 C for about an hour, bits of programs' GUIs started breaking and i could faintly smell singed dust wafting from my case, no joke), the result is that my \Clouds\Textures folder is down to 66MB from 208MB (almost two whole B9 installations' worth of difference) which should greatly ease loading.

if anyone else is running astronomer's visual pack and could do with the smaller textures you can get them here (7zip).

now to see if i can normalize the clouds issue. i might try setting up an exclusion for all of \BoulderCo from ATM if i have to but i'll just recache, mess around and see.

[LitaAlto]

if anyone else is running astronomer's visual pack and could do with the smaller textures you can get them here (7zip).

FYI Astronomer is attempting to re-release the latest AVP with reduced textures. He's having problems with Curse accepting his file, but it should be a matter of time before an official reduction is available.

[jstnj]

So I re-cached everything with this version.. >.> . Worked good on first load, had clouds. Shut down and restart KSP, no clouds and same errors in output_log. Bunch of NREs from Clouds.Clouds.Update in EVE.

Same here

[EXC 22:20:10.133] NullReferenceException: Object reference not set to an instance of an object	OverlaySystem.TextureSet.initTexture ()
	OverlaySystem.TextureSet..ctor (.ConfigNode textureNode, Boolean bump)
	Clouds.Clouds.LoadConfigNode (.ConfigNode node, System.String url, Boolean useVolume, Boolean defaults)
	Clouds.Clouds.loadCloudLayers (Boolean defaults)
	Clouds.Clouds.Awake ()
	UnityEngine.GameObject:AddComponent(Type)
	AddonLoader:StartAddon(LoadedAssembly, Type, KSPAddon, Startup)
	AddonLoader:StartAddons(Startup)
	AddonLoader:OnLevelWasLoaded(Int32)

[jstnj]

update:

unfortunately i'm also getting the no clouds in EVE/AVP bug too right now. however, to ease the burden of recaching the ludicrous textures, i resized the astronomer's 8K textures to 4K (which is enough for me anyway, personally) and crushed them more aggressively than the originals (at no small cost; my CPU hit 65 C for about an hour, bits of programs' GUIs started breaking and i could faintly smell singed dust wafting from my case, no joke), the result is that my \Clouds\Textures folder is down to 66MB from 208MB (almost two whole B9 installations' worth of difference) which should greatly ease loading.

if anyone else is running astronomer's visual pack and could do with the smaller textures you can get them here (7zip).

now to see if i can normalize the clouds issue. i might try setting up an exclusion for all of \BoulderCo from ATM if i have to but i'll just recache, mess around and see.

So you got clouds to appear by downsizing them?

[rbray89]

Same here

[EXC 22:20:10.133] NullReferenceException: Object reference not set to an instance of an object	OverlaySystem.TextureSet.initTexture ()
	OverlaySystem.TextureSet..ctor (.ConfigNode textureNode, Boolean bump)
	Clouds.Clouds.LoadConfigNode (.ConfigNode node, System.String url, Boolean useVolume, Boolean defaults)
	Clouds.Clouds.loadCloudLayers (Boolean defaults)
	Clouds.Clouds.Awake ()
	UnityEngine.GameObject:AddComponent(Type)
	AddonLoader:StartAddon(LoadedAssembly, Type, KSPAddon, Startup)
	AddonLoader:StartAddons(Startup)
	AddonLoader:OnLevelWasLoaded(Int32)

basic or advanced? what OS?

[jstnj]

basic or advanced? what OS?

Basic Scale=1

Linux x64

KSP x64

[daawgees]

Gonna have to add my name to the list of folks unable to get clouds with ATM/AVP. Took about 90 minutes to load and the second load was crazy fast. I'm not worried about it though, I'm sure this'll be addressed. You modders are freakin' amazing. That is all.

If I can provide any further information in order to help, besides what I've written below, let me know.

Win8.1x64

KSP x86

Using ATM basic.

[stoddard14]

Gonna have to add my name to the list of folks unable to get clouds with ATM/AVP. Took about 90 minutes to load and the second load was crazy fast. I'm not worried about it though, I'm sure this'll be addressed. You modders are freakin' amazing. That is all.

If I can provide any further information in order to help, besides what I've written below, let me know.

Win8.1x64

KSP x86

Using ATM basic.

I'm also chalking my name up there. Gonna keep an eye out for the eventual fix. You guys do great work! I'm also available to lend information if its needed.

Win7 64bit

KSP x32

ATM Agressive