rbray89

[1.0][Release-5-0][April 28, 2015] Active Texture Management - Save RAM!

Recommended Posts

I had a quick look at the code and had an idea on how to to multi-threading with minimal code change.

Share this post


Link to post
Share on other sites

Thanks for this mod. It did a good job on cutting down my initial memory usage but I'm having a problem with a memory leak though (goes from 2.2 gb used to 3.7 gb before crashing) after simply leaving it on the VAB screen for an hour or so. I tried a new install (downloaded all new files from steam) and tried each mod I use individually and none alone seem to cause the issue. This was very time-consuming due to having to wait after each mod was installed to see if it caused the issue. So now I'm in the situation of trying to determine which combination of two or more mods is causing my memory leak.

Does anyone happen to have suggestions on how I might best go about this as the permutations are too numerous for an ad-hoc approach to be helpful?

Share this post


Link to post
Share on other sites
I'm having the same issue but its with EVE's cloud.cloud.update method as far as I can tell so far. Not sure where to post the logs.. >.<

Very frustrating because even if you change the config "Compress flag" to NOT compress files without a config...it still compresses BoulderCo

Share this post


Link to post
Share on other sites

What if I've already converted my textures? Painfully loading the mods, and adding exceptions to the dds.py script until all the icons worked, and my game loads in less than 15 minutes (it's actually under 2 minutes, but it used to seem to be around 15)... Did I mention I'm running 64-bit Linux?

Would I be better off reinstalling all my mods and letting ATM compress them, or would I run into the same issue of having textures compressed that shouldn't be? Would you like my exception list, so you can merge it with yours (or more appropriately, create new .tcfg files for you to add)? Which reminds me... Kerbal Alarm Clock got mangled because the CKAN entry is installing it in the wrong place... Need to pop over and correct that...

Share this post


Link to post
Share on other sites
What if I've already converted my textures? Painfully loading the mods, and adding exceptions to the dds.py script until all the icons worked, and my game loads in less than 15 minutes (it's actually under 2 minutes, but it used to seem to be around 15)... Did I mention I'm running 64-bit Linux?

Would I be better off reinstalling all my mods and letting ATM compress them, or would I run into the same issue of having textures compressed that shouldn't be? Would you like my exception list, so you can merge it with yours (or more appropriately, create new .tcfg files for you to add)? Which reminds me... Kerbal Alarm Clock got mangled because the CKAN entry is installing it in the wrong place... Need to pop over and correct that...

The big benefit is that this mod will ensure all your textures get managed automatically without having to micro-manage your mods' textures. Really, it is up to you, but keep in mind that this mod also gives you the ability to set scaling however you want as well.

Share this post


Link to post
Share on other sites
The big benefit is that this mod will ensure all your textures get managed automatically without having to micro-manage your mods' textures. Really, it is up to you, but keep in mind that this mod also gives you the ability to set scaling however you want as well.

OK. Then I think I will "downgrade" my installs to the original png/mbm/tga files, and let ATM manage the compression/caching. I'll just have to review the exceptions I set up so I can make sure I don't mangle my toolbars and plugin GUIs...

Share this post


Link to post
Share on other sites
Also, this won't require a cache update. So no painful re-loading :)

That's great! Should I still re-chache (I am doing that as we speak) the "Clouds" folder for it to work? I tried without doing that and still no clouds so I am hoping the answer is yes :blush:

I really appreciate all your hard work! I am sure I am not alone.

Share this post


Link to post
Share on other sites
Will it be OK with the use of Unity's functionality? (Mathf, Vector, Color32, etc...)

If in doubt just wrap them up in a critical section, or whatever C# calls the exclusive access mechanism. It's best to keep such exclusion sections as short and fast as possible, for obvious reasons, it can also be tricky to debug any problems caused by shared resources that should be in a CS but for some reason are not. Still, iirc from my tests the gains were near linear (2 threads == half the time etc) so well worth investigating?

Share this post


Link to post
Share on other sites
That's great! Should I still re-chache (I am doing that as we speak) the "Clouds" folder for it to work? I tried without doing that and still no clouds so I am hoping the answer is yes :blush:

I really appreciate all your hard work! I am sure I am not alone.

:/ Hmmm... It shouldn't. I don't understand where these issues are stemming from. I'm using the current release of ATM and EVE that are on Github... You might try deleting the BoulderCo folder inside the texture cache folder.

Share this post


Link to post
Share on other sites
If in doubt just wrap them up in a critical section, or whatever C# calls the exclusive access mechanism. It's best to keep such exclusion sections as short and fast as possible, for obvious reasons, it can also be tricky to debug any problems caused by shared resources that should be in a CS but for some reason are not. Still, iirc from my tests the gains were near linear (2 threads == half the time etc) so well worth investigating?

That's the thing though, I use the Unity operations pretty much everywhere. And we all know unity wasn't designed around multithreaded operations.

Share this post


Link to post
Share on other sites
That's the thing though, I use the Unity operations pretty much everywhere. And we all know unity wasn't designed around multithreaded operations.

From a quick look around it appears the Math functions are thread safe eg

answers.unity3d.com/questions/754873/c-how-to-make-a-job-queue-for-noise-function.html

I'm unsure of other functions but it would appear the main threaded concern comes about when two or more threads try to access the same object. tbh, most functions written in a stack based high level language are thread safe, it's only when accessing single shared resources that problems occur.

Of course ymmv.

Edit:

Color operations could be safe too

http://answers.unity3d.com/questions/486584/how-to-create-a-parallel-for-in-unity.html

Some more info, mentions vectors and some custom helper code

http://forum.unity3d.com/threads/multithreading-solutions.90676/

Edited by ecat

Share this post


Link to post
Share on other sites
From a quick look around it appears the Math functions are thread safe eg

answers.unity3d.com/questions/754873/c-how-to-make-a-job-queue-for-noise-function.html

I'm unsure of other functions but it would appear the main threaded concern comes about when two or more threads try to access the same object. tbh, most functions written in a stack based high level language are thread safe, it's only when accessing single shared resources that problems occur.

Of course ymmv.

Haha, that is what I worry about :) Thinking about it, the libsquish implementation would be more likely able to be multi-threaded, but the Nvidia one uses static objects. Probably why the NVidia implementation seems faster as it is(less dynamic allocation & garbage collection). I'll give it a shot, and see where we get.

Share this post


Link to post
Share on other sites
:/ Hmmm... It shouldn't. I don't understand where these issues are stemming from. I'm using the current release of ATM and EVE that are on Github... You might try deleting the BoulderCo folder inside the texture cache folder.

Re caching the Clouds folder did the trick! It works!! Thank you!

Share this post


Link to post
Share on other sites
You are wrong. DLL used by KSP plugins work on OSX and Linux without any change. They are .Net compiled code and those are platform agnostic.

Very interesting, thanks for the correction.

Share this post


Link to post
Share on other sites

The quick and dirty method of experimenting with these things:

1) Create a wrapper function for each and every Unity function you wish to use

2) inside each wrapper create a critical section around the the Unity function call

3) Write the threaded code, it should work fine since everything is protected.

4) Start to selectively remove the CS protection.

5) profit :) Or lose a great deal of hair. Multi-threadding is great fun but can be tricky :)

I may have some time at the weekend if you hit any problems.

Share this post


Link to post
Share on other sites

2 hours... still going D:

I should have used the nvidia tool...

Share this post


Link to post
Share on other sites
2 hours... still going D:

I should have used the nvidia tool...

Mine took about 8 hours all said and done...but I've got a lot going on in the game data folder.

Share this post


Link to post
Share on other sites
Haha, that is what I worry about :) Thinking about it, the libsquish implementation would be more likely able to be multi-threaded, but the Nvidia one uses static objects. Probably why the NVidia implementation seems faster as it is(less dynamic allocation & garbage collection). I'll give it a shot, and see where we get.

I saw that the Nvidia one is full of static while trying :)

I ll have a go at libsquish on my side too.

Share this post


Link to post
Share on other sites

speaking of \BoulderCo\Clouds, caching the latest version of Astronomer's Visual Pack with its 10 or so 8K textures is truly a herculean effort unto itself. i can practically hear the wailing and gnashing of teeth coming from within my GameData folder, but the libsquish algorithm seems to be slowly and painfully getting along with it.

that aside, i'd just like to extend my thanks to you for developing ATM all this time. i feel like you essentially saved the majority of the KSP community twice in a row now; first when the proliferation of good mods resulted in everyone running into severe memory limitations, and then when those people installed so many mods that loading the game was getting to be a ridiculously drawn-out process (especially when troubleshooting; i will not miss the days when i would make a 30-second emendation to GameData, load the game for 6-10 minutes, see that the same problem was still there, and then repeating the same process for a whole evening).

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.