[0.90] DDSLoader 1.9 (Mar 9) - Loads DDS Texture "Boringly fast" edition

[A_Burnt_Rodent]

@blowfish Okay thanks for the info, I'll keep my eyes on this and I just saw lilleman has released a conversion tool . Maybe the process will be simplified enough for a luddite like me to cope with.

[Rybec]

I have ~70 mods installed. I used lilleman's converter tool with it's default settings. My load time used to be 15-20 minutes, it's now down to around 5, almost all of which is modulemanager doing it's thing.

Not only does the game's initial load happen faster, EVERYTHING is faster. I have significantly better framerates, scene transitions are near-instantaneous, everything feels more responsive, and because everything has mipmaps built into it the game even LOOKS better. Ram usage is lower, but not significantly so (but then again I'm using texture replacer and it's texture-unloading goodness so this is to be expected).

In short, sarbian for president. This should be a default unity feature.

[Artophwar]

I just want to say that this helped me save a lot more RAM than ATM and it allowed me to run things at higher resolution. ATM would make things a lot more blurry and still take another 400+ MB of RAM. Thank you so much.

[rottielover]

To paraphrase the immortal R. Lee Ermey " Sarbian, you can come over to my house and meet my sister".

I've had the exact same experience, moar mods, less trouble. Seriously hope SQUAD takes notice! If Unity doesn't incorporate this, I think they should! Between your loader and lilleman's tool, the process now couldn't be simpler. Step 1: cut a ho... wait wrong procedure....

Step 1: Run lilleman's tool and pick your gamedata folder, sit back and let the magic happen. (shhh just let it happen)

Step 2: Install your mod an load the game. Soak in the goodness.

Used to be with ATM only, I was red-lining the memory usage and able to do maybe 1 to 2 missions (launch, do something, return) before the game would run out of mem and crash. Now I've done at least 10, even left the game running all night in 2x to see if it would stay running doing a SCANsat scan, got up this morning and yep, still running.

In short, you're the god of the kerbals now, and you should be immortalized with a statue at the KSC (SQUAD!)

[rbray89]

Sarbian/lilleman:

I have done most of the work for ATM to save/load the cache as dxt, but it looks like the image rotation is off (uv mismatch on square textures, graphical issues with non-square textures). Have you two run into this in either the converter or the loader? I am using the libsquish implementation that I ported to c#. I tried porting the Nvidia texturetools source first, but then once I had all the porting work taken care of, I couldn't figure out how to use the dang thing (it is in an interim commit if you are interested: https://github.com/rbray89/ActiveTextureManagement/commit/08989f0a7892bba6dd27697fdfb8b88f9ef17dcc) I still have a few other misc. Bugs to work out, but otherwise it is starting to look good!

Just thought I'd check in to see if it has all been figured out before heading out on my own.

[sarbian]

You have to save the image upside down for it to be loaded correctly in game, but that's the only thing I can think of.

[rottielover]

I've encountered my first hiccup... Every now and then, maybe every 10th or 12th load of the game, it will load most of the way through and then "hang" the loading screen on a squad texture (seemingly random so far). KSP-AVC is still clickable, the loader just hangs.

Closing out ksp.exe and starting it again usually always works fine for another 9-11 loads of the game. Any thoughts?

[rbray89]

You have to save the image upside down for it to be loaded correctly in game, but that's the only thing I can think of.

Interesting... I fixed the issue I was encountering, and I ended up not having to flip anything...

[sarbian]

Any thoughts?

Any logs ?

rbay : strange. Maybe it's because you save and load them yourself. Did you try to load the dds in an image viewer ?

[rbray89]

Any logs ?

rbay : strange. Maybe it's because you save and load them yourself. Did you try to load the dds in an image viewer ?

No, I never bothered with populating the bitfields. I'm just writing the raw data portion, and storing the relevant information in the cache config files.

[sarbian]

Makes sense. I can't wait for the release

[rottielover]

Any logs ?

No sorry, it happens so infrequently that I didn't capture them. Usually I just kill the exe and start it again and it's fine. Next time it happens I will grab them and see if there is anything in the logs.

[Saint-Stanislas]

I tried the new version of ATM and it's great, but i am now unsure if DDS loader is helping or not

http://forum.kerbalspaceprogram.com/threads/59005-0-25-Release-3-8-Oct-16-2014-Active-Texture-Management-Save-RAM%21?p=1585291&viewfull=1#post1585291

In fact, I'm unsure how all this work, but in the end, if I understand correctly:

- ATM is a scaling, caching utility, that save RAM with compression/scaling

- DDS loader is saving RAM by throwing stuff directly at the GPU, stuff that would have been thrown at the GPU anyways but with higher ram usage because of the live file format conversion process?

I'm trying to optimize my configuration and now that I saved all that RAM, I'm wondering if there is some stuff I could bring back in the RAM to save on the GPU or CPU. So again, I don't have any clue of how all this work, but is DDS loader pre-loading some stuff in the GPU that would normally stays in the RAM until needed ( I guess you use CPU over this then ) by the game or ... I don't know. I guess it depends on if DDS is saving ram by avoiding decompression (without it you have the original file and the converted-for-GPU loaded in the RAM at once ? or the original in the RAM and the converted end ups directly in the gpu ? and with ddsloader you just have one dds file loaded in the ram and/or in the gpu ?) or if it's saving ram by loading the GPU with stuff it wouldn't normally load at that precise moment ?

My main concern is to lighten the burden on the graphic card even though I don't understand much about how the computer is handling these information. Thank you in advance to lighten up my comprehension off all this, and to correct my misconceptions

[sarbian]

Err, It's quite an open ended question(s).

First DDS are in a format native to the GPU. So the main benefits is that we can push it to the GPU without doing much more than reading it from the disk. It makes loading faster and that's why I wrote DDSLoader.

Most of the memory saving comes from the fact that the converter use a compressed DDS format. So the texture are loaded and stay compressed in memory. With KSP laoding the texture is loaded, uncompressed and then converted into a GPU usable format (and sometimes compressed too)

Then there is the fact that DDSLoader is a bit more agressive with the ReadOnly flag that makes a texture unreadable by code and avoid keeping 2 copy (memory and GPU) of the texture.

But I guess the main question is about DDLoader and the new ATM instlled together.

Rbay89 know his stuff and ATM makes the whole process a lot easier. You don't have to convert by hand, it resize if you ask it, it already do a lot of check to avoid converting stuff that should not be and handle the ReadOnly flag. However ATM ignore all texture already converted in DDS.

DDSLoader will stay for plugins that need to load a DDS (like RSS), mod that ship with RSS and those who want to do the DDS convertion themselves.

TL;DR version : use both, let ATM handle the convertion and DDSLoader load the DDS shipped with mods.

[Saint-Stanislas]

Ok thanks

On atm thread i was talking about getting the file list that atm touched after one start to then convert the very few things left behind that could still be converted to dds. We are now talking about really small peanuts like toolbar buttons and stuff i guess .... It makes a lot of sense to just have ddsloader in the game data without converting anything ( I got so focused on the idea that I needed to convert the files myself that I forgot that a bunch were dds from start and that the main point of your plugin was to load these )

[requimrar]

So I've woken up from my deep slumber. After trying weird things like Wine and stuff to try and load lilleman's converter (on OS X), I have failed. I then tried using https://github.com/ducakar/img2dds... Which seems to work, but not without some effort on the end user's part:

Steps for OS X (since windows and linux binaries are provided):

1. Ensure you have both cmake and make available. I recommend using 'homebrew' (http://brew.sh) to install both.

2. You will also need FreeImage -- "brew install FreeImage" will work.

3. You will need libsquish. Checkout the svn tree from https://code.google.com/p/libsquish/source/checkout.

3a. Installing libsquish: LD on OS X does not recognise the -soname flag. Delete this portion from 'Makefile': "-Wl,-soname ..." till the next space.

3b. make && make install

3c. Ensure that you have squish.h under /usr/local/include (by default) and libsquish.so.0.0 in /usr/local/lib (again by default)

4. Compiling img2dds

4a. Clone or download the repo somewhere (github link above)

4b. Run "cmake ." (the period is important)

4c. You cannot compile yet. Open ImageBuilder.cc.

4c (i). Near the top, write "#include <vector>" somewhere with the other #includes.

4c (ii). Do a search for "vector<char> buffer". Replace this with "std::vector<char> buffer"

5. Run "make"

6. You should have an 'img2dds' binary file in the folder

7. Setting up the tree

7a. You need to create a temporary folder somewhere, I used my Desktop.

7b. Inside this temporary folder, copy the 'dds.py' file.

7c. Also inside this temp folder, create a folder named 'img2dds', and inside that create a folder named 'linux64'

7c (aside). For some reason linux64 is the default SYSTEM as found by the python script on OS X.

7d. Run dds.py and point it to your GameData folder.

I cannot tell you what the end result is, because it is still converting my GameData folder as I write this post. But since it got to that point (and at least ran), I'm hoping for good things.

Also to the creator of this tool: Thank you for your work. You might want to put clearer instructions in your README.

1. "using namespace std" isn't a good idea.

2. You use std::vector but never #include <vector>. I don't know what compiler you're using but this is an error.

3. Your indentation is atrocious with a capital A, which makes it Atrocious.

To sarbian:

Thank you for your work on this! Hopefully it will be another revolutionary mainstay mod like ModuleManager and will convince all mod developers to switch to DDS instead.

[shaw]

So I've woken up from my deep slumber. After trying weird things like Wine and stuff to try and load lilleman's converter (on OS X), I have failed. I then tried using https://github.com/ducakar/img2dds... Which seems to work, but not without some effort on the end user's part:

Steps for OS X (since windows and linux binaries are provided):

1. Ensure you have both cmake and make available. I recommend using 'homebrew' (http://brew.sh) to install both.

2. You will also need FreeImage -- "brew install FreeImage" will work.

3. You will need libsquish. Checkout the svn tree from https://code.google.com/p/libsquish/source/checkout.

3a. Installing libsquish: LD on OS X does not recognise the -soname flag. Delete this portion from 'Makefile': "-Wl,-soname ..." till the next space.

3b. make && make install

3c. Ensure that you have squish.h under /usr/local/include (by default) and libsquish.so.0.0 in /usr/local/lib (again by default)

4. Compiling img2dds

4a. Clone or download the repo somewhere (github link above)

4b. Run "cmake ." (the period is important)

4c. You cannot compile yet. Open ImageBuilder.cc.

4c (i). Near the top, write "#include <vector>" somewhere with the other #includes.

4c (ii). Do a search for "vector<char> buffer". Replace this with "std::vector<char> buffer"

5. Run "make"

6. You should have an 'img2dds' binary file in the folder

7. Setting up the tree

7a. You need to create a temporary folder somewhere, I used my Desktop.

7b. Inside this temporary folder, copy the 'dds.py' file.

7c. Also inside this temp folder, create a folder named 'img2dds', and inside that create a folder named 'linux64'

7c (aside). For some reason linux64 is the default SYSTEM as found by the python script on OS X.

7d. Run dds.py and point it to your GameData folder.

I cannot tell you what the end result is, because it is still converting my GameData folder as I write this post. But since it got to that point (and at least ran), I'm hoping for good things.

Also to the creator of this tool: Thank you for your work. You might want to put clearer instructions in your README.

1. "using namespace std" isn't a good idea.

2. You use std::vector but never #include <vector>. I don't know what compiler you're using but this is an error.

3. Your indentation is atrocious with a capital A, which makes it Atrocious.

To sarbian:

Thank you for your work on this! Hopefully it will be another revolutionary mainstay mod like ModuleManager and will convince all mod developers to switch to DDS instead.

Thanks. Could you upload the binaries you built, so I can include them in the download package? I don't posssed a Mac to compile them. Do they work if you put the libraries in the same directory as the binaries or you need to have them in /usr? I don't know how OS X works, but on Linux there's rpath property in ELF files where you can specify direcories where to search for shared libraries. Does OS X have anything similar?

And one more thing: could you check what is the value of "sys.platform" Python variable on OS X? I can't support OS X in the script unless I know that.

1. Why? It's in a .cc, not in a header.

2. I'll fix that. It compiles with both GCC 4.9 and Clang 3.5 on Arch Linux (I think the latter is the same compiler as used on Mac).

3. What's wrong? Two spaces are probably the most common indentitation found in FOSS.

Edited December 8, 2014 by shaw

[shaw]

@requimrar:

I've done some research on OS X rpath. Could you fetch the new commit and compile img2dds the following way:

cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_RPATH=@executable_path .
make install DESTDIR=.
strip usr/local/bin/img2dds

mkdir -p img2dds/osx64
cp usr/local/bin/img2dds img2dds/osx64

then, copy freeimage dynamic library to `img2dds/osx64` and uninstall freeimage. If I got rpath correctly `dds.py` should work fine.

[jstnj]

So are these conversion scripts superfluous now that ATM is now converting to dds?

[requimrar]

@shaw:

Thanks for your efforts! Unfortunately sys.platform on OS X (I just tried with python's REPL) is not 'OSX', but 'darwin'.

So just replace that part appropriately (I don't see a need to pull request this right)?

Also: squish compiles to libsquish.so.0.0 by default instead of libsquish.dylib, no idea why -- had to rename mine to work.

Anyway, here's the osx64 folder to include in your releases: https://dl.dropboxusercontent.com/u/20627716/osx64.zip

EDIT: That won't work: I forgot to uninstall FreeImage from /usr/local/lib before trying, turns out it still expects the dylib there, instead of in the osx64/ folder... not sure what you can do about this.

Turns out the makefile complains "No rule to make target `/usr/local/lib/libfreeimage.dylib', needed by `img2dds'" as it appears to put a hard dependency on that.

Same for lib squish, it expects it in /usr/local/lib as well.

As an aside, it is literally impossible to Control-C your script....

Edited December 9, 2014 by requimrar

[MartGonzo]

Do skyboxes work in DDS format?

I converted mine to DXT5 w. mipmaps, interpolated alpha but they aren't showing up in game.

EDIT - nevermind, just went into the game and realised that not only do the skyboxes not load but none of the games textures load

Edited December 9, 2014 by MartGonzo

[sarbian]

What did you use to convert ? Can you post one of the file somewhere ?

[shaw]

@shaw:

Thanks for your efforts! Unfortunately sys.platform on OS X (I just tried with python's REPL) is not 'OSX', but 'darwin'.

So just replace that part appropriately (I don't see a need to pull request this right)?

Also: squish compiles to libsquish.so.0.0 by default instead of libsquish.dylib, no idea why -- had to rename mine to work.

Anyway, here's the osx64 folder to include in your releases: https://dl.dropboxusercontent.com/u/20627716/osx64.zip

EDIT: That won't work: I forgot to uninstall FreeImage from /usr/local/lib before trying, turns out it still expects the dylib there, instead of in the osx64/ folder... not sure what you can do about this.

Turns out the makefile complains "No rule to make target `/usr/local/lib/libfreeimage.dylib', needed by `img2dds'" as it appears to put a hard dependency on that.

Same for lib squish, it expects it in /usr/local/lib as well.

As an aside, it is literally impossible to Control-C your script....

I mean, freeimage should be uninstalled AFTER building, just to check if the executable loads the one from its directory. And if libsquish is also compiled as dynamic library, you should do the same for it. (BTW why that happens? By default it only builds the static library.)

[MartGonzo]

What did you use to convert ? Can you post one of the file somewhere ?

I tried the DDS4KSP 0.1.7, when that didn't work I tried converting the skybox myself in Paint.net

OS is Win 7 x64

all the part textures were white and the skybox was the default

I didn't keep the files but can do it again, but I'm not sure how to post it, presumably just posting it here as an image file wouldn't be any help?

it's not really a big deal as obviously there's no problem with the mod as so many people are using it and I can still play KSP with just PNG's and TGA's

[sarbian]

Well, if some DDS format don't work I'd like to look into it even if it's not as critical now that ATM 4 do the convert/cache.