[0.90] DDSLoader 1.9 (Mar 9) - Loads DDS Texture "Boringly fast" edition

[Beduino]

The script I posted doesn't work with all the textures:

- 8k textures just fail to convert

- Textures with resolutions non multiple of 4 fail to convert.

- Some normal maps are lost in the conversion, a lot of them actually. I didn't noticed the effect back then but you can see an example here. http://imgur.com/a/qkUKs#0

- Some textures become invisible, no idea why. (Only two as far as I know).

So I would recommend to wait for Lilleman converter unless you know how to properly convert the textures or just want to play with wrong normal maps, (not that bad actually, I didn't even noticed until I saw the example picture).

So it seems that loading is around 30 to 40% faster and memory consumption reduced by around 500 MB with KSP (+nearly 50 mods) sitting at around 1900MB.

Some of the big textures failed to convert but they are still being loaded just fine and im not seeing any wrong textures.

Normals seems fine to me, even checked that rockomax tank, seems fine.

I then decided to put in 2 big mods, KW and novapunch and then rerun the script. KSP is then sitting at around 2050MB, loading times pretty much the same.

Not miraculous better, but still pretty good I would say.

PS: using opengl since forever

Edited October 24, 2014 by Beduino
PS

[slumpie]

To avoid some harmless warning messages. KSP won't try (and fail) to convert normal maps without "NRM" suffix from RGB to GGGR during model loading.

It doesn't have to convert any dds normal map from xyz to _y_x if they're already saved that way. That's what I don't understand.

We're talking about dds, not mbm, png, tga or even jpeg, right?

Is your intention to not swizzle the normal maps? That would result in noticeably lower quality.

EDIT: Ok, apparently only to prevent those messages, well then...

Im wondering though why have u used the primitive nvdxt instead of the modern version. https://developer.nvidia.com/gpu-accelerated-texture-compression

Maybe it would help with some of the problems.

Because it is an extremely cut down converter. Nvdxt offers much more than the ddstool, even almost every other library is far ahead in this regard.

And no, it wouldn't solve any problems, because it's not nvdxt that's doing anything wrong.

Edited October 24, 2014 by slumpie

[Lilleman]

I just tried the 1.4 build, (until now I was still experimenting with 1.1) and it seems like some textures needs to not be loaded as "read-only".

For instance, while TextureReplacer's reflections textures can be converted and used with 1.1, since 1.2, I've got this kind of errors spamming all over my logs:

"[TR.TextureReplacer] UnityEngine.UnityException: Texture 'TextureReplacer/EnvMap/PositiveX' is not readable, the texture memory can not be accessed from scripts. You can make the texture readable in the Texture Import Settings.

at (wrapper managed-to-native) UnityEngine.Texture2D:GetPixels (int,int,int,int,int)"

Full logs are here if needed, from an unmodded (well, apart from this mod and TextureReplacer, of course) x86 KSP.

...Or did I missed something again, like setting a flag or such? (It wouldn't be so surprising, as I said I kind of run straight into every possible obstacle)...

Edited October 24, 2014 by Lilleman

[Telanor]

Where does the loader expect the DDS files to be? In the same folder as the png/mbm files we created them from? Do we leave the original files there or remove them?

[Urgante]

Where does the loader expect the DDS files to be? In the same folder as the png/mbm files we created them from? Do we leave the original files there or remove them?

Same folder and name than the originals.

You must delete the originals or at least move them out of the gamedata folder. If the original and the dds archive are in the same folder one of them is randomly selected to be used by KSP, (but both are loaded to memory).

[NathanKell]

Not random, it's whichever has the alphabetically-prior file name AFAIK, but yes, both are loaded.

[Beduino]

I was able to squeeze in these mods under 2700MB with dds.

KW, Kethane, Karbonite, FTT, B9, Interstellar, MKS, NearFuture, LLL, SXT, Novapunch, Extraplanetary, DMagic, StationScience, StationPartsExpansion, KAS, InfernalRobotics, FAR, DeadlyReentry, RealChutes, KerbinSide, Astronomers Pack and couple of other smaller mods.

Pretty impressive, never got so many mods in.

[juanml82]

So, is there a user (player?) friendly instruction guide of how to convert the textures that work fine already?

[Urgante]

So, is there a user (player?) friendly instruction guide of how to convert the textures that work fine already?

Short answer--> No.

Better wait for somebody to make a converter, I hope that will happen soon.

Long answer --> See my post in page 9 for the code and the links for the mbm converter and nvdtx.

BUT that code is far from prefect and It's probably not user friendly enough.

Citing myself:

The script I posted doesn't work with all the textures:

- 8k textures just fail to convert

- Textures with resolutions non multiple of 4 fail to convert.

- Some normal maps are lost in the conversion, a lot of them actually. I didn't noticed the effect back then but you can see an example here. http://imgur.com/a/qkUKs#0

- Some textures become invisible, no idea why. (Only two as far as I know).

So I would recommend to wait for Lilleman converter unless you know how to properly convert the textures or just want to play with wrong normal maps, (not that bad actually, I didn't even noticed until I saw the example picture).

[brian6712]

So, I made some quick tests: I made a copy of my modded install (1.2GB of mods, it's not that much but I don't use ATM), and converted everything to dxt5 with crunch (1376 textures converted, to be precise).

RPM don't seems to like it, so I had to switch back to old textures format for this mod. Also, some textures seems to be flipped (see the Chatterer icon in the screen). Other than that, everything seems to be normal.

As for the most important part, the loading time: I used a chronometer, started it as soon as I hit the "enter" key to launch KSP, and stop it as soon as I hear the first note of the main menu theme:

I went from 3min20 to 1min37... It litterally divide my loading time by 2!

That's very promising! More rep for you, my good sir!

Edit : Whoops, it seems like I spoke too soon, once again... Some mods don't load textures correctly, while others are running just fine... I'll try that again later, one mod at a time.

Do you remember which mods off the top of your head you were able top convert? I really believe that if I could convert only the top 3 of the most memory hungry mods I would be ok.

[brian6712]

Here's a more Linux-y (and Mac-y?) conversion script, using convert from imagemagick and nvcompress from nvidia-texture-tools. It's not entirely foolproof, and my bash-fu is probably terrible, but it works. I don't recommend running this on GameData in a heavily-modded install; it works fine for KWRocketry, Squad, and NASAMission parts, but RoverDude's ART and OKS/MKS parts don't like it at all.

#!/bin/bash

mbm2png=/tmp/mbm2png/mbm2png64
nvcompress=$(which nvcompress)
convert=$(which convert)

image_extensions=("mbm" "png" "tga" "jpg")

normal_pattern="(NRM|nm|Normal|Norm|_n)\."
mbm_pattern="\.mbm$"

normal_args="-bc3n -nomips"
color_args="-bc3"

function isNormal {
    if [ ! $# -eq 1 ]
    then
        echo "isNormal must be called with exactly one argument." 1>&2
        exit 2
    fi

    match=$(echo "$1" | grep -Ei $normal_pattern)

    echo $match
}

function isMBM {
    if [ ! $# -eq 1 ]
    then
        echo "isMBM must be called with exactly one argument." 1>&2
        exit 2
    fi

    match=$(echo "$1" | grep -Ei $mbm_pattern)

    echo $match
}

function convertMBM {
    if [ ! $# -eq 1 ]
    then
        echo "convertMBM must be called with exactly one argument." 1>&2
        exit 2
    fi

    if [ ! -e "$1" ]
    then
        echo "File not found: $1" 1>&2
        exit 1
    fi

    $mbm2png "$1" >/dev/null && rm $1

    echo "${1/mbm/png}"
}

if [ $# -eq 0 ]
then
    paths="."
else
    paths="$*"
fi

for ext in "${image_extensions[@]}"
do
    echo Searching for $ext
    find "$paths" -iname "*.${ext}" | while read file
    do
        [ ! -z "$(isMBM "$file")" ] && file=$(convertMBM "$file")

        if [ ! -e "$file" ]
        then
            echo File not found: "$file", skipping 1>&2
            continue
        fi

        $convert -flip "$file" "$file"

        if [ ! -z "$(isNormal "$file")" ]
        then
            args="$normal_args"
        else
            args="$color_args"
        fi

        $nvcompress "$args" "$file" "${file/$ext/dds}" && rm "$file"
    done
done

Comments welcomed.

I'm definently a noob for asking this but where does the code go and do i just run ksp.exe when I input the txt?

- - - Updated - - -

Here's a more Linux-y (and Mac-y?) conversion script, using convert from imagemagick and nvcompress from nvidia-texture-tools. It's not entirely foolproof, and my bash-fu is probably terrible, but it works. I don't recommend running this on GameData in a heavily-modded install; it works fine for KWRocketry, Squad, and NASAMission parts, but RoverDude's ART and OKS/MKS parts don't like it at all.

#!/bin/bash

mbm2png=/tmp/mbm2png/mbm2png64
nvcompress=$(which nvcompress)
convert=$(which convert)

image_extensions=("mbm" "png" "tga" "jpg")

normal_pattern="(NRM|nm|Normal|Norm|_n)\."
mbm_pattern="\.mbm$"

normal_args="-bc3n -nomips"
color_args="-bc3"

function isNormal {
    if [ ! $# -eq 1 ]
    then
        echo "isNormal must be called with exactly one argument." 1>&2
        exit 2
    fi

    match=$(echo "$1" | grep -Ei $normal_pattern)

    echo $match
}

function isMBM {
    if [ ! $# -eq 1 ]
    then
        echo "isMBM must be called with exactly one argument." 1>&2
        exit 2
    fi

    match=$(echo "$1" | grep -Ei $mbm_pattern)

    echo $match
}

function convertMBM {
    if [ ! $# -eq 1 ]
    then
        echo "convertMBM must be called with exactly one argument." 1>&2
        exit 2
    fi

    if [ ! -e "$1" ]
    then
        echo "File not found: $1" 1>&2
        exit 1
    fi

    $mbm2png "$1" >/dev/null && rm $1

    echo "${1/mbm/png}"
}

if [ $# -eq 0 ]
then
    paths="."
else
    paths="$*"
fi

for ext in "${image_extensions[@]}"
do
    echo Searching for $ext
    find "$paths" -iname "*.${ext}" | while read file
    do
        [ ! -z "$(isMBM "$file")" ] && file=$(convertMBM "$file")

        if [ ! -e "$file" ]
        then
            echo File not found: "$file", skipping 1>&2
            continue
        fi

        $convert -flip "$file" "$file"

        if [ ! -z "$(isNormal "$file")" ]
        then
            args="$normal_args"
        else
            args="$color_args"
        fi

        $nvcompress "$args" "$file" "${file/$ext/dds}" && rm "$file"
    done
done

Comments welcomed.

I'm definently a noob for asking this but where does the code go and do i just run ksp.exe when I input the txt?

[blowfish]

You will need to run it from a command line.

[shaw]

Here's my DDS converter I upgraded in the recent days to convert KSP textures to DDS.

Source code: https://github.com/ducakar/openzone (`src/ozFactory/ImageBuilder` and `src/tools/ozDDS`)

Linux binaries

It's under GPL3 licence and can also convert 8k x 8k textures and has normal map detection. It's a command-line tool without a (user-friendly) GUI.

[Lilleman]

I'm still working on the user-friendly version for windows, but I've just realized it might ask for DirectX runtimes. So long for the "stand-alone" idea, the full download for this thing is 100MB...

As it's something that might be required for some games, hopefully some people already have it, but now I'm trying to work around that. I'm actually considering rewriting it to use SharpDX, originals DX libraries have been a pain to work with since the beginning... It might have been a little optimistic to say "before the end of the week", but I'll see what I can do today.

[skeevy]

I'm definently a noob for asking this but where does the code go and do i just run ksp.exe when I input the txt?

I'm definently a noob for asking this but where does the code go and do i just run ksp.exe when I input the txt?

That's a bash script and you run it from a Linux terminal.

It would be interesting to combine that script with Shaw's DDS converter. If I understood all the fine nuances of the various image formats and converting them, I would.

@Lilleman

You said you're still working on the user-friendly version for Windows....does that mean you have something working for Linux or am I just thinking too hopeful?

And I'd be happy if your tool supported some ATM like functions like resizing/scaling images for those of us with crappy CPUs or GPUs...and if possible, the ability to change textures like Texture Replacer. An all in one texture tool/mod for KSP would be awesome.

[Urgante]

You can convert texture replacer textures to dds, some of the texture packs are already being released in dds format to use them along with ddsloader.

The option to resize the textures would be interesting.

Shaw, can you give some instructions on how to use your converter from windows please? (or just a good old nice .exe file).

I tried running the binaries from a ubuntu live cd but aparently they needed some libraries and my linux skills are ... almost nonexistant.

Thanks for the converter anyway

[shaw]

What libraries did it need? I thought I included all of them.

I'm still struggling to compile Windows version ...

[Urgante]

What libraries did it need? I thought I included all of them.

I'm still struggling to compile Windows version ...

It says this --> urgante@urgante-VirtualBox:~/Escritorio/linux64$ ./ozDDS./ozDDS: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./ozDDS)

But as I said my linux skills are nonexistant, I tied to execute it putting "./" before it but I'm not even sure if that it's what I'm supposed to do.

If you are struggling to compile it for windows then It's way harder than I thought.

[shaw]

http://stackoverflow.com/questions/20357033/how-to-fix-program-name-usr-lib-x86-64-linux-gnu-libstdc-so-6-version-cxx

It was built on Arch Linux, which uses GCC 4.9 and hence newer, not backward-compatible, standard C++ library.

Could you download ozDDS.zip again, I've added libstdc++ from Arch Linux?

[Urgante]

I downloaded it again but the result was the same error.

It's probably my fault so I will describe how I tried to execute this:

-From a Ubuntu live cd I downloaded the ozDDD.zip, unzip It and execute with "./ozDDS" from a terminal.

- It says -->"urgante@urgante-VirtualBox:~/Escritorio/linux64$ ./ozDDS./ozDDS: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./ozDDS)"

- So I manually copy the libstdc++.so.6 file to the "lib", "lib/x86_64-linux-gnu" and "lib64" folders, but the result is the same error.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

In the end I just installed the 4.9 version of GCC wich was actually easy and fast once I tried and that made the converter work. (maybe not the most elegant solution but easy).

I will convert some textures now and test it.

[Urgante]

I wasn't able to convert non multiple of 4 textures with shaw converter but everything else seems to work, (not a big problem anyway).

The only real problem is that for convert all textures at once I need the linux version of a .bat file with the correct options to detect normal maps, do the color changes and all that complex stuff.

I tried to modify toadicus script for that but It's too complex for me.

[wasmic]

So I tried this out and used paint.NET to convert a lot of the files, making sure to use DXT-1 for textures without alpha channels and DXT-5 for those with. I wasn't sure about the nomral maps, so I let them be. When I start the game, though, many of the textures that I converted have been scrambled, it seems like the UV mapping was broken somehow. I'll try to get some screenshots tomorrow.

[blowfish]

So I tried this out and used paint.NET to convert a lot of the files, making sure to use DXT-1 for textures without alpha channels and DXT-5 for those with. I wasn't sure about the nomral maps, so I let them be. When I start the game, though, many of the textures that I converted have been scrambled, it seems like the UV mapping was broken somehow. I'll try to get some screenshots tomorrow.

Try flipping them vertically. That seems to be a thing which needs to be done...

[Beduino]

I wasn't able to convert non multiple of 4 textures with shaw converter but everything else seems to work, (not a big problem anyway).

The only real problem is that for convert all textures at once I need the linux version of a .bat file with the correct options to detect normal maps, do the color changes and all that complex stuff.

I tried to modify toadicus script for that but It's too complex for me.

FYI: I tested nvcompress x86 (because x64 was not working with pngs) to convert the big pngs and the irregular ones into dds and it seems to work fine.

PS: Did not test ingame, not sure if it needs to be flipped top to bottom or something fancy, but at least the conversion with cuda is also way faster.

Edited October 27, 2014 by Beduino

[Beduino]

Maybe a good way to make DDSLoader more popular would be to write a platform independent script using something like python that uses the nvcompress and mbm tool to convert everything to dds.