[0.90] DDSLoader 1.9 (Mar 9) - Loads DDS Texture "Boringly fast" edition

[Urgante]

I changed the code a little to add .jpg and tried different things to improve the textures, this are the results:

:: Initialization of environment states and variables.
SETLOCAL ENABLEDELAYEDEXPANSION
SET localdir=%CD%\GameData
SET DDSTool="D:\Kerbal Space Program\nvdxt.exe"
SET mbm="d:\Kerbal Space Program\mbm2png7\mbm2png.exe"


FOR /R "%localdir%" %%F IN ("*.mbm") DO (
    CALL %mbm% "%%F"
    IF EXIST "%%~dpnF.png" ( del "%%F" )
)


pause 


FOR /R "%localdir%" %%F IN ("*_NRM.png") DO (
    CALL %DDSTool% -dxt5nm -outsamedir -quality_highest -nomipmap -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)


FOR /R "%localdir%" %%F IN ("*_NRM.tga") DO (
    CALL %DDSTool% -dxt5nm -outsamedir -quality_highest -nomipmap -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)


FOR /R "%localdir%" %%F IN ("*.png") DO (
    CALL %DDSTool% -dxt5 -outsamedir -quality_highest -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)


FOR /R "%localdir%" %%F IN ("*.tga") DO (
    CALL %DDSTool% -dxt5 -outsamedir -quality_highest -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)
pause


FOR /R "%localdir%" %%F IN ("*.jpg") DO (
    CALL %DDSTool% -dxt5 -outsamedir -quality_highest -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)
pause

My current KSP loading time is 1:15 with 50 mods, some of them big, (pimp my kerbals, Better Atmospheres V5 in glorious HD,

KW_Rocketry...).

I actually changed to opengl due to all the mods using more of the maximum ram for 32 bits. (Less fps this way but little difference)

The ram now stays below 2gb so I will probably add even moar mods. http://i.imgur.com/KH98RHc.jpg

In-game loading time when change scenes seems faster.

Now the problems:

- Very big textures make nvdxt.exe fail when trying to convert them. For example GameData\BoulderCo\Clouds\Textures\kerbin1.png

from Better Atmospheres.

Right now I left that particular texture in .png ... but I'm working on that. Paint.net does covert it but the resulting texture is

too big, almost 50mb, so I'm doing something wrong.

- Some textures with extreme proportions turns invisible when ingame, (only noticed jool rings from PlanetaryEditingToolsUI mod).

Since they are small I just not change them.

- nvdxt.exe doesn't convert non multiple of 4 textures. I tried rescaling them but -rescale nearest deforms the image and -rescale hi

doesn't work in the toolbar icons, (they appear purple when loading the game).

Paint.net does convert them without rescaling ... but also increases the size.

Since all the problematic files are small I just let them remain as .png for the moment.

And now the idea for improve this:

I found a tool to access the .tex textures inside the .assets files, convert them to .dds and put them back as .tex.

Better explained here --> http://zenhax.com/viewtopic.php?f=9&t=36

That is interesting by itself and allow things like the sunflare for Better Atmospheres.

So the idea is to turn that .tex images in .dds and make KSP load them as .DDS ... in case of that being a good thing because I

really don't know if the tex format is better or worst than .DDS, I just found it and share the idea so somebody can tell if the .tex

format is good as it is or not.

And now some thoughts:

- There is only 6 days from the creation of the DDSLoader mod and you can see already some mods releasing new versions using this

format, this could very probably be the new texture format to use from now on KSP mods.

- From my original 4 minutes of loading time to 1:15 minutes is a huge difference, and I didn't have any "beautiful textures" mod

back then, my KSP Gamedata folder compressed in DDS was at 600mb, now is at 900mb, (I use the HD version of Better Atmospheres now ).

So I can say that one of the problems of KSP, (long loading times), is solved for me as this mod is right now and it will probably

solve it for everybody in the near future, so thanks sarbian for the mod and thanks to everybody here for testing it and share the

results.

[madlemur]

- nvdxt.exe doesn't convert non multiple of 4 textures. I tried rescaling them but -rescale nearest deforms the image and -rescale hi

doesn't work in the toolbar icons, (they appear purple when loading the game).

Paint.net does convert them without rescaling ... but also increases the size.

Since all the problematic files are small I just let them remain as .png for the moment.

I'm guessing paint.net pads the images out to a power of four, which would increase the file size without deforming the textures.

I've found on more than one occasion that the button icons I converted via the script didn't show up, so I'd load it up in paint.net, and save it back out as DXT1. That seems to work (although it could just be paint.net saving).

I also tweaked the script a bit to include converting mbm files to pngs, then converting the textures. I also treat *NRM, *nm, *norm, and *normal filenames as normal maps.

Need to look for a paint.net command line reference...

[Urgante]

Paint.net doesn't change the Height/Weight, the only problem is the size in megabytes of the resulting .dds image being too big.

I'm using DXT5 as the output format --> http://i.imgur.com/d3OZABE.jpg?1

I guess that I just need to choose the rigth output options ... but unfortunately the real problem here is that I dont know them

Using DXT1 the size is good and it works but the quality of the image is bad.

I could use it for the small toolbar icons but I prefer just use them as .png rather than decreasing quality.

That paint.net command line reference would be really usefull, with that and the rigth output options the problem will be solved.

[slumpie]

Using DXT1 the size is good and it works but the quality of the image is bad.

There is no difference regarding the colours between DXT1c and DXT5. DXT1a is slightly less accurate though.

Now the problems:

- Very big textures make nvdxt.exe fail when trying to convert them. For example GameData\BoulderCo\Clouds\Textures\kerbin1.png

from Better Atmospheres.

8k x 4k is the max possible size it can work with, is the texture breaking this limitation?

Edited October 19, 2014 by slumpie

[slumpie]

-dxt5nm in nvdxt destroys normal maps that do already have the red (x vector) saved in the alpha channel!

Problem and solution (EDIT: Only works if regular xyz or xy_ (the ones that appear purple) normal maps are saved without alpha channel, which sadly isn't always the case.)

To make it easier to understand what normal maps are meant, it's the ones that appear grey if you look at them, not the purple ones (although basically they all are).

As example the RSS normal maps, which do already have swapped channels:

Problem:

As example, RSS uses normal maps with red in alpha and green in green (and somehow also in red and blue, but that doesn't matter in this case)

If you use the -dxt5nm command in nvdxt, then it will replace the alpha data (the correct one for X vector) with the red data (which is wrong) no matter what. It doesn't care about the alpha channel being used.

You will get a texture with all channels being the same,

YOU COMPLETELY LOSE THE X VECTOR!

In other words your normal map is no longer a normal map.

Solution:

First search for normal maps by looking for *nrm.* textures (are there normal maps without this ending?).

Then generally convert them into DXT5nm, but set an exception for 32bit textures (with alpha like the RSS ones).

Example:

FOR /R "%localdir%" %%F IN ("*_NRM.png") DO (

CALL %DDSTool% -dxt5nm -32 dxt5 -outsamedir -quality_highest -flip -file "%%F"

IF EXIST "%%~dpnF.dds" ( del "%%F" )

)

pause

Note:

"-32 dxt5" sets the exception, it doesn't work as a standalone command though.

Important:

• 
  [*=1]You can't use bit size commands such as -32 <format>, -24 <format> without defining a general format, you can only use them as exeptions. I tested it and it didn't work, maybe it's a bug in nvdxt, I don't know.
  [*=1]You can't use dxt5nm as format for the bit exceptions (-32,-24,etc), because nvdxt will throw out an error message and simply refuse to convert the textures.
  

EDIT:

WARNING:

Out of whatever reason are some XYZ or XY_ normal maps (the ones that appear purple) saved with alpha channel. Makes no sense, because they absolutely don't use the alpha, but it makes the -32 command catch them.

Result: -32 uses dxt5 on normals that should still be converted with -dxt5nm...

Edited October 21, 2014 by slumpie

[Urgante]

8k x 4k is the max possible size it can work with, is the texture breaking this limitation?

That particular texture is 8192x4096 and 25mb in png.

I can convert it in paint.net or gimp with DDS plugin using DXT5 resulting in 43mb with mipmaps or 32mb without.

At first I considered that size in mb too big...now I am starting to think that it can't be less without loosing quality and I'm using it.

The image has transparency and needs to have perfect quality so DXT1 is not good enough.

I did found a program to convert all the non multiple of 4 textures to DXT5 no mipmap --> http://developer.amd.com/tools-and-sdks/archive/legacy-cpu-gpu-tools/the-compressonator/

The program itself is easy to use BUT doesn't flip the images and names them *.DDS instead of *.dds so I ended up using fastone image viewer to flip them and other program to rename them.

The whole process is complex so maybe It's not worth it, (arround 300 images converted but they are under 1 mb in total).

In any case I'm currently using 99.9% .dds textures in the gamedata folder without any problem. (Jool rings texture will have to wait until tomorrow).

[slumpie]

8192x4096 is indeed bigger than 8k x 4k, so that sounds right what you're describing.

The calculated size (without mipmaps) for a 8192x4096 texture in DXT5 is indeed 32MB. Be aware that this has nothing to do with the compression qualitly, DXT5 is a format with fixed bitrate. With mipmaps you get roughly 1/3 bigger files.

I'm not sure about the problems with rescaling you're facing, I must test this first.

Same for the Jook rings, I hope I can find the one you're talking about...

[NathanKell]

DXT1 8192x4096 = 4bpp * 8192 * 4096 / 8 / 1024 / 1024 = 16MB

DXT5 8192x4096 = 8bpp * 8192 * 4096 / 8 / 1024 / 1024 = 32MB

DXT is a *constant bit rate* format. An image of given dimensions will *always* have the same file size in DXT1, and that file size will always be half what it is in DXT5. The exception, as you mention, is mipmaps, but images of the same dimension will, again, have the same file size, and DXT1 will be half the size of DXT5.

slumpie: 8192x4096 is what is meant by 8k x 4k, I think Though you ninja'd the rest of my post

[slumpie]

slumpie: 8192x4096 is what is meant by 8k x 4k, I think Though you ninja'd the rest of my post

Depends, there's no definite answer to this.

The one 4k or 8k resolution doesn't exist, these terms have been used to describe tons of different resolutions.

For 8k from 8k FUHD (7680x4320) up to 8k fulldome (8192x8192). 8k is two times 4k in both dimensions? But it's the same weird situation for 4k. Don't even start with 2k, it's even worse with that one. Not even the aspect ratios are given.

I don't know the exact limititation, all the Nvidia description says is "8k x 4k". 8000x4000 maybe? Because they were too lazy to write the full number?

Edited October 19, 2014 by slumpie

[madlemur]

I have encountered normal maps with nrm, norm, normal, and even no suffix at all. It's something of a pain in the ass...

[rbray89]

I have encountered normal maps with nrm, norm, normal, and even no suffix at all. It's something of a pain in the ass...

This. This is why ATM has configs for this, among other things.

[slumpie]

@rbray or sarbian (I hope)

Does the shader fail if a normal map still has X in the red channel or is it smart enough to detect that?

How does the stock game or unity differ between those textures anyway? I mean it has to load and convert those normal maps somehow.

---------

The program itself is easy to use BUT doesn't flip the images and names them *.DDS instead of *.dds so I ended up using fastone image viewer to flip them and other program to rename them.

You can use "-format .dds" as command for TheCompressonator.

Edited October 19, 2014 by slumpie

[rbray89]

@rbray or sarbian (I hope)

Does the shader fail if a normal map still has X in the red channel or is it smart enough to detect that?

How does the stock game or unity differ between those textures anyway? I mean it has to load and convert those normal maps somehow.

---------

You can use "-format .dds" as command for TheCompressonator.

No, the shaders require AG normal maps.

The game loads MBMs directly (AG) but assumes png and tga (likely others) as RG and converts them to AG.

[slumpie]

No, I mean how does it differ between as example diffuse and normal maps that do not end with nrm*, normal* or similar.

It doesn't?

Edited October 19, 2014 by slumpie

[NathanKell]

It doesn't. Those will fail to work, and an error will be placed in the log like "Could not apply zzzzz as normal map, incorrect format"

[slumpie]

But... why do we even care about them?

[Urgante]

I have found the part of the code that I think changes the color of the jool rings textures while in-game:

PlanetJool{
	AdditionalData
	{
		Stock = True
		AddAtmoFx = False
		AtmoWaveColour = RGBA(0.6745098, 0.5960785, 0.8509804, 0.000)
		AddOceanFx = False
		AddRings = True
		Rings
		{
			Ring
			{
				Tilt = 8
				OuterRadius = 1850
				InnerRadius = 1900
				Colour = RGBA(0.300, 0.724, 0.300, 1.000)
			}
			Ring
			{
				Tilt = 8
				OuterRadius = 1650
				InnerRadius = 1850
				Colour = RGBA(0.700, 1.000, 0.483, 1.000)
			}
			Ring
			{
				Tilt = 8
				OuterRadius = 1400
				InnerRadius = 1650
				Colour = RGBA(0.700, 1.000, 0.483, 1.000)
			}
		}
	}


}

So besides having KSP accepting the image I need to also use the correct format for this code.

Well, trial and error doesn't worked.

I think the key of this is in the alpha channel and that RGBA thing but I don't know the correct paint.net option to choose.

[Ph34rb0t]

But... why do we even care about them?

Because of "HIGH QUALITY ULTRA HD 8092 RESOLUTION TEXTURES ON A 52 INCH SCREEN SUCK IT LOSERS" video quality harlots?

[toadicus]

Here's a more Linux-y (and Mac-y?) conversion script, using convert from imagemagick and nvcompress from nvidia-texture-tools. It's not entirely foolproof, and my bash-fu is probably terrible, but it works. I don't recommend running this on GameData in a heavily-modded install; it works fine for KWRocketry, Squad, and NASAMission parts, but RoverDude's ART and OKS/MKS parts don't like it at all.

#!/bin/bash

mbm2png=/tmp/mbm2png/mbm2png64
nvcompress=$(which nvcompress)
convert=$(which convert)

image_extensions=("mbm" "png" "tga" "jpg")

normal_pattern="(NRM|nm|Normal|Norm|_n)\."
mbm_pattern="\.mbm$"

normal_args="-bc3n -nomips"
color_args="-bc3"

function isNormal {
	if [ ! $# -eq 1 ]
	then
		echo "isNormal must be called with exactly one argument." 1>&2
		exit 2
	fi

	match=$(echo "$1" | grep -Ei $normal_pattern)

	echo $match
}

function isMBM {
	if [ ! $# -eq 1 ]
	then
		echo "isMBM must be called with exactly one argument." 1>&2
		exit 2
	fi

	match=$(echo "$1" | grep -Ei $mbm_pattern)

	echo $match
}

function convertMBM {
	if [ ! $# -eq 1 ]
	then
		echo "convertMBM must be called with exactly one argument." 1>&2
		exit 2
	fi

	if [ ! -e "$1" ]
	then
		echo "File not found: $1" 1>&2
		exit 1
	fi

	$mbm2png "$1" >/dev/null && rm $1

	echo "${1/mbm/png}"
}

if [ $# -eq 0 ]
then
	paths="."
else
	paths="$*"
fi

for ext in "${image_extensions[@]}"
do
	echo Searching for $ext
	find "$paths" -iname "*.${ext}" | while read file
	do
		[ ! -z "$(isMBM "$file")" ] && file=$(convertMBM "$file")

		if [ ! -e "$file" ]
		then
			echo File not found: "$file", skipping 1>&2
			continue
		fi

		$convert -flip "$file" "$file"

		if [ ! -z "$(isNormal "$file")" ]
		then
			args="$normal_args"
		else
			args="$color_args"
		fi

		$nvcompress "$args" "$file" "${file/$ext/dds}" && rm "$file"
	done
done

Comments welcomed.

[slumpie]

Because of "HIGH QUALITY ULTRA HD 8092 RESOLUTION TEXTURES ON A 52 INCH SCREEN SUCK IT LOSERS" video quality harlots?

Doesn't prevent you from using proper endings.

- nvdxt.exe doesn't convert non multiple of 4 textures. I tried rescaling them but -rescale nearest deforms the image and -rescale hi

doesn't work in the toolbar icons, (they appear purple when loading the game).

Paint.net does convert them without rescaling ... but also increases the size.

Since all the problematic files are small I just let them remain as .png for the moment.

Rescale nearest worked fine for the textures I used, what deformations did you see? It always adjusted the size nicely so that it stayed the same.

Rescale hi worked just as well.

What toolbar icons are you talking about, the ones for the toolbar mod?

Edited October 19, 2014 by slumpie

[Urgante]

Rescale nearest worked fine for the textures I used, what deformations did you see? It always adjusted the size nicely so that it stayed the same.

Rescale hi worked just as well.

What toolbar icons are you talking about, the ones for the toolbar mod?

The problem is solved now, I used other programs to do the conversion without resizing, (used compressonator back then, I have a batch plugin for paint.net now),

A lot of textures worked fine rescaling, and -rescale hi was good enough in quality and size terms. The problem was that some toolbar mod icons didn't worked once rescaled, KSP itself load the rescaled texture fine but they appear either purple or deformed once in-game.

It only happens to some of the non multiple of 4 textures with transparency and the particular effect depends of the program used for the conversion.

Easy to spot example textures would be Ascent_Guidance.png or Landing_Guidance.png in GameData\MechJeb2\Icons folder.

Example and code used to rescale:

http://i.imgur.com/kKs9XTN.jpg

FOR /R "%localdir%" %%F IN ("*.png") DO (    CALL %DDSTool% -dxt5 -outsamedir -rescale hi -quality_highest -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)

I guess that it should be possible to use -rescale hi and some particular DDS format to do the job, default DXT1 has poor quality with transparency and default DXT5 turns purple or just doesn't load.

I would really apreciate some explanation in that RGB and RGBA thing or at least the correct nvdxt.exe command to use them, there is a lot of diferents formats and combinations and that optional alpha thing has something to do with transparency but I don't understand it ....yet, (I'm returning from jool at this moment and I want to load those rings in .DDS next time I go there http://i.imgur.com/uExvSRU.jpg).

[slumpie]

I don't know what the "PlanetaryEditingToolsUI" mod is. Where can I find that texture?

You can try -u8888 for uncompressed 32bit RGBA dds, but 32bit textures are damn big. Size is 4 times as big compared to a DXT5 compressed texture.

I'll have a look at MJ. I have it installed, but never used the toolbar icons.

Edited October 19, 2014 by slumpie

[Urgante]

The mod itself can be found here --> http://forum.kerbalspaceprogram.com/threads/68619-Plugin-WIP-KittopiaTech-Ingame-Terraforming-Tools-V0-125

The texture and the file that changes it's color in-game is added by better atmospheres. Here are those two files only so you don't have to download it all and search them, just combine the two folders https://mega.co.nz/#!VFJEVAYb!A2jeWjhDfb2rsojPWmq7F1ufSonltVuxZt0cSAfSvo4

I will try the -u8888 command right now.

(EDITED) Nope, the command works fine but jool rings remain invisible in-game.

Edited October 19, 2014 by Urgante

[slumpie]

I can see the ring, it's very dark though. Not sure if it's supposed to be like this, but actually Jool itself is relatively dark too.

Dark doesn't mean it's grey or black, I can see details and some green, it's just dark.

Are you sure you downloaded the 1.1 verison of ddsloader? Because version 1.0 didn't have support for u8888 dds files.

Edit: Meh, what am I talking here, just have a look:

---

By the way, I have an idea for icons. I think 1.5.5.5 would work well with icons, because the sharp alpha avoids blurry edges and makes them appear sharper.

There's also no need for rescaling, because it supports any resolution, it doesn't have to follow the power of two rule.

The command would be -u1555. I must test this a bit more though.

Edited October 19, 2014 by slumpie

[NathanKell]

Problem is that KSP (and mods) require certain sizes of certain icons. Like 38x38, which is not divisible by 4. It's worth checking that icons don't appear to shift/stretch/shrink after conversion/resize-to-4power.