[0.90] DDSLoader 1.9 (Mar 9) - Loads DDS Texture "Boringly fast" edition

[sarbian]

shaw sent me a small patch that fix something I did not check for : the texture were not properly set as read only, so they would would eat memory instead of going to the GPU.

So here is 1.2 with a 1 line fix

[slumpie]

Well, I was focusing on mbm files. And they seems to have their own rules regarding swizzling... I just ran some tests with TGAs, and this time Unity expect xGxR format. So you were all right about that, RGRA isn't a format Unity should use (I really don't get why this is working.)

No Lilleman, I did test mbm normal maps, I even used the very same part, the X200-16 tank.

It doesn't matter what's in the red and blue channel. I tested it with mirrored green, with red and blue set to 1, to 0, to anything. I put different greyscale images with smilies into the red and blue channel and it didn't make the slightest difference (maybe a few more artefacts).

EDIT: Here's the mbm to dds converted normal map with two different smilie images in the red and blue channel: https://www.dropbox.com/s/w3iqxng8znuvlqa/model001.dds?dl=0

NOTE: THIS IS FOR THE X200-16 FUEL TANK!

Edited October 21, 2014 by slumpie

[Lilleman]

Well, as I said, I don't really understand why it works. It shouldn't. I tried different swizzling from a raw image (mbm files are 20bytes header + raw data) without being able to get the "correct" format to work. Here is the dds my prog produce (using RGBA->RGRA logic), and this is what it looks like in-game...

If someone has any explanation to this, please let me know, I'm trying to code a tool to automatize the process, and, even I got it working somehow, I want it to be working properly...

Edit: this is still for the Rockomax x200-16.

Edit2: This dds probably contains mipmaps, so you might get some weird results if you're zooming too far, that's just because I didn't implement the option yet (it's planned).

Edit3 : Sarbian, Github is up to date, but the link for the build in front page still point to 1.1.

Edited October 21, 2014 by Lilleman

[KevinCathcart]

Lilleman: The correct swizzling for mbm is RGBA=>RGBA (I.E. no swizzling.)

That is because all mbm textures are RAW decompressed versions of DXT* textures dumped straight from the unity game engine.

That is literally how part-tools creates them. Check the source code of an older version of part-tools, it is as simple as can be. All that is needed to to simply re-compress them.

I repeat that again, because it is important: After the header, MBM files are RGBA dumps of raw texture data at mip level 0.

Also since MBM are a dump textures, they should not need to be flipped in the conversion process, unlike the other formats. That is because unity already did the flipping relative to the original source format.

So basically you have

non-normal map:

non-mbm:

If 24 bit or 32 bit with all alphas zero, flip and convert to DXT1.

If it has non-zero alpha values, flip convert to DXT5.

mbm:

If 24 bit or 32 bit with all alphas zero, convert to DXT1.

If it has non-zero alpha values, convert to DXT5.

normal maps:

non-mbm:

Swizzle RGBA (XYZ-) to GGGR (-Y-X) and convert to DXT5, resulting in DXT5nm.

mbm:

Do not swizzle, but convert directly to DXT5, resulting in DXT5nm

[shaw]

That's not correct. It should be:

non-normal map:

non-mbm:

If 24 bit or 32 bit with all alphas

255

, flip and convert to DXT1

with mipmaps (unless an UI icon)

.

If it has

non-255

alpha values, flip and convert to DXT5

with mipmaps (unless an UI icon)

.

mbm:

If 24 bit or 32 bit with all alphas

255

, convert to DXT1

with mipmaps (unless an UI icon)

.

If it has non-

255

alpha values, convert to DXT5

with mipmaps (unless an UI icon)

.

normal maps:

non-mbm:

Flip

, swizzle RGBA (XYZ-) to GGGR (-Y-X) and convert to DXT5

with mipmaps

, resulting in DXT5nm.

mbm:

Do not swizzle, but convert directly to DXT5

with mipmaps

, resulting in DXT5nm

[Lilleman]

Thanks for the clarification, I was thinking mbm files were handled the same way as any other file, and it was driving me nuts. The RGRA thing was something I came up with after trying to reproduce nvdxt's dxt5nm files. I guess I can cut that swizzling part out of my code now. I've spent way too much time trying to figure this out... I would like to apologize to everyone who tried to help me, somehow I tried to make things more complicated than they really were...

For the rest, I ended up with a (kind of) similar logic for conversion (sort of, I actually convert non-mbm normals to (0x80)G(0x80)R instead of GGGR, but that's not really important in this case), so fortunately I won't have to re-write anything in my code. The next tricky part will be to determine if a TGA/PNG file is a normal (I hope checking filenames and blue channel will be enough, because it's my only idea for now).

@shaw: Are you sure normals should have mipmaps? I made one test, converting the av-r8 winglet normal with mipmaps and I got weird results when zooming out.

Edited October 22, 2014 by Lilleman

[slumpie]

Experiment with the different filtering options and try to normalize the mipmaps. Nvdxt can do all of this.

[Lilleman]

My bad, I missed an argument when loading my normals trough D3D's TextureLoader... Apparently if you don't apply a filter to mipmaps manually, DX decide to convert those into a blurry mess... But only for normals, regular textures mipmaps looks fine...

I still got some tests to do to see wich one provide the best quality, but at least I know where the problem come from.

Sorry for wasting everyone's time again...

[blowfish]

Sorry for wasting everyone's time again...

On the contrary - someone needs to figure out what format KSP needs its textures in so that people can make working mods with the best visual quality. Keep up the good work

[shaw]

I just discovered that DDS supports normal map flag. I noticed GIMP recognises DXT5nm-saved files apart from regular DXT5. It converts DXT5nm files back to RGB=XYZ when opened.

Here's the header that defines (non-standard) DDPF_NORMAL flag used in DDS_PIXELFORMAT.dwFlags field: http://gimp-dds.googlecode.com/svn-history/r215/trunk/dds.h

[sarbian]

A quick search shows that blender also support this flag. Let's use that then

I won't be able to look into it before my return on the 5th however.

Edit : and it's in nivdia tools too it seems. less and less non-standard

[Lilleman]

So, just to be sure I can keep the code I already made for modifying byte#35: This time, byte#76=0x80, right?

I hope it is, in this case I'll just have to modify one line. Anyway, I've almost finished my converter, just have some cleaning to do in the source.

I swear this is the last time I deal with DirectX directly, next time I'll go with an external library: I finished squeezing memory leaks at 3AM, and it still sometimes randomly crash when it load a tga or a png (that sounds familiar).

For now it's very basic: it can load a mbm, tga or png and convert it automatically into a dds, and it can do the same with a whole folder (with a list of folder/files to exclude if needed).

Since I plan to release it eventually (hopefully before the end of the week), anyone think of a function I could implement while I'm at it?

On the contrary - someone needs to figure out what format KSP needs its textures in so that people can make working mods with the best visual quality. Keep up the good work

I don't know if running straight into every possible obstacle could be considered as "good work", but, yeah, I guess someone had to do it...

Edited October 23, 2014 by Lilleman

[sarbian]

shaw submited the patch for the DDPF_NORMAL flag, and with hte magic of github and jenkins I just built 1.4

[Urgante]

A million of thanks for working in that converter Lilleman , I tried a lot of converters and configurations in the nvdxt .bat without finding a universal solution, (what worked for some textures didn't work for others), so having a tool to convert all of them is just what we needed.

[frash23]

How stable is this for a lightly modded 0.25 game?

Are there any downsides? How much are loading times improved?

[slumpie]

Since I plan to release it eventually (hopefully before the end of the week), anyone think of a function I could implement while I'm at it?

A filter for a minimum size (pixel or perhaps kB) to exclude icons or similar. Maybe with an additional converting option, as example uncompressed u888/u8888.

How do you handle rescaling of textures with block compression incompatible resolution?

shaw submited the patch for the DDPF_NORMAL flag, and with hte magic of github and jenkins I just built 1.4

And what exactly are the advantages of this?

Edited October 23, 2014 by slumpie

[shaw]

And what exactly are the advantages of this?

To avoid some harmless warning messages. KSP won't try (and fail) to convert normal maps without "NRM" suffix from RGB to GGGR during model loading.

[shand]

How stable is this for a lightly modded 0.25 game?

Are there any downsides? How much are loading times improved?

i've got what i consider a heavily modded version, but its not heavy by most peoples count (NEAR, realchutes, procedural fairings, mechjeb, remotetech, RCT, contracts+ and a few USI mods).

what i've noticed is roughly half loading time (nice on my low spec machine). my only issues have been it pretty much broke the icons for the toolbar (oh i have toolbar too), so i've had to copy them back in the original format... that said i have only a basic understanding what i am doing and havent followed this conversation for a few pages so they may have worked out what was up and things by now)

Summary: halfs load time, seems pretty damn stable to me with the exception of toolbar icons when i did it.,

[Lilleman]

A filter for a minimum size (pixel or perhaps kB) to exclude icons or similar. Maybe with an additional converting option, as example uncompressed u888/u8888.

How do you handle rescaling of textures with block compression incompatible resolution?

For now, it only ignore 1*1pixel images, but I can add an option for a minimal resolution. Since I got most icons working properly, I don't think this will be necessary, though.

By default, the program check a text file listing every exception, it can be a single file or a whole folder. For now, it only has 1 line to exclude some kOS textures. The rest seems to be working, but I'm not done testing, so I'm sure this list will grow.

As for block compression, nothing is rescaled, but I didn't encountered problems yet. Any mod in particular where this is a problem? I would like to see if this works.

I didn't tried uncompressed formats yet, I'll test those and see how that works.

I'm wondering if adding an option to reduce texture resolutions would be a good idea... It's kind of stealing ATM's work...

Edit: I feel like I'm polluting Sarbian's thread with all this, maybe this should go to another thread while it's not ready to be released.

Edited October 23, 2014 by Lilleman

[frash23]

i've got what i consider a heavily modded version, but its not heavy by most peoples count (NEAR, realchutes, procedural fairings, mechjeb, remotetech, RCT, contracts+ and a few USI mods).

what i've noticed is roughly half loading time (nice on my low spec machine). my only issues have been it pretty much broke the icons for the toolbar (oh i have toolbar too), so i've had to copy them back in the original format... that said i have only a basic understanding what i am doing and havent followed this conversation for a few pages so they may have worked out what was up and things by now)

Summary: halfs load time, seems pretty damn stable to me with the exception of toolbar icons when i did it.,

Amazing. I'll have to get around that script

[Beduino]

My current KSP loading time is 1:15 with 50 mods, some of them big, (pimp my kerbals, Better Atmospheres V5 in glorious HD,

KW_Rocketry...).

I actually changed to opengl due to all the mods using more of the maximum ram for 32 bits. (Less fps this way but little difference)

The ram now stays below 2gb so I will probably add even moar mods. http://i.imgur.com/KH98RHc.jpg

In-game loading time when change scenes seems faster.

After reading this.. im going to give it a go. My ksp takes like 6 minutes to load and is using too much memory recently and im not running as many mods as before, like interstellar, kw, novapunch, nearfuture, b9.

Edited October 23, 2014 by Beduino

[Urgante]

After reading this.. im going to give it a go. My ksp takes like 6 minutes to load and is using too much memory recently and im not running as many mods as before, like interstellar, kw, novapunch, nearfuture, b9.

The script I posted doesn't work with all the textures:

- 8k textures just fail to convert

- Textures with resolutions non multiple of 4 fail to convert.

- Some normal maps are lost in the conversion, a lot of them actually. I didn't noticed the effect back then but you can see an example here. http://imgur.com/a/qkUKs#0

- Some textures become invisible, no idea why. (Only two as far as I know).

So I would recommend to wait for Lilleman converter unless you know how to properly convert the textures or just want to play with wrong normal maps, (not that bad actually, I didn't even noticed until I saw the example picture).

[Beduino]

The script I posted doesn't work with all the textures:

- 8k textures just fail to convert

- Textures with resolutions non multiple of 4 fail to convert.

- Some normal maps are lost in the conversion, a lot of them actually. I didn't noticed the effect back then but you can see an example here. http://imgur.com/a/qkUKs#0

- Some textures become invisible, no idea why. (Only two as far as I know).

So I would recommend to wait for Lilleman converter unless you know how to properly convert the textures or just want to play with wrong normal maps, (not that bad actually, I didn't even noticed until I saw the example picture).

Thanks for all the details, im in the middle of the conversion right now, script is running. So im going to check load times and memory consumption at least.

Im wondering though why have u used the primitive nvdxt instead of the modern version. https://developer.nvidia.com/gpu-accelerated-texture-compression

Maybe it would help with some of the problems.

[rbray89]

Thanks for all the details, im in the middle of the conversion right now, script is running. So im going to check load times and memory consumption at least.

Im wondering though why have u used the primitive nvdxt instead of the modern version. https://developer.nvidia.com/gpu-accelerated-texture-compression

Maybe it would help with some of the problems.

OOOH! And it has an MIT license

[KevinCathcart]

That's not correct. It should be:

Delayed reply:

You are obviously right. I forgot the flip on mbm, and I always get the alpha value backwards. I think of alpha as "transparency", when it is actually "opacity".

As for not specifing "with mipmaps", that was because it should be obvious, and that was already covered by previous posts.

The exception being textures that get editited live like certain RasterPropMon textures, since re-calculating mipmaps each frame is too expensive. But it sounds like everything that needs to be said on this topic is now said.