[0.90] DDSLoader 1.9 (Mar 9) - Loads DDS Texture "Boringly fast" edition

[sarbian]

In my previous post, I've said that I had to ditch ATM after conversion to make everything work.

I just understood what is going on. The code I gave rbay for the new ATM disable all the images loader, including this one. I'll have to submit him a patch so that it only disable the stock one. But I can't do it today (unless he reads me ).

Edited October 15, 2014 by sarbian

[slumpie]

Okay. I've removed it from the post containing my script.

I'm not really familiar with NVidia's tools, only with Imagetool that comes with Flight Simulator X' SDK. To generate a valid FSX bump map in DDS format with it, you use its -RedInAlpha switch and simply I assumed that -n4 was something similar. My first attempt at a conversion actually was with Imagetool, but it's awfully picky regarding input format and outright refused to read the second .tga it had to process.

Red in Alpha switch?

That's pretty much what DXT5nm does. I don't know if you did read it, but that was exactly what I suggested in this post (click). (Actually I suggested a whole lot more than just that.)

The difference between normals in DXT5 and DXT5nm is 5.6 bit compared to 6.8 bit, all that for the exact same file size. Actually another factor that speaks for DXT5nm is that the channels are compressed seperately and therefore they don't influence each other, which usually causes artefacts and colour shifts.

Edited October 15, 2014 by slumpie

[rbray89]

I just understood what is going on. The code I gave rbay for the new ATM disable all the images loader, including this one. I'll have to submit him a patch so that it only disable the stock one. But I can't do it today (unless he reads me ).

Just saw this. I can take care of it tonight.

[Ph34rb0t]

Red in Alpha switch?

That's pretty much what DXT5nm does. I don't know if you did read it, but that was exactly what I suggested in this post (click). (Actually I suggested a whole lot more than just that.)

I've just skimmed over it, TBH. I thought DXT5nm was just a switch supported by the newer TextureTools, so I initially disregarded it. I've updated the scripts on page 7 for DXT5nm. Thanks.

The difference between normals in DXT5 and DXT5nm is 5.6 bit compared to 6.8 bit, all that for the exact same file size. Actually another factor that speaks for DXT5nm is that the channels are compressed seperately and therefore they don't influence each other, which usually causes artefacts and colour shifts.

That might explain why some normalmaps by some add-on authors for FSX don't work properly as they've merely exported it in DXT5 from Photoshop. I've always suggested converting normalmaps from the source file with Imagetool and the appropriate -RedInAlpha parameter. Never knew until now that this actually produces a DXT5nm bitmap (that can be directly exported from GIMP). So thanks again.

And to be perfectly honest, a lot of posts in this thread are rather philosophical and overly technical to me (no offense), instead of providing practical, concise and workable solutions. That's why my interest in digging through everything isn't really profound. (If I had more time, though...)

[slumpie]

And to be perfectly honest, a lot of posts in this thread are rather philosophical and overly technical to me (no offense), instead of providing practical, concise and workable solutions. That's why my interest in digging through everything isn't really profound. (If I had more time, though...)

Well, I tried that at least.

I can give you a very straightforward suggestion:

Use -quality_highest instead of normal

It doesn't affect the resulting file size, because the formats have locked bitrates. It will take a little longer to compress, but since you only do that once...

[Nuke]

Microsoft doesn't seem to think they're significant enough to support.

thats because its in their interest to get people to upgrade.

[Ph34rb0t]

Well, I tried that at least.

I can give you a very straightforward suggestion:

Use -quality_highest instead of normal

It doesn't affect the resulting file size, because the formats have locked bitrates. It will take a little longer to compress, but since you only do that once...

Noted and implemented. Thanks.

[Lilleman]

Sooooo... Win64 KSP with DX9 is working again, with more than 4GB of mods loaded, using this plugin (and after converting every single texture in the GameData folder)... Since the release of 0.25, it crashed even without mods (I had no problem running it with OpenGL, though).

I keep in mind it might just be some luck, since the behaviour of the win64 build is known to be random (to say the least), but, anyone else tried? Does it worked?

Because now I'm suspecting the KSP's default texture loader to be the cause of memory leaks...

Edited October 17, 2014 by Lilleman

[Urgante]

First of all, sorry for my humble attempt at english, I am working on it.

Now to the point:

I found a .mbm converter https://www.dropbox.com/s/txthm7h5jwqgnz5/mbm2png7.zip?dl=1

I used it to convert all my .mbm to png and then used Ph34rb0t code to convert to dds.

This Reduced my ksp load time from 4min to 1,5min. (I have a SSD), and my Gamedata folder from almost 2gb to 600mb.

Unfortunately the memory usage once KSP is running is the same or very similar, (hard to tell).

As far as I can tell all the textures work perfectly, including transparent toolbar icons.

I didn't noticed a difference in texture quality but I guess that there is one, (pass from 2gb to 600mb without change the quality doesn't seem probable).

I copy the code I used here so you can tell me if any of this is actually good or the .mbm didn't needed a change to begin with .... because, let's face it ... I have no idea what the heck did I really achieved, I just found it and put some code together to see what happens.

:: Initialization of environment states and variables.
SETLOCAL ENABLEDELAYEDEXPANSION
SET localdir=%CD%\GameData
SET DDSTool=[B]"D:\Kerbal Space Program\nvdxt.exe"[/B]
SET mbm=[B]"D:\Kerbal Space Program\mbm2png7\mbm2png.exe"[/B]


FOR /R "%localdir%" %%F IN ("*.mbm") DO (
    CALL %mbm% "%%F"
    IF EXIST "%%~dpnF.png" ( del "%%F" )
)
pause
FOR /R "%localdir%" %%F IN ("*_NRM.png") DO (
    CALL %DDSTool% -dxt5nm -outsamedir -quality_highest -nomipmap -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)
pause
FOR /R "%localdir%" %%F IN ("*_NRM.tga") DO (
    CALL %DDSTool% -dxt5nm -outsamedir -quality_highest -nomipmap -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)
pause
FOR /R "%localdir%" %%F IN ("*.png") DO (
    CALL %DDSTool% -dxt5 -outsamedir -quality_highest -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)
pause
FOR /R "%localdir%" %%F IN ("*.tga") DO (
    CALL %DDSTool% -dxt5 -outsamedir -quality_highest -flip -file "%%F"
    IF EXIST "%%~dpnF.dds" ( del "%%F" )
)
pause

This will search gamedata folder for .mbm, convert them to .png and delete original .mbm, after that it just uses Ph34rb0t code to do the usual .DDS conversion.

At the end all textures will be in DDS format...except the ones with sizes not multiple of 4. ¿Any way to convert them without deformations?

It takes a lot of time to do the conversion, around 15 minutes depending your mods.

You need to put the nvdxt.exe file from http://es.originaldll.com/file/nvdxt.exe/39376.html in your KSP folder along with the uncompressed folder of the .mbm converter and/or change the paths.

Thanks for all the work doing the mod and testing it, i can now load my KSP in less than half the time

Edited: Thanks for the alternative link technicalfool

Edited October 18, 2014 by Urgante
Blocked link

[technicalfool]

Please note that the site you linked to has been blocked by the site filters here. It's likely that it's an illegal scrape of the old Kerbal Space Port website before it was taken down. Here's a working link to the mbm-to-png converter:

https://www.dropbox.com/s/txthm7h5jwqgnz5/mbm2png7.zip?dl=1

[NathanKell]

Note that since all the KSP loader does *is* convert the mbms to DXT (dds) it's not surprising memory usage will remain the same.

What you did is just doing that step once and saving the result.

It is however good to know that the mbm loader doesn't leak, if your memory usage didn't decrease.

[slumpie]

Unfortunately the memory usage once KSP is running is the same or very similar, (hard to tell).

If you use that batch file to convert all textures you are actually wasting memory.

The ony difference between alpha-less textures in DXT1a/c and DXT5 is their size, to be more precise DXT5 will bloat them to double the size, literally.

I mentioned some very plausible ways in a previous post, at least for KSP in its current state, as to how to differ between textures with and without alpha just by using nvdxt commands. Thanks to how uniform the textures are that Unity spits out and the few formats KSP accepts.

As far as I can tell all the textures work perfectly, including transparent toolbar icons.

I didn't noticed a difference in texture quality but I gees that there is one, (pass from 2gb to 600mb without change the quality doesn't seem probable).

There's no real difference, because you basically just did the same as the game would normally do.

At the end all textures will be in DDS format...except the ones with sizes not multiple of 4. ¿Any way to convert them without deformations?

It takes a lot of time to do the conversion, around 15 minutes depending your mods.

Nvdxt can rescale textures to resolutions that follow the power of two rule.

-

Read this post: http://forum.kerbalspaceprogram.com/threads/96729-0-25-DDSLoader-1-1-%28Oct-14%29-Loads-DDS-Texture-Yes-it-is-boring-edition?p=1478347&viewfull=1#post1478347

Edited October 18, 2014 by slumpie

[NathanKell]

slumpie: except it was done without the memory leaks and loader bugs that the game's PNG and TGA loaders suffer.

[slumpie]

Sure, but there's still some potential left.

It really doesn't make much sense to convert 24bit textures into DXT5. It's a matter of a single additional command.

Of course is it already an improvement, but why not try and make it even better?

[NathanKell]

Oh, indeed! That I don't take issue with in the slightest.

I was taking issue only with your statement that it was doing nothing that the game itself didn't, but since I didn't quote it, how were you to intuit my intentions? :]

Actually, it would be best if it detected if a file was a normal map based first on checking ATM's list of normal maps, and then (if that fails) perhaps if the file was mostly blue-purple to catch the leakers, or already in DXT5nm-ish format just in png (to catch RSS normal maps).

Then compress all remaining 24bit images to DXT1, and the 32s to DXT5.

Edited October 18, 2014 by NathanKell

[slumpie]

Actually, it would be best if it detected if a file was a normal map based first on checking ATM's list of normal maps, and then (if that fails) perhaps if the file was mostly blue-purple to catch the leakers,

I guess you're talking about normal maps that were not created via Unity? Because the normal maps Unity creates do always end with _nrm.*, don't they?

Let's assume they're not marked:

If all tangent normal maps were saved with only 2 "active" channels, namely only red and green, it probably wouldn't be too difficult. Then it's only a matter of finding files with blue set to 1 (or was it the other way round...?);(it would hit diffuse textures with full blue in them though...).

------------- (technical blabla you don't really have to read)

Some tangent normal maps do still make use of the blue channel as a form of alignment for re-normalization, although it's actually not needed.

BUT, if you could do a quick mathematical check on those normal maps with saved blue channel, then it would very much be possible. The colours are just vectors, as a result you can always calculate/check for instance the blue channel with this formula: z(blue) = sqrt(1 - x*x(red) + y*y(green)) and give it a certain degree of tolerance (because you will never get 100% precision).

Clarification: I don't think they're really used in KSP, but I'm not sure. Unity definitely doesn't use them.

-------------

or already in DXT5nm-ish format just in png (to catch RSS normal maps).

Ehm... you normally don't put the red colour into the alpha channel. Although it's actually a simple matter of swapping channels, but yeah...

EDIT: I see, I see... that's actually the case.

I think you'd have to use -dxt5 then, because the -dxt5nm command would delete the alpha and replace it with the red instead if it isn't smart enough to detect already swapped channels, even though that was already done. I must test this first though...

Edited October 19, 2014 by slumpie

[rbray89]

I guess you're talking about normal maps that were not created via Unity? Because the normal maps Unity creates do always end with _nrm.*, don't they?

If all tangent normal maps were saved with only 2 channels, namely only red and green, it probably wouldn't be too difficult. The problem is that many tangent normal maps do still make use of the blue channel as a form of alignment, although it's actually not needed. That's the reason why formats such as DXT5nm, BC3, BC5, etc, come in very handy, because they work with two component colour source data.

BUT, if you could do a quick mathematical check on those normal maps with saved blue channel, then it would very much be possible. The colours are just vectors, as a result you can always calculate/check for instance the blue channel with this formula: z(blue) = sqrt(1 - x*x(red) + y*y(green)) and give it a certain degree of tolerance (because you will never get 100% precision).

If it matches -> you have a normal map

If it doesn't -> something else, probably diffuse made by an insane designer.

But don't ask me how to write a shader... meh!

Ehm... you normally don't put the red colour into the alpha channel. Although it's actually a simple matter of switching channels, but yeah...

DXT5nm was never meant to be optimal, it was just a misused DXT5 if you want so...

Alternatively, V8U8 comes to my mind, a 16bit dds format without compression. Although I'm not sure about the hardware or even software support.

By the way nvdxt reads .tga, .bmp, .gif, .ppm, .jpg, .tif, .cel, .dds, .png, .psd, .rgb, *.bw .rgba files.

Pretty much.

Regarding the 16bit (who uses those anyway? ) and 8bit (very rare) textures: The 16bit textures are too hard to distinguish and the 8bit ones are not worth any effort, actually both are not really worth the effort. I wouldn't touch them at all.

In standard Unity shaders, all normal map info is stored in the Green and Alpha. This is combined with tangent information to form a vector.

You don't store anything in the other colors to make sure that you get the best range from A&G with DXT5 compression.

[Dr. Jet]

Sarbian, you are a genius! Most .mbm based folders like Squad and KWRocketry have debloated twice in size and almost twice and half in loading time!

BTW, to convert mods one-by-one (with compatibility checking afterwards) it's better to use "SET localdir=%1" in that converter batch file. Drag folder you want on .bat and it will convert.

[slumpie]

In standard Unity shaders, all normal map info is stored in the Green and Alpha. This is combined with tangent information to form a vector.

If you select standard shaders unity saves red in alpha, but if you use KSP shaders it doesn't?

I was talking about the source, the png, tga or mbm textures.

Edited October 19, 2014 by slumpie

[maracuja]

please excuse my dumbness (it's just few days as I'm "playing" KSP), but is this thing doing same stuff as Active Texture Management?

[blowfish]

please excuse my dumbness (it's just few days as I'm "playing" KSP), but is this thing doing same stuff as Active Texture Management?

No, but it's related. ATM compresses all textures to DDS on game load. This allows DDS textures to be loaded directly from files.

[maracuja]

No, but it's related. ATM compresses all textures to DDS on game load. This allows DDS textures to be loaded directly from files.

thank you for reply! is there any point of using both of them or should i choose just one? if second, then which one is better?

[Ph34rb0t]

I've converted all the textures again from scratch, this time including the .mbms. (Thanks for the amended script, Urgante!)

Main loading time doesn't seem to have decreased, but the in-game loading times are even a wee bit faster than before.

RAM Usage with about 50 mods is 2.2 GB in OpenGL (KSP Win x86) mode.

thank you for reply! is there any point of using both of them or should i choose just one? if second, then which one is better?

This involves "manually" converting all files in the GameData folder and thus altering mods directly while ATM basically does it automatically for you upon game launch without affecting the source textures in the mod directory.

[maracuja]

so amirite that basically this tool is to control of which textures to be converted and i should be fine with ATM while i'm still not-so-deep in KSP?

[slumpie]

It's a matter of running a batch file once, but yeah stick with ATM for the beginning.