[0.90] DDSLoader 1.9 (Mar 9) - Loads DDS Texture "Boringly fast" edition

[blowfish]

Maybe a good way to make DDSLoader more popular would be to write a platform independent script using something like python that uses the nvcompress and mbm tool to convert everything to dds.

I think in the long term mods should just distribute textures as DDS but in the short term that might be a good idea.

[Beduino]

I think in the long term mods should just distribute textures as DDS but in the short term that might be a good idea.

True that can be done, but what about stock textures, that can be optimized as well and there's still a need to install DDSLoader since it isn't native.

There's also the possibility of leaving the modders free to choose whatever image format they are comfortable with and have the script do the not so hard work, which is a better option in my opinion.

PS: I was checking that mbm2png tool and it seems so simple that it could be ported directly in python code..

heck i'll probably write the python script myself :/ there i said.

Edited October 27, 2014 by Beduino

[shaw]

I forked DDS converter out of my engine. Here's the standalone version:

https://github.com/ducakar/img2dds/releases

It can convert PNGs, JPEGs, TGAs, MBMs etc. to DDS format, supports compression of images with arbitrary dimensions, mipmap generation, and can automatically detect normal maps.

The release is compiled for Linux i686, Linux x86-64 and Win32 i686. If you don't use a very up-to-date Linux distribution, you may need to install newer libstdc++ from GCC 4.9 (it's probably in "gcc-libs" package or something like that).

For a model texture you should run it as

img2dds -vmcNs texure.png

-v: flip vertically

-m: generate mipmaps

-c: compress (DXT1 if non-transparent, DXT5 if transparent pixels are detected)

-N: set normal map flag if a PNG/JPEG/TGA "looks like" the "blue" (RGB=XYZ) normal map

-s: swap channels RGB -> GGGB if a normal map (as detected by -N)

-N and -s flags are ignored for MBM normal maps. MBM normal map flag is preserved in the final DDS.

UI textures should be compiled like

img2dds -v texture.png

or

img2dds -vc texture.png

if texture compression doesn't cause visible artefacts.

Edited October 27, 2014 by shaw

[Lilleman]

Here's one for windows. I don't release the source yet, because it's a uncommented mess and I'm still working on it, if you want to take a look at them just PM me.

If it crash at opening, you'll probably need to download some DX components. I've planned to switch to another library to avoid this, but some DX9 games required this to be installed, so maybe you already have it installed on your computer.

Please PM me directly for any feedbacks/suggestions/problems.

Edited October 28, 2014 by Lilleman
Updated link

[timmers_uk]

Just tried @Lilleman 's exe on my dev/test install (was stripped but I haven't yet stripped it since moving to 0.25), and holy balls the game loads so much faster after conversion! So much so, that I probably won't bother stripping all the stock parts out of my dev/test install, as the bootup is so buttery smooth and fast compared to standard.

Thanks Muchly, all.

Timmers

[BigD145]

Here's one for windows. I don't release the source yet, because it's a uncommented mess and I'm still working on it, if you want to take a look at them just PM me.

If it crash at opening, you'll probably need to download some DX components. I've planned to switch to another library to avoid this, but some DX9 games required this to be installed, so maybe you already have it installed on your computer.

Please PM me for directly feedbacks/suggestions/problems.

So is this dropped into KSP root and it just goes through everything?

[Lilleman]

So is this dropped into KSP root and it just goes through everything?

You can extract the archive anywhere, then run the program. You can open a single file, or a whole folder (usually, a mod folder or directly the GameData folder if you like to live dangerously). It should detect automatically the best format for conversion, but it will not convert anything by magic: the program can still fail to detect the correct format (or to save in a correct format), and in this case some arguments can be setted in a external file.

I already put some exceptions for kOS, Toolbar, EVE, and to avoid resizing of flags and agencies logos. (edit: also, reflections textures used by textureReplacer are skipped, because it still can't convert them properly).

It generate a log.txt file to keep track of things, so you can check what happened to your files.

Edited October 27, 2014 by Lilleman

[Crater]

Here's one for windows. I don't release the source yet, because it's a uncommented mess and I'm still working on it, if you want to take a look at them just PM me.

If it crash at opening, you'll probably need to download some DX components. I've planned to switch to another library to avoid this, but some DX9 games required this to be installed, so maybe you already have it installed on your computer.

Please PM me directly for any feedbacks/suggestions/problems.

Sorry, but I have a bug report for you.

Every time it hits a .png with the format it decribes as A8L8, it crashes out with an unhandled exception.

See the end of this message for details on invoking 
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.Runtime.InteropServices.ExternalException: A generic error occurred in GDI+.
   at System.Drawing.Image.Save(Stream stream, ImageCodecInfo encoder, EncoderParameters encoderParams)
   at System.Drawing.Image.Save(Stream stream, ImageFormat format)
   at DDS4KSP.IMGManager.gs_flip(GraphicsStream& gs)
   at DDS4KSP.IMGManager.convertFileToDDS(String srcFilePath, en_OutFormats format, Boolean b_DeleteOnSuccess, Boolean b_Uncompressed, Boolean b_GenerateMipmaps, Double dResizeRatio)
   at DDS4KSP.FolderLoader.processList(List`1& l, Boolean b_Uncompressed, String sFileFormat, Int32& iStep, Int32& FileCount, ProgressBar& PB, Label& lInfos, Double dResizeRatio)
   at DDS4KSP.FolderLoader.processFileLists(String sFolderPath, ProgressBar& PB, Label& lInfos, Boolean b_Uncompressed, Double dResizeRatio)
   at DDS4KSP.FRM_main.ExportAllToDDSToolStripMenuItem_Click(Object sender, EventArgs e)
   at System.Windows.Forms.ToolStripItem.RaiseEvent(Object key, EventArgs e)
   at System.Windows.Forms.ToolStripMenuItem.OnClick(EventArgs e)
   at System.Windows.Forms.ToolStripItem.HandleClick(EventArgs e)
   at System.Windows.Forms.ToolStripItem.HandleMouseUp(MouseEventArgs e)
   at System.Windows.Forms.ToolStripItem.FireEventInteractive(EventArgs e, ToolStripItemEventType met)
   at System.Windows.Forms.ToolStripItem.FireEvent(EventArgs e, ToolStripItemEventType met)
   at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEventArgs mea)
   at System.Windows.Forms.ToolStripDropDown.OnMouseUp(MouseEventArgs mea)
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   at System.Windows.Forms.ToolStrip.WndProc(Message& m)
   at System.Windows.Forms.ToolStripDropDown.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.8009 (FX35W81RTMGDR.050727-8000)
    CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll
----------------------------------------
DDS4KSP
    Assembly Version: 0.0.0.1
    Win32 Version: 0.0.0.1
    CodeBase: file:///C:/Dropbox/KSP%20DDS%20Test/GameData/DDS4KSP/DDS4KSP.exe
----------------------------------------
Microsoft.VisualBasic
    Assembly Version: 8.0.0.0
    Win32 Version: 8.0.50727.8007 (FX35W81RTMGDR.050727-8000)
    CodeBase: file:///C:/Windows/assembly/GAC_MSIL/Microsoft.VisualBasic/8.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.8009 (FX35W81RTMGDR.050727-8000)
    CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Windows.Forms
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.8008 (FX35W81RTMGDR.050727-8000)
    CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Windows.Forms/2.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.8007 (FX35W81RTMGDR.050727-8000)
    CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Drawing/2.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Runtime.Remoting
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.8003 (FX35W81RTMGDR.050727-8000)
    CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Runtime.Remoting/2.0.0.0__b77a5c561934e089/System.Runtime.Remoting.dll
----------------------------------------
System.Configuration
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.8007 (FX35W81RTMGDR.050727-8000)
    CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Configuration/2.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.8009 (FX35W81RTMGDR.050727-8000)
    CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Xml/2.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
Microsoft.DirectX.Direct3D
    Assembly Version: 1.0.2902.0
    Win32 Version: 9.05.132.0000
    CodeBase: file:///C:/Windows/assembly/GAC/Microsoft.DirectX.Direct3D/1.0.2902.0__31bf3856ad364e35/Microsoft.DirectX.Direct3D.dll
----------------------------------------
Microsoft.DirectX
    Assembly Version: 1.0.2902.0
    Win32 Version: 5.04.00.2904
    CodeBase: file:///C:/Windows/assembly/GAC/Microsoft.DirectX/1.0.2902.0__31bf3856ad364e35/Microsoft.DirectX.dll
----------------------------------------
Accessibility
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.8007 (FX35W81RTMGDR.050727-8000)
    CodeBase: file:///C:/Windows/assembly/GAC_MSIL/Accessibility/2.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
System.Core
    Assembly Version: 3.5.0.0
    Win32 Version: 3.5.30729.7903 built by: Win9Rel
    CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Core/3.5.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
Microsoft.DirectX.Direct3DX
    Assembly Version: 1.0.2902.0
    Win32 Version: 5.04.00.3900
    CodeBase: file:///C:/Windows/assembly/GAC/Microsoft.DirectX.Direct3DX/1.0.2902.0__31bf3856ad364e35/Microsoft.DirectX.Direct3DX.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
    <system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

The log shows all the successful ones, followed but a sudden stop, for example

LOG : Reading C:\Dropbox\KSP DDS Test\GameData\B9_Aerospace\Parts\Engine_SABRE_M\model_sabre_250_emissive.png, X8R8G8B8, 1024x1024
LOG : Converting C:\Dropbox\KSP DDS Test\GameData\B9_Aerospace\Parts\Engine_SABRE_M\model_sabre_250_emissive.png to C:\Dropbox\KSP DDS Test\GameData\B9_Aerospace\Parts\Engine_SABRE_M\model_sabre_250_emissive.dds
    Format : Dxt1, normalmap : False, res: 1024x1024
LOG : Reading C:\Dropbox\KSP DDS Test\GameData\B9_Aerospace\Parts\Engine_SABRE_Intake\model_sabre_intake_250.png, A8R8G8B8, 1024x1024
LOG : Converting C:\Dropbox\KSP DDS Test\GameData\B9_Aerospace\Parts\Engine_SABRE_Intake\model_sabre_intake_250.png to C:\Dropbox\KSP DDS Test\GameData\B9_Aerospace\Parts\Engine_SABRE_Intake\model_sabre_intake_250.dds
    Format : Dxt5, normalmap : False, res: 1024x1024
LOG : Reading C:\Dropbox\KSP DDS Test\GameData\B9_Aerospace\Parts\Engine_SABRE_Body\model_sabre_body.png, A8L8, 1024x1024

and then it's all over. Program dies, I add the failing texture to the exceptions, and it carries on.

So far, I've hit these A8L8 textures in B9, USI, MechJeb, Contracts Window, Procedural Wings, Kerbal Alarm Clock... and still processing. There are only tiny numbers of them... 20-30 out of 2500 textures converted so far.

System is Windows 8.1 x64, KSP is 0.25 32 bit, with a LOT of mods.

Edit : I'm using Open Folder / Export All, with all default settings for conversion

Edited October 27, 2014 by Crater

[Telanor]

I had to add the following exceptions to get Lilleman's program to work without crashing:

UmbraSpaceIndustries\MKS\Flags

TriggerTech\KerbalAlarmClock\Textures\img_Play.png

Squad\Parts\FuelTank\fuelTankX200-16\model001.png

MechJeb2\Icons\

BoulderCo\Clouds\Textures\kerbin1.png

The toolbar icon for Kerbal Engineer got corrupted in the conversion process and PartCatalog's button icons are invisible. I'm also getting non-stop spam in the console of "Couldn't load error Icon" with no helpful information at all about the error in the log. Startup times do seem to be significantly better, though I didn't time it.

Edit: While replacing KerbalEngineer's broken texture, I did a comparison of its ToolbarBackground texture and the dds one was significantly lower quality:

I ran the same texture through nvcompress manually and it matched the png perfectly, so there seems to be a quality issue somewhere.

Edit2: The console spam is gone after putting PartCatalog's icons back as pngs

Edited October 27, 2014 by Telanor

[Lilleman]

@Crater: I'll download these mods and take a look at it asap. I'll update the link then.

Thanks for the report. You can PM them directly to me, as they're not related to DDSLoader. I should probably open new thread, I'll see after some feedbacks.

I'll try to fix the A8L8 issue. I know the program has trouble with P8 image format too, you might want to be aware of that (edit: and, after a quick test, also L16...in fact, anything different from 24Bpp or 32Bpp).

The fuelTankX200-16 conversion should be working.... I'll look into this too...

Edited October 27, 2014 by Lilleman

[BigD145]

Just converting mbmm's from the Squad folder, with ATM installed, I went from 3.06GB RAM usage mbmm's on the title screen to 3.2GB with dds's from Lilleman's exe using all defaults. I think I'll stick with mbmm's as they seem to have a smaller RAM footprint.

[blowfish]

Just converting mbmm's from the Squad folder, with ATM installed, I went from 3.06GB RAM usage mbmm's on the title screen to 3.2GB with dds's from Lilleman's exe using all defaults. I think I'll stick with mbmm's as they seem to have a smaller RAM footprint.

Did you remember to delete the mbms afterward? If they're both present then they will both be loaded.

[BigD145]

Hmm. Lilleman's utility is supposed to do that by default and I did not check to make sure. Perhaps I'll try it again and see.

[juanml82]

Great, that converter seems to work fine. Kerbal Alarm Clock and KER should be added to the exception lists, though - KER's toolbar icon doesn't work and while KAC's blizzy toolbar icon works fine, all the icons inside the KAC window do not.

[Lilleman]

For a folder conversion, the checkbox for deleting files after conversion should be checked by default. But about memory usage, you will see better resuts if you do the other way around: keep mbm and convert TGAs/PNGs (for a vanilla install, there shouldn't be any problem for converting everything).

And about icons quality: when the program encounter a texture with a resolution (width or height) not equal to a multiple of 4, it try to resize it and transform it into a blurry mess (from the coder POV: I can't seems to choose the compression method for DXT, only filters, so I've got to experiment with other libraries and it might take a while)... I'll see if I can implement a workaround, but for now I recommend setting the "minimal size for conversion" to 48*48, so it won't try to convert most of icons and let them to their original formats.

I've made a quick fix, so it should not crash when it encounter a unknown format and put a error in the log instead. Got some tests to do, but I'll release it later today.

Edited October 28, 2014 by Lilleman

[slumpie]

Edit: While replacing KerbalEngineer's broken texture, I did a comparison of its ToolbarBackground texture and the dds one was significantly lower quality:

http://i.imgur.com/4QXY9LV.png

I ran the same texture through nvcompress manually and it matched the png perfectly, so there seems to be a quality issue somewhere.

That on the left is a mipmap.

[Lilleman]

Quick fix available! (For my converter, not for DDSLoader, of course).

New, amazing features:

-If it fails to load a texture, it will just skip the conversion and put a new entry in the log, instead of crashing. I'm still looking for a way to convert them (all 16Bpp images fail to load).

-If the resolution is lower than 64(width or height), it will always save the file in an uncompressed format. This should avoid blurry icons.

...And that's it for today...

I also updated the modsExceptions file, to include some new mods I've tested:

KerbalAlarmClock is not working, but I think it's looking for pngs, directly from the code.(Anyone can confirm if this is the case, or should I look into file formats?)

B9 can now be processed in a single batch (instead of crashing, yay!), but lots of textures are skipped...

No luck with RealSolarSystem...

Though, I got most parts pack converted without errors (KAS, KWRocketry, Novapunch, InfernalsRobotics, TACLifeSupport, Karbonite, and some more).

Again, you can PM me directly your error reports and logs if you encounter any problem.

Edited October 28, 2014 by Lilleman

[NathanKell]

Just a friendly note: if you're releasing anything that's compiled, you do need to supply (at least a link to) source with it.

(-your friendly neighborhood mod who is *very* interested in these developments...)

[Lilleman]

I wanted to wait a while to release the source, since it's probably one of the messiest code I've made and I'm still cleaning it (I've just some bugs with the way it handles arguments, I'm rewriting that whole module, lots of old debugging function are still here, etc...)... Please don't judge me...

Link updated. Licence is GNU GPL, I guess ("do what you want with, don't take me for responsible if it fails you", or something like that).

[timmers_uk]

Would there be anyone able to wrap the functionality of something like lilleman's converter into a plugin which runs at startup? Whilst its really handy, it'd be even handier if I didn't have to go back through and remember to re-run a converter program/script every time I muck about with my mods, update etc?

If no-one else is in a position to do that (or indeed if I don't get a 'dummy, that's what ATM does already' response - I don't think it does, as my DDSified install loads a lot faster than when running ATM), I could take a look at it, but I'm pretty noobish on plugin development, and really ought to get KeepFit fit for purpose first.

Similarly, is it possible for ddsloader to load ddses by precedence when there is both a .dds file and a corresponding .tga/.png/.mbm file? that way ddsifying my install wouldn't require deletion of all the existing image assets which makes updating more awakward?

Thanks for the work on this.

Timmers

Edited October 28, 2014 by timmers_uk
spelling and punctuation and grammar stuff

[slumpie]

I just discovered that DDS supports normal map flag. I noticed GIMP recognises DXT5nm-saved files apart from regular DXT5. It converts DXT5nm files back to RGB=XYZ when opened.

Which is incredibly annoying and probably the biggest anti-feature the dds plugin has.

Would there be anyone able to wrap the functionality of something like lilleman's converter into a plugin which runs at startup? Whilst its really handy, it'd be even handier if I didn't have to go back through and remember to re-run a converter program/script every time I muck about with my mods, update etc?

You mean it should convert and save the textures?

Edited October 28, 2014 by slumpie

[timmers_uk]

Which is incredibly annoying and probably the biggest anti-feature the dds plugin has.

You mean it should convert and save the textures?

Only those which haven't already been converted, yes - feel free to point out exactly how I'm a maroon for suggesting it (seriously!), but right now what I'm running separately from the KSP exe converts to DDS, then blows away any textures it finds - as I understand it, if I don't blow away the pre-DDS textures, then I'll end up with the worst of both worlds, by having KSP try to load both?

Given some of the mods I use update regularly, every time I update I have to remember to re-run the convert + destroy pass.

Clearly its not the end of the world (and I'm also running some comparison boots to quantify exactly how much time I'm really saving here), I'm just trying to work out the most streamlined workflow, especially if I'm getting back into mod dev where one thing that killed my enthusiasm for it was the debug cycle.

Timmers

[shaw]

Which is incredibly annoying and probably the biggest anti-feature the dds plugin has.

Why? I prevents a bunch of warning messages during loading and I see no downsides in this.

[shaw]

Would there be anyone able to wrap the functionality of something like lilleman's converter into a plugin which runs at startup? Whilst its really handy, it'd be even handier if I didn't have to go back through and remember to re-run a converter program/script every time I muck about with my mods, update etc?

If one wants to make (platform-independent) plugin with such functionality, one needs C# implementation of S3TC/DXTn texture compression algorithm. And that's nonexistent AFAIK. One could convert an existing C++ library, like squish, to C# but that's really a lot of work.

[Sean Mirrsen]

If one wants to make (platform-independent) plugin with such functionality, one needs C# implementation of S3TC/DXTn texture compression algorithm. And that's nonexistent AFAIK. One could convert an existing C++ library, like squish, to C# but that's really a lot of work.

Last time I was faced with a problem like that, I just used commandline NVidia Texture Tools. (NVidia Texture Tools 2, to be precise) Works pretty much perfectly.

I had a program that needed to create a DXT-compressed texture (in XNB format, for an XNA-based game), but I had no luck finding a working compression/decompression algorithm. So instead I just had the program convert images to TGA, and then call the Texture Tools from the command line, then wrote the resulting DDS texture file and wrote the contents into the XNB. It's quite reliable, if you set it up right.

Edited October 28, 2014 by Sean Mirrsen