[WIN] KSP to DDS texture converter

[Lilleman]

Hi everyone! As you might know, Sarbian came up with a quite nice plugin after the release of KSP 0.25, allowing the game to load DDS files, instead of standard TGA and PNG files.

The main benefits is a faster loading time. REALLY faster. Twice as fast on certain setups... It should also avoid some memory issues due to TGA loading (that need confirmation, but I have good reasons to think this comes from DX9, and not from Unity).

The only "problem" with this plugin is, you have to convert all textures manually, using an external program, like nvdxt or Crunch (it's not a problem with the plugin actually, it works quite well). And having all textures converted properly can be quite tedious.

So, to allow more people to enjoy this, I'd like to share a tool to convert all those textures automatically.

The main reason I've made a new thread in this section of the forum is to keep Sarbian's thread clean from this development. There's too much exceptions to be added, and maybe too much bugs to be crushed, to keep it in the same thread as another plugin.

The principle is almost the same as Active Texture Management (who am I kidding? It's the exact same thing...), except textures are converted once and for all, instead of being converted every time you launch KSP. So, like ATM, it needs some exceptions to be defined in an external text file. As it's still a WIP, I didn't had time to test every popular mod and check if everything looks right. So it might need some updates for this tool to convert all mods properly. And, most of all, I need some feedbacks to update it and make it as complete as possible.

For now, only RGB8/ARGB8/XRGB8 textures are converted, into DXT1/DXT5(nm) formats. As those are usually the heaviest textures, it should already have a huge impact on loading time.

I tried to make this tool user-friendly, and I think you won't need any documentation to use it, but I still expect you to know what you are doing with it. Don't convert any folder or file unless you've made a backup. And keep an eye on the KSP's logs after converting a mod folder, to check if everything is loaded properly.

It use the .NET Framework (3.5), and DirectX9, so for now it's a tool for Windows only. If there's demand for it, I'll re-write it to use open-source, cross-platforms libraries.

You'll need DDSLoader to use converted textures in KSP, and some DirectX components, if the program crash at startup (as those components are required by some DX9 games, hopefully you won't have to download them).

DOWNLOAD. (Executable+sources)

Any feedback (positive or negative) will be appreciated, if you have any suggestion or new exception to add to the list, please let me know about it, and I'll update it.

Edited November 5, 2014 by Lilleman
06/11: v0.1.7:Updated ModsExceptions.txt

[lincourtl]

The principle is almost the same as Active Texture Management (who am I kidding? It's the exact same thing...), except textures are converted once and for all, instead of being converted every time you launch KSP.

Except you're not actually compressing the textures so they use less memory, right?

Also, as I understand it, ATM does not convert the textures every single time it runs either. It creates a TextureCache directory, and then loads from that.

Is it OK to run DDSLoader with ATM?

[slumpie]

It does use S3 texture compression.

Edited October 31, 2014 by slumpie

[NathanKell]

ATM loads and compresses textures each time it runs. So does KSP. The advantage here is doing that step ahead of time, so the texture can simply be loaded into memory directly, already compressed.

What ATM does cache is when it *rescales* a texture. But it caches them uncompressed, so it has to compress on load whether or not it has cached a rescaled texture.

And yes, it's fine to use ATM too, although ATM only touches textures in GameData which aren't DDS, and this converts them all to DDS, so...ATM won't be doing much of anything.

[dreadicon]

So, I can confirm that, at least with OpenGL or directX 11, this does something no other mod has been able to do. It makes my arrangement of mods playable. 'nuff said. Not saying this will work for everyone, but it worked for me.

some stats:

Directx9(Vanilla): crash @ ~50% progress (out of memory)

Directx9 + ATM: crash @ ~80% progress (out of memory)

Directx9 + ATM + LoD: crash @ ~90% progress (out of memory)

OpenGL + ATM: loads @ 3.6 GB, crashes on VAB/SPH (out of memory)

DirectX11 + ATM: Crash @ ~90% progress (out of memory)

DirectX9 + Full DDS conversion: Crash @ ~80% (out of memory)

DirectX11 + Full DDS conversion: Loads @ 2.6 GB; Runs generally stable with some texture artifacts (black or missing textures, mostly in stock assets). Peaks at 3.0 GB usage while running.

OpenGL +Full DDS conversion: Loads @ 2.3 GB; Runs generally stable with sparse texture artifacts (mostly things like toolbar icons or PartCatalogue icons). Peaks at around 2.6 GB

My GameData folder before conversion was 2.6 GB. After DDS application, it is 1.7 GB.

Definitely some glitches and oddities to work out here and there, but this is wonderful news for me and my KSP endeavors!

Edit: after more thorough investigation, looks like almost all the squad engines and about 4 of the tanks are broken by the conversion to DDS format. Looks like this is because, for whatever reason, Squad decided to put the textures in entirely different folders than the configs and models/meshes. perhaps some MM configs or an exemption could fix this? Will play with solutions as I have time. Just thought I would give a heads up. Also, the stock satellite/orbital icons seem to be broken, as do some mod icons for the toolbars as mentioned prior.

Edited October 31, 2014 by dreadicon

[slumpie]

Edit: after more thorough investigation, looks like almost all the squad engines and about 4 of the tanks are broken by the conversion to DDS format. Looks like this is because, for whatever reason, Squad decided to put the textures in entirely different folders than the configs and models/meshes.

No, the textures are in the same folders and conversion into dxt does not break them. At least not on my install, I used nvdxt and not this tool though.

Maybe you're using some sort of stock texture replacement mod.

Also, the stock satellite/orbital icons seem to be broken, as do some mod icons for the toolbars as mentioned prior.

Satellite/orbital icons don't break because of this, no idea what went wrong or what this tool does wrong.

He's aware of the icons and said he's going to fix it.

Edited October 31, 2014 by slumpie

[Lilleman]

@lincourtl: If you don't use the rescaling option, memory usage should be about the same. Maybe a little less since TGA textures seems to cause memory issues. But for PNG and MBM files, it should not make any difference. Except loading times, of course.

It's not compatible with ATM and thats why I've implemented a rescaling option, this will have a noticeable impact on memory usage.

@dreadicon: DX11 seems to have his own glitches, some textures are turning black, others pink... I would recommend to stick with DX9 or OpenGL, but if you have memory issues with a vanilla install, you should consider the rescaling option. For a vanilla install, I got everything converted properly, icons included. Can you tell me what's broken with tanks and engines? Or post a screenshot?

To avoid blurry icons, all images with a resolution lower than 64*64 are saved in an uncompressed format, that should keep them free from compression artifacts.

I'll test PartCatalog, and see what exceptions needs to be added. Any other particular mod that add an icon to the toolbar?

Thanks for the feedbacks!

Edit: PartCatalog seems to be looking for png files directly from the code. I've updated the link in OP, but if you've already downloaded it, you can just add those two lines to ModsExceptions.txt, so it'll skip the mod during a conversion:

//PartCatalog is looking for png, hard-coded
PartCatalog

Edit2: I forgot to mention: This tool will convert only mods/part textures, and will make them incompatible with ATM. But ATM still compress a lots more textures that this tool can't reach. So it's not totally incompatible. It just let less textures available for ATM to compress. I have some test to run, with ATM, LOD and this, to see if they work together. Normally, they should.

Edited October 31, 2014 by Lilleman

[lincourtl]

@lincourtl: If you don't use the rescaling option, memory usage should be about the same. Maybe a little less since TGA textures seems to cause memory issues. But for PNG and MBM files, it should not make any difference. Except loading times, of course.

It's not compatible with ATM and thats why I've implemented a rescaling option, this will have a noticeable impact on memory usage.

Merci! I'll try it out later.

[timmers_uk]

Hi,

Any chance that instead of deleting the original texture files, you could have an option to add a '.ddsified' suffix to the file names. That way the process is easier to revert (than copy pasta the whole gamedata dir and running a sync on it) than deleting, and (ideally) we could have an option on the tool to revert by deleting the .dds files and removing the '.ddsified' suffix from the file names.

I'd try and do this myself and submit a patch (if source was on github I could try for the whole submitting a pull request thing, but in any case I'm not down with the kids enough to be familiar with doing that anyway) ... but I've not got VB devtools setup on my machine, and haven't ever done any VB coding.

Thanks,

Timmers

[Lilleman]

Sure. I'm on it, I'll update the link later today.

Sorry about VB, I'm just used to code with it. I suppose I should rewrite this in C# for everyone's convinience, That might take some times, though.

Update: The backup option is here! It would have been better if I could have them stored in an archive, to spare some disk space, but it would have required an external library (I should really consider using those). For now, it just add the ".ddsified" suffix (at first I wanted to use a suffix like ".bak" or something more formal, but at least I'm sure this one will not be taken by another program), to original files, and allow to revert a folder to it's original state.

Edited October 31, 2014 by Lilleman

[dreadicon]

So, it was my mistake about satellite icons. What i was seeing was, in fact, the landing site icons from Kerbal Konstructs. They are replaced by SNES glitch screen patterns Not a biggie though.

Don't worry about dx11, I just included it for completeness. I am using OpenGL for all examples going forward (because dx9 gains no benefit from this RAM wise, far as I can tell)

I do want to point out, however, for whatever reason, there is a very, very clear RAM usage reduction resulting from this mod for me when running as OpenGL or DirectX 11. It reduced my RAM usage by about 1.3 GB over OpenGL + ATM alone. I am guessing it has to do with how dx9 keeps all textures backed in system RAM, with no alternative option. But that's just a wild guess.

Concerning the missing textures, I will get you a screenshot when I get home from work. Basically, the parts are 'invisible'. their tiles show up in the editor, they have stats, if selected they have nodes with which to attach them, but the models and their textures don't show up at all. I will also have to do some control testing on this to make sure it's not some other mod. Off the top of my head, the Poodle and Skipper engines were affected, along with most of the rest of them. Again, will do proper research this afternoon; the least I could do to repay is to offer good, well-researched feedback.

I can also confirm that ATM 'works', in so much as it doesn't crash the game, or make anything worse. I believe ATM does also reduce memory use for some things like EVE which cannot pre-convert to dds, but I don't have numbers to back that up.

An odd side effect seems to have been a massive speedup in my framerates overall (in-atmosphere flight still laggy, but that's just FAR). When in orbit or parked on the ground, or in the VAB/SPH, I can get double previous framerates (though lag spikes still abound, but that's not new). My load times are moderately improved, but load times were never my main concern.

Finally, I just wanted to thank you again for all your work on this (and, of course, sarbian for DDS Loader). This obliterated my RAM troubles when using OpenGL like no other method could. I had RAM to burn after this (which I promptly used on EVE and visual enhancement packs)

[timmers_uk]

Sure. I'm on it, I'll update the link later today.

Sorry about VB, I'm just used to code with it. I suppose I should rewrite this in C# for everyone's convinience, That might take some times, though.

Update: The backup option is here! It would have been better if I could have them stored in an archive, to spare some disk space, but it would have required an external library (I should really consider using those). For now, it just add the ".ddsified" suffix (at first I wanted to use a suffix like ".bak" or something more formal, but at least I'm sure this one will not be taken by another program), to original files, and allow to revert a folder to it's original state.

No problem on the VB, I'm not a hater, I just can't contribute much to VB stuff, and meanwhile ... awesome! thanks for the quick turnaround on the backup option. As soon as I've found which mod which is causing opening the revert popup to cause NPEs, I'll take the plunge and try this on my main career install

Timmers.

[Telanor]

It would have been better if I could have them stored in an archive, to spare some disk space

Don't bother, that's the point of png, compressing images. Compressing it again in a zip/rar isn't going to make it any smaller (it might even make it bigger)

[timmers_uk]

Couple more configs I've noted aren't coming through unscathed. I've disabled compression for the below in my install, maybe they could be compressed but the setting tweaked? :-

//KSP-AVC

KSP-AVC\Textures

//FinalFrontier

Nereid\FinalFrontier

//StageRecovery - its only toolbar icons

StageRecovery -nm -nr

//ContractsWindow - toolbar icon doesn't look right converted

Contracts Window\Textures\ContractsIcon.png

//Scansat - toolbar icon

SCANsat\Icons

[Lilleman]

Thanks for the reports!

I've found a "bug" with icons (okay it was my fault, sorry). Saving 24BPP icons to uncompressed (RGB24 in this case) format turned them into a grey mess. Now they're saved to ARGB32 format to avoid this.

I've also included exceptions for AVC, and for Stage Recovery. Thanks to the fix Final Frontier, Contracts Window and ScanSat don't need them. I also removed the one for USI Kolonization System, since they are converted properly now.

Edit: I realized I forgot to mention it, but Kerbal Konstructs icons should be fine too, now.

Edited November 1, 2014 by Lilleman

[Justin Kerbice]

I just wonder when someone finally make a proper tool for this after sarbian's DDSLoader plug-in.

Thanks Lilleman.

[starkline]

Hi, I also found a few things that needed tweaking. I didn't, however, try to fix any problems, I just added exceptions. I know a few were mentioned a post or two ago, but I'm too lazy to edit them out. Also, I think I recall seeing that the Orbital Logistics and Station Manager Icon from USI had been fixed to point to their new location. If not, they've moved to UmbraSpaceIndustries\Kolonization\MKS\Assets in the newest experimental builds (which likely means RoverDude's new release will retain the change).

//Ambient Light Adjustment
blizzy\AmbientLightAdjustment
//MechJeb Warp Helper Icon
MechJeb2\Icons\Warp_Helper.png
//NavUtils
KerbalScienceFoundation\NavInstruments
//SCANSat
SCANsat\Icons
//ConnectedLivingSpaces
ShipManifest\Plugins\
ConnectedLivingSpace\assets\IconOff_24.png
ConnectedLivingSpace\assets\IconOn_24.png
//PartAngleDisplay
PartAngleDisplay

Also, suggestion for the fantasic tool: it would be cool if the windows explorer folder picker included an entry field whereby it is possible to manually type or paste in the location. I know it's a small thing, but it would save from having to manually go through all the folder to pick the target.

Cheers!

starkline

Edited November 1, 2014 by starkline

[Lilleman]

Thanks! I've updated the link in OP to include the new exceptions (I've also added one for Interstellar).

If you already have downloaded the v0.1.5, you can just copy-paste this into your ModsExceptions.txt.

I'll try to implement the new folder browser dialog for the next release. Also, Sarbian is looking for a way to improve compatibility with mods (loading some textures as "read/write", instead of "read-only"), so hopefully some exceptions will be removed soon.

Edited November 1, 2014 by Lilleman

[Lilleman]

Up(date): v0.1.6 is out, with a proper folder browser dialog. This part comes from a bunch of code from here. This small part of the program actually have a different license from mine (GPL GNU v2 vs v3), so it comes in an external dll, and sources are separated. As it's mostly a copy-pasta, all credit for this should go to the original author.

Edited November 1, 2014 by Lilleman

[Telanor]

I was bored today, so I converted the converter to C# and uploaded it to github: https://github.com/Telanor/DDS4KSP. Release is here. I did some light testing of it and it seems to work properly, but there might be some issues I missed, VB is a strange language...

[Lilleman]

Damn, now everyone can see all the dumb comments I put in my code:D.

The only change I've made with this last release was the folder dialog, so if you used Ooki you shouldn't have to rewrite anything. I've just updated ModsExceptions this afternoon, so feel free to share the new one as well.

Can I use this as a base for future releases?

Edit: Thank you!

Edited November 1, 2014 by Lilleman

[Telanor]

The only change I've made with this last release was the folder dialog, so if you used Ooki you shouldn't have to rewrite anything. I've just updated ModsExceptions this afternoon, so feel free to share the new one as well.

Just updated and pushed the ModsExceptions changes.

Can I use this as a base for future releases?

Absolutely, please do. If you want me to add you as a collaborator I can, or you can just fork it, whatever works for you.

[Lilleman]

I have to learn a few things about Github before being added as a collaborator (I never used it), so for the time being I'll go with forking, but I'll let you know if there's any change. For now the main problem is indexed image formats (L8, P8, etc...), so I guess the flipping function won't stay, and the normal detection needs to be rewritten as well (it kind of works, but I'll check Shaw's converter sources to see how he does it). Thanks again!

[starkline]

One more change, as RoverDude just updated all USI components:

This exception moved:

remove: UmbraSpaceIndustries\AES\Spaces
add:    UmbraSpaceIndustries\ExpPack\AES\Spaces

Oh, and I noticed that the exceptions ended with Toolbar, did the USI icons get cut off accidentally?

//USI Kolonization System, those toolbar icons will look like a grey mess if converted
UmbraSpaceIndustries\Kolonization\MKS\Assets\OrbLogisticsIcon.png
UmbraSpaceIndustries\Kolonization\MKS\Assets\StationManager.png

Cheers!

[Lilleman]

Those last two are converted properly now, so their exceptions have been removed, I'll download the last version of USI pack to check if there's anything new to add (apart from this moved folder), and I'll update the OP later today.