[WIP] Loading textures only as required

[raxo2222]

Apparently b9 causes crashes, here is log: http://www73.zippyshare.com/v/24000377/file.html (incorrect parameter or something about b9 part).

It happens only in VAB/SPH.

Edit: Another "incorrect parameter" crash: http://www1.zippyshare.com/v/86110506/file.html

This time Taurus derped.

I still have like 900 MB free on these crashes.

I set LoD compression to "False", as I have ATM, that compresses textures. It crashed instantly in VAB, when I set resize tecture scale to 4.

Edited May 27, 2014 by raxo2222

[Kolago]

After converting all .tga files to .png in Gamedata and editing all .mu files to the new .png files LoadOnDemand runs without crashs and the game needs about 400MB less memory!

Only LLL Greenhouse has a problem with this, the windows are fogged and not clear any more.

Edited May 30, 2014 by Kolago

[BananaDealer]

After converting all .tga files to .png in Gamedata and editing all .mu files to the new .png files LoadOnDemand runs without crashs and the game needs about 400MB less memory!

Only LLL Greenhouse has a problem with this, the windows are fogged and not clear any more.

Kind sir, would you mayhaps share your ways?

[SpacedInvader]

Kind sir, would you mayhaps share your ways?

I might be crazy, but I think Faark said the next release will handle tga files so this will be unnecessary.

EDIT: I did find this, which looks like it does that whole process in one package. I still think Faark said the next rev will cover files like tga and mbm, but I plan on trying this in the mean time as it may be some time before 2.3 is out.

Edited May 30, 2014 by SpacedInvader

[Alewx]

I might be crazy, but I think Faark said the next release will handle tga files so this will be unnecessary.

But Faark also has a RL and so the next version might take days, weeks, months....(I just don't want to think further )

So every bit of memory saving is worth it.

[SpacedInvader]

But Faark also has a RL and so the next version might take days, weeks, months....(I just don't want to think further )

So every bit of memory saving is worth it.

Look above this post, I just edited my last post.

[SpacedInvader]

I guess it won't work to convert tga to png, though there are lots of available options for this. I think the biggest problem though is that the exe has to be in the directory of the files for conversion, which means if you're like me you're talking about 1500+ files in 1500+ folders that all need the exe run on them individually. Maybe someone who's way better at scripting than I can write a batch file that could handle the job without too much trouble?

[Alewx]

Look above this post, I just edited my last post.

Cool thanks.

[SpacedInvader]

So I did a little testing, and you don't need to update the .mu files at all, just converting the files from mbm / tga to png seems to work just fine. Still not sure if it will affect performance at this point... haven't figured out how to just convert them all in one go.

[SebFierce]

Most likely you can use the BatchConversion of IrfanView to do it.

Does it improve the stability? My game crashes fairly often and I'd like to improve on that

[SpacedInvader]

Most likely you can use the BatchConversion of IrfanView to do it.

Does it improve the stability? My game crashes fairly often and I'd like to improve on that

I did a batch convert with xnview. Game seems to work ok, but with that said, I'm actually getting more memory usage than before. Initial entry into the VAB usually had me sitting at 2.8gb, now I'm around 3gb. Maybe setting the correct associations in the .mu files is the trick there, but it would need a batch file to run the mueditor from that converter I linked a few posts ago in each and every folder with a model in it. I have 1400 .mu files in my gamedata directory and have very little desire to manually run that program that many times.

EDIT: I'm hoping someone else has the skills to write this as I definitely am far out of practice for a complicated thing like this.

Edited May 30, 2014 by SpacedInvader

[Faark]

Okay, the transparency issue is definitively caused by the TGA loader LOD uses. Its this 3rd party code and should be able to handle transparency, though. No idea why it always detects the img format as rgb without alpha. Would love to replace it, but there doesn't seem much to choose from...

So yes, for now you have to convert TGAs with alpha channel to e.g. png or make LOD ignore them via its config. Btw, when converting make sure KSP doesn't load both the PNG and TGA version! This would explain an increase in used memory (if your lucky LOD thinks it needs both and "only" prepares 32x32 versions of them). Deleting or renaming them to sth like "*.unusedtga" should do the job.

It shouldn't be necessary to touch anything else (mu or mbm files) for the transparency issue and shouldn't really save memory as well.

Only LLL Greenhouse has a problem with this, the windows are fogged and not clear any more.

Are you talking about the 2x1 greenhouse containing green leafs? Works fine for me once i got rid of its 3 tag's in favor of png's.

Anyways, could you guys please test this version and report and any changes you notice? Especially stuff like slower loading times for high res textures or other regressions...

[SpacedInvader]

Okay, the transparency issue is definitively caused by the TGA loader LOD uses. Its this 3rd party code and should be able to handle transparency, though. No idea why it always detects the img format as rgb without alpha. Would love to replace it, but there doesn't seem much to choose from...

So yes, for now you have to convert TGAs with alpha channel to e.g. png or make LOD ignore them via its config. Btw, when converting make sure KSP doesn't load both the PNG and TGA version! This would explain an increase in used memory (if your lucky LOD thinks it needs both and "only" prepares 32x32 versions of them). Deleting or renaming them to sth like "*.unusedtga" should do the job.

It shouldn't be necessary to touch anything else (mu or mbm files) for the transparency issue and shouldn't really save memory as well.

Are you talking about the 2x1 greenhouse containing green leafs? Works fine for me once i got rid of its 3 tag's in favor of png's.

Anyways, could you guys please test this version and report and any changes you notice? Especially stuff like slower loading times for high res textures or other regressions...

Actually, I had backed up a copy of my GameData folder elsewhere and then set the batch process to delete the original and replace it with the new PNG. Essentially, after the process there were absolutely no tga's left in my entire GameData folder. The 3gb use seemed to vary quite a bit less (without the conversion, memory would fluctuate all over the place, but usually stay under 3gb), but it also didn't seem to improve stability or performance. If you're saying that no changes should need to be made to the .mu file, then I'm going to say that means going all png isn't the way to go.

Also.... Yay for a new relase!

EDIT: Perhaps it would be better to go the other direction? Convert all the parts that don't have transparency issues to TGA?

Edited May 30, 2014 by SpacedInvader

[SpacedInvader]

First experience with new release:

Very slow first prep (~5 mins for 2108 textures)

Extremely fast prep afterward (~10sec for the same 2108 textures)

Immediate CTD on entering VAB (3x already)

EDIT: Also loads to main menu with about 200MB more memory use than last version (2.9GB this version, 2.7GB last version).

EDIT2: What does this do? "DontLoadEditorCatalogParts = False"

EDIT3: VAB crash is definitely OOM error. Entering VAB while tracking memory use shows immediate increase of 250MB and then climbing to crash.

Edited May 30, 2014 by SpacedInvader

[Alewx]

Anyways, could you guys please test this version and report and any changes you notice? Especially stuff like slower loading times for high res textures or other regressions...

Roger, will do it asap as I get home.

EDIT:

@SpaceInvader, like it says it should avoid the loading of the vab catalog parts if set to true. I except it will Keep the parts in the catalog with 32*32 Images to avoid early crashes and smaller loading times in the VAB.

Edited May 30, 2014 by Alewx

[SpacedInvader]

New update:

Setting DontLoadEditorCatalogParts = True solved CTD issue and has left VAB browsing memory use rock solid at whatever is actually loaded in the hangar. Still seems to want more memory on initial loading, but on entering a save, it dropped down to 2.7GB again, and then actually stayed there on entering the VAB, even while browsing pages, until I loaded a saved craft, at which time it went up to 2.85GB and stayed again. Seems to be the answer many of us have been looking for.

[Alewx]

New update:

Setting DontLoadEditorCatalogParts = True solved CTD issue and has left VAB browsing memory use rock solid at whatever is actually loaded in the hangar. Still seems to want more memory on initial loading, but on entering a save, it dropped down to 2.7GB again, and then actually stayed there on entering the VAB, even while browsing pages, until I loaded a saved craft, at which time it went up to 2.85GB and stayed again. Seems to be the answer many of us have been looking for.

Looks like what I was guessing, the parts in the catalog are smaller scaled versions of the regular parts and use the complete textures.

Sounds like it will be a really nice evening from what you discovered in this short time

Could you still provide a little "how to" for the tga->png conversion?

[Faark]

First experience with new release:

Very slow first prep (~5 mins for 2108 textures)

Extremely fast prep afterward (~10sec for the same 2108 textures)

Immediate CTD on entering VAB (3x already)

EDIT: Also loads to main menu with about 200MB more memory use than last version (2.9GB this version, 2.7GB last version).

EDIT3: VAB crash is definitely OOM error. Entering VAB while tracking memory use shows immediate increase of 250MB and then climbing to crash.

Thanks. This new version keeps some memory reserved for disk IO, thus some increase is fine. But it should only be between 32mb and your max texture file size (even though about 2x max size for faster loading, but kinda happy now that i didn't). I'll investigate on my test install, but some logs would be nice as well, especially those where DontLoad = false did crash the game. Please also re-download LOD from the link above, since it now also logs disk buffer size changes.

@TGAs

The slow / manual / non-batch way is to open the problematic TGA with e.g. GIMP and export as png (default settings should work).

To make LOD ignore a particular file, open GameData\LoadOnDemand\LoadOnDemand.cfg, find the tga file and add "SkipImage = True" to the files configs.

*edit*

Okay, i tried my smaller test install:

v2, 2nd start: 12sec, 0g915 w, 1g165 c
v3, 1st start: 43sec, 1g182 w, 1g275 c
v3. 2nd start: ~3sec, 1g082 w, 1g197 c
(w = "Memory (private working set)", c = "Commit size")

v2, 1st start: 66sec, 1g126 w, 1g212 c

Its interesting that the "working set" size with v3 is mentionable larger while the commit size is kinda the same (+the expected 32mb). Commit size is what is important in terms of 32bit oom crashes, so i hope we are fine. The task mgr doesn't show this column by default, btw.

Edited May 30, 2014 by Faark

[raxo2222]

Hmm it uses 1800 MB for me on menu.

And it goes to 1950 MB after loading save.

It sits at 2050 MB, while in VAB.

It went up to 2400 MB after loading craft into VAB.

And still around 2400, when I loaded different craft. I have 50 or so mods . 600 MB free now.

Exiting to spaceport lowers memory usage to 2150 MB.

I have don't load parts in editor on True. I also I have ATM on Agressive, that compresses textures to half of their res. My graphic settings are on maximum.

Also I would crash in VAB, when trying to have browser and game running at once - Firefox is using 300 MB, and anti-virus another 150 MB.

So for now its working fine - like your mod entered Beta stage

Imma see what happens, if I enable compression for LOD.

Nothing changed, but uses 30 MB less in VAB - sits around 2020 MB.

Hmm it doesn't do anything - after loading craft is 2400 MB.

Edited May 30, 2014 by raxo2222

[Faark]

Imma see what happens, if I enable compression for LOD.

Nothing changed, but uses 30 MB less in VAB - sits around 2020 MB.

Hmm it doesn't do anything - after loading craft is 2400 MB.

Anything else would be surprising since texture compression isn't yet implemented. I just added those compression flag since this feature was already planned when i created the config in the first place.

[raxo2222]

Anything else would be surprising since texture compression isn't yet implemented. I just added those compression flag since this feature was already planned when i created the config in the first place.

Hmm weird, that option was crashing me on V2.2 was that bug?

[Alewx]

Yeah success, tested the new v3 and got an CtD from the VAB with OoM and not with crashing texture problem!

[Kolago]

Kind sir, would you mayhaps share your ways?

Sure, I have done this with a batch image converter (Pixillion Image Converter) for about 600 tga files.

The size of the png files was in most cases 50% or less of the tga source file.

I batch converted the .mu files with UltraEdit from .tga to .png directly (search & replace).

Then I delete all the .tga files after that.

The conversion wend down in 5 minutes with these tools and KSP on my RAM-Disk. Before that I made some test runs with a single mod directory.

Make a backup before starting!

KSP memory usage was 2800MB before in the VAB and 2400MB after TGA conversion.

Edited May 30, 2014 by Kolago

[Alewx]

OK I now noticed a quite strange bug, and have no idea what this might be.

If I delete the complete LoD Folder and start the game everything works perfect. then in the VAB the game crashes, and as I restart it, LoD seems to be stuck as I arrive in the main menue.

Normally it counts up the completed textures, but now it gets stuck at 0 or 3 or 10 and then nothing again ever happens, while the game can be used regulary, just without part textures.

Because I know logs are the only thing that matters, I have a little surprise for you

LoadOnDemand.log

[SpacedInvader]

Looks like what I was guessing, the parts in the catalog are smaller scaled versions of the regular parts and use the complete textures.

Sounds like it will be a really nice evening from what you discovered in this short time

Could you still provide a little "how to" for the tga->png conversion?

With XnView, it was as simple as going to tools > batch processing, then adding the gamedata folder, setting it to use original path, delete the original file, and output type png. Make sure to save a backup of your gamedata folder. Took about 5 mins on my SSD.

Thanks. This new version keeps some memory reserved for disk IO, thus some increase is fine. But it should only be between 32mb and your max texture file size (even though about 2x max size for faster loading, but kinda happy now that i didn't). I'll investigate on my test install, but some logs would be nice as well, especially those where DontLoad = false did crash the game. Please also re-download LOD from the link above, since it now also logs disk buffer size changes.

@TGAs

The slow / manual / non-batch way is to open the problematic TGA with e.g. GIMP and export as png (default settings should work).

To make LOD ignore a particular file, open GameData\LoadOnDemand\LoadOnDemand.cfg, find the tga file and add "SkipImage = True" to the files configs.

*edit*

Okay, i tried my smaller test install:

v2, 2nd start: 12sec, 0g915 w, 1g165 c
v3, 1st start: 43sec, 1g182 w, 1g275 c
v3. 2nd start: ~3sec, 1g082 w, 1g197 c
(w = "Memory (private working set)", c = "Commit size")

v2, 1st start: 66sec, 1g126 w, 1g212 c

Its interesting that the "working set" size with v3 is mentionable larger while the commit size is kinda the same (+the expected 32mb). Commit size is what is important in terms of 32bit oom crashes, so i hope we are fine. The task mgr doesn't show this column by default, btw.

I will recrash my game with the flag set to false for you . As for which type of file to go for, would you say its better to use TGA or PNG? With compression, PNG can be much smaller, but I'm not sure if KSP takes compression into account when dealing with its textures, so I'm not sure if it helps or hinders.

Yeah success, tested the new v3 and got an CtD from the VAB with OoM and not with crashing texture problem!

I'm guessing you mean you got the same results as I did after setting the flag to true?

Sure, I have done this with a batch image converter (Pixillion Image Converter) for about 600 tga files.

The size of the png files was in most cases 50% or less of the tga source file.

I batch converted the .mu files with UltraEdit from .tga to .png directly (search & replace).

Then I delete all the .tga files after that.

The conversion wend down in 5 minutes with these tools and KSP on my RAM-Disk. Before that I made some test runs with a single mod directory.

Make a backup before starting!

KSP memory usage was 2800MB before in the VAB and 2400MB after TGA conversion.

I will have to give this a try. I have a gut feeling that not changing the file reference within the .mu files may be taking extra memory for some reason.

OK I now noticed a quite strange bug, and have no idea what this might be.

If I delete the complete LoD Folder and start the game everything works perfect. then in the VAB the game crashes, and as I restart it, LoD seems to be stuck as I arrive in the main menue.

Normally it counts up the completed textures, but now it gets stuck at 0 or 3 or 10 and then nothing again ever happens, while the game can be used regulary, just without part textures.

Because I know logs are the only thing that matters, I have a little surprise for you

LoadOnDemand.log

Are you sure this isn't just the DLL rebuilding all of the cache thumbnails? this process took quite a while when I did it and it seems to hang LOD a lot...