[WIP] Loading textures only as required

[SpacedInvader]

Ok, here are the crash logs from the VAB OOM crash with the flag set to false.

[Alewx]

With XnView, it was as simple as going to tools > batch processing, then adding the gamedata folder, setting it to use original path, delete the original file, and output type png. Make sure to save a backup of your gamedata folder. Took about 5 mins on my SSD.

I will recrash my game with the flag set to false for you . As for which type of file to go for, would you say its better to use TGA or PNG? With compression, PNG can be much smaller, but I'm not sure if KSP takes compression into account when dealing with its textures, so I'm not sure if it helps or hinders.

I'm guessing you mean you got the same results as I did after setting the flag to true?

I will have to give this a try. I have a gut feeling that not changing the file reference within the .mu files may be taking extra memory for some reason.

Are you sure this isn't just the DLL rebuilding all of the cache thumbnails? this process took quite a while when I did it and it seems to hang LOD a lot...

1. Yes got the exact same result as you. its awesome see it improving

2. Acutally I was afk for dinner and the DLL didn't load anything, and also it shouldn't reprocess all Images again, that happend already in the run before.

[SpacedInvader]

1. Yes got the exact same result as you. its awesome see it improving

2. Acutally I was afk for dinner and the DLL didn't load anything, and also it shouldn't reprocess all Images again, that happend already in the run before.

Have you tried deleting it again? Was your first / second run after the delete with "DontLoadEditorCatalogParts = True"? I'm thinking logs would be in order here. One hypothesis is that if the flag is set to true, it doesn't properly build all of those cache files, though I'm not sure why it would do that.

[SpacedInvader]

Sure, I have done this with a batch image converter (Pixillion Image Converter) for about 600 tga files.

The size of the png files was in most cases 50% or less of the tga source file.

I batch converted the .mu files with UltraEdit from .tga to .png directly (search & replace).

Then I delete all the .tga files after that.

The conversion wend down in 5 minutes with these tools and KSP on my RAM-Disk. Before that I made some test runs with a single mod directory.

Make a backup before starting!

KSP memory usage was 2800MB before in the VAB and 2400MB after TGA conversion.

So I replicated your process and the result was the same for me as it was before: Working was about 3gb, and now that I'm tracking it, commit was about 3.15gb, so I'm getting worse memory usage with all PNG. I'm going to try now the reverse, setting everything to TGA and see what I get for memory usage.

[Alewx]

Have you tried deleting it again? Was your first / second run after the delete with "DontLoadEditorCatalogParts = True"? I'm thinking logs would be in order here. One hypothesis is that if the flag is set to true, it doesn't properly build all of those cache files, though I'm not sure why it would do that.

Yes I deleted them all again after every lauch of the game, and yeah you are right I had them all the time set to true, I try it with false now.

EDIT: yep same behaviout as before with the DontLoadEditorCatalogParts = False, so no it is not that what is causing it from getting stuck.

Edited May 30, 2014 by Alewx

[SpacedInvader]

I tried going the other way to all TGA and three interesting things happend:

1: Transparency issues were gone for some reason.

2: The game still used more memory. This time it was about 2.9gb working and 3.1gb commit across the board.

3: Working in the VAB became choppy and slow, I would say my fps dropped to about 10-15.

Basically, it seems that converting all textures to either PNG or TGA is not the right way to go, at least in my case. Both directions increased overall memory usage, and neither seemed to improve stability.

[SpacedInvader]

Yes I deleted them all again after every lauch of the game, and yeah you are right I had them all the time set to true, I try it with false now.

EDIT: yep same behaviout as before with the DontLoadEditorCatalogParts = False, so no it is not that what is causing it from getting stuck.

Hmm, so it worked once and then stopped? That's strange... have you made any other changes besides the new LOD DLL? I experienced some strange behavior with LOD after I made a typo in a Module Manager config a while back...

[Alewx]

Nope, new LoD was the only change I made.

Currently I try to get some parts welded, but these mods were already installed before I got the new LoD.

But yes the current thing is it works once and then it stops, and it does not matter if I change something in the config.

[SpacedInvader]

Nope, new LoD was the only change I made.

Currently I try to get some parts welded, but these mods were already installed before I got the new LoD.

But yes the current thing is it works once and then it stops, and it does not matter if I change something in the config.

Maybe Faark has a better answer... I can only say it seems to be working great for me. Perhaps re-downloading the dll (he made some more changes to that upload since he released it) might fix the issue?

[SpacedInvader]

New update:

This version seems to use much more memory while in play. A rocket that used to have a working memory of about 2.8gb on the pad now uses about 3.3gb with a commit of 3.4gb. It is fairly complex with about 110 parts, but this could mean trouble for complex structures like stations.

My question to Faark now would be if its possible to use the old method while in play, but the new method while in the VAB / SPH?

EDIT: I think I should state for the record that I, at least, am far more worried about memory management than loading speed. If I could have things my way, I'd love to have the old memory management model with much less memory usage, but longer load times, combined with the option to disable the loading of textures while in the editor. IMO this would buy me an extra ~400MB of memory, while also cutting out the texture loading crashes that were so common for everyone.

Edited May 30, 2014 by SpacedInvader

[Faark]

Ok, here are the crash logs from the VAB OOM crash with the flag set to false.

Ty, will be interesting to compare that to my test install.

OK I now noticed a quite strange bug, and have no idea what this might be.

If I delete the complete LoD Folder and start the game everything works perfect. then in the VAB the game crashes, and as I restart it, LoD seems to be stuck as I arrive in the main menue.

Normally it counts up the completed textures, but now it gets stuck at 0 or 3 or 10 and then nothing again ever happens, while the game can be used regulary, just without part textures.

Because I know logs are the only thing that matters, I have a little surprise for you

LoadOnDemand.log

There is some mod messing around with LOD textures (try to remove TR for testing, disabled its compression and mipmap feature, etc). v2 could kinda handle this situation but i apparently missed a bug that prevents v3 from kinda-recovering. Bad in both cases, since it forces LOD to re-create its cache...

1: Transparency issues were gone for some reason.

You re-created those tga's from the png files using some program? My guess is that the original files don't fully follow TGAs standard (and/or the loader doesn't), but the program did a better [/ different] job creating TGAs.

[..]

Ofc not. I'll look into it.

[SpacedInvader]

You re-created those tga's from the png files using some program? My guess is that the original files don't fully follow TGAs standard (and/or the loader doesn't), but the program did a better [/ different] job creating TGAs.

Ofc not. I'll look into it.

Ofc not? I'm guessing that means of course not, but of course not what? Sorry...

[Alewx]

There is some mod messing around with LOD textures (try to remove TR for testing, disabled its compression and mipmap feature, etc). v2 could kinda handle this situation but i apparently missed a bug that prevents v3 from kinda-recovering. Bad in both cases, since it forces LOD to re-create its cache...

Ok I will test that tomorrow after wakeup.

Good night guys.

[Alewx]

Tested it with TR installed but disabled mipmaps and compression, and oh wonder now LoD works perfectly. now I will again test this will partcatalog disabled and TR enabled and then report back.

EDIT: First tests in VAB with some crafts did keep me at around 2.8 - 3GB RAM that is already 150MB saved to previous Version. no Crash, not trouble, no problems currently.

EDIT2: It is so awesome mega monster super fast with loading the textures, it is amazing, just 2 to 3 seconds and then ready to play.

Edited May 31, 2014 by Alewx

[frash23]

Hello,

I tried this mod, but it gives me the error "Currently only supports Direct3D 9.0c".

I use linux, so I know that the game is run using mono, but I have no idea on the Direct3D part.

Is there any hope for me?

[Faark]

I use linux, so I know that the game is run using mono, but I have no idea on the Direct3D part.

Hi frash,

this mod works on the layer below the unity engine and directly interacts with DirectX. I'm afraid the Linux version uses OpenGL, what this mod currently doesn't support and probably won't for quite a while, thus making this mod Windows only for now.

@Alewx

Found and fixed the issue causing the hang. Still not recommended to let TR mess around with LOD textures for the reasons mentioned, ofc.

@SpacedInvader

Did you ever buy a car that didn't work and then asked you mechanics to put the case of it around your old one, since the new one looks that nice? All changes i made to LOD i made for a reason and ofc won't support the old version. But i can't imagine what change could result in a large increase of memory required and thus as stated will look into the issue you have reported. That means I will try to reproduce the issue, find the cause and fix it. I didn't yet had time to do so, but as usually with mod issues you can help by giving fool-proof instructions on how to reproduce the issue. Preferably with a stock + lod install (or the instructions including what other mods are absolutely necessary to reproduce)

[SpacedInvader]

@SpacedInvader

Did you ever buy a car that didn't work and then asked you mechanics to put the case of it around your old one, since the new one looks that nice? All changes i made to LOD i made for a reason and ofc won't support the old version. But i can't imagine what change could result in a large increase of memory required and thus as stated will look into the issue you have reported. That means I will try to reproduce the issue, find the cause and fix it. I didn't yet had time to do so, but as usually with mod issues you can help by giving fool-proof instructions on how to reproduce the issue. Preferably with a stock + lod install (or the instructions including what other mods are absolutely necessary to reproduce)

Unfortunately, I've got 80ish mods installed, so it could be LOD interacting with any one of them. That being said, there is a clear increase of 500MB in working memory (no clue about commit just yet) while in play mode with a complex craft from the previous version to this. I'm definitely getting much better results than with v2.2 as each added part only increases memory by a few MB, allowing me to comfortably work on my ships without the constant fear of crashing, but I have concerns that in game crashes while in play mode are going to become more common... I've already had one, just didn't think to grab the logs because I wanted to get back to where I was ASAP. That being said, would a non-crash log help at all in this situation, perhaps a before and after of going from VAB to pad?

As for the changes you've made, I'm not unhappy with them at all, I was just throwing out some ideas as well as voicing my personal opinion that I'd rather have a game that takes longer to load, but never crashes than a game that loads faster, yet rides closer to the edge on memory usage, leaving the potential open for more crashes. I'd still by far rather work with your newest version than the older version, as I was able to build and test rockets all night and the only crash I experienced was during play during atmospheric entry, so reloading brought me almost back to the same place. With the previous version, I could easily expect a crash every 5-30 minutes in the VAB, leading to multiple long reloads (80 mods takes about 5min from desktop to loaded save and I've got KSP on a SSD) before I ever got to space.

Edited May 31, 2014 by SpacedInvader

[frash23]

Hi frash,

this mod works on the layer below the unity engine and directly interacts with DirectX. I'm afraid the Linux version uses OpenGL, what this mod currently doesn't support and probably won't for quite a while, thus making this mod Windows only for now.

I see.

If it bothers me too much, I'll just install wine and run it through that.

Edited May 31, 2014 by frash23

[smart013]

Hi, I really try to get this mod working for me but whatever I do I get error along the lines of this one:

[LOG 16:56:11.881] AssemblyLoader: KSPAssembly 'KSPInterstellar' V10.3

[LOG 16:56:11.882] AssemblyLoader: KSPAssemblyDependency 'OpenResourceSystem' V1.1

[LOG 16:56:11.885] AssemblyLoader: Loading assemblies

[LOG 16:56:12.035] AddonLoader: Instantiating addon 'LegacyInterface' from assembly 'LoadOnDemand'

[LOG 16:56:12.037] AddonLoader: Instantiating addon 'ActivityGUI' from assembly 'LoadOnDemand'

[LOG 16:56:12.039] AddonLoader: Instantiating addon 'DevStuff' from assembly 'LoadOnDemand'

[LOG 16:56:12.044] LoadOnDemand: No config found, starting with defaults.

[LOG 16:56:12.045] LoadOnDemand: Todo19: Consider a mechanism to remove old/unsued cached files! (both dead refs and unused cfgs)

[LOG 16:56:12.046] AddonLoader: Instantiating addon 'Startup' from assembly 'LoadOnDemand'

[LOG 16:56:12.060] LoadOnDemand: Device: Direct3D 9.0c [nvd3dum.dll 9.18.13.3788]

[LOG 16:56:12.091] LoadOnDemand: ERROR: Native load failed with error code 126.

[EXC 16:56:12.101] AggregateException: Failed to create NativeBridge, even legacy mode!Contained exceptions:

---------------------------------------

System.Exception: Failed to load native library: ---> System.Runtime.InteropServices.COMException (0x8007007e):

at System.Runtime.InteropServices.Marshal.ThrowExceptionForHR (Int32 errorCode) [0x00000] in <filename unknown>:0

at LoadOnDemand.UnmanagedLibrary..ctor (System.String fileName) [0x00000] in <filename unknown>:0

--- End of inner exception stack trace ---

at LoadOnDemand.UnmanagedLibrary..ctor (System.String fileName) [0x00000] in <filename unknown>:0

at LoadOnDemand.NativeBridge.SetupNativeBridge () [0x00000] in <filename unknown>:0

---------------------------------------

System.NotImplementedException: Legacy mode isn't even close to a working state!

at LoadOnDemand.NativeBridge.SetupNativeBridge () [0x00000] in <filename unknown>:0

I run KSP on Win7/64 with Working DirectX9/11, .NET Framework (at least working everyelse than LOD) and vcredist_x86.exe installed. Last Time I checked Direct3D 9.0c [nvd3dum.dll 9.18.13.3788] was already installed in System32 (C:\Windows) and SysWOW64 (C:\Windows). I dont know if this is important but a filesearch found lots of other Versions of nvd3dum.dll in the FireRepository like for example [9.18.13.3489]. I am no coder, so sorry if this is redundant info.

I really dont know what to install / do to at least get your beautiful mod running. And every help to run KSP better would be highly appreciated

[raxo2222]

I crashed only once so far

I think ATM is obsolete by now - I removed it and no memory usage increase comparing previously.

[Pockrtplanesairways]

This is weird. KSP has to have this modded in, while, like, EVERY other game has this built in. But then again, it causes severe glitches sometimes.

[SpacedInvader]

I crashed only once so far

I think ATM is obsolete by now - I removed it and no memory usage increase comparing previously.

This is weird. KSP has to have this modded in, while, like, EVERY other game has this built in. But then again, it causes severe glitches sometimes.

I wouldn't say that ATM is obsolete now.... These two mods work extremely well together, and working together, I consider them to be must have add-ons. With this and ATM in aggressive mode, I'm able to run with a 3.2GB GameData folder, and with this newest version, do it at about 90%-95% stability. Without these mods, I wouldn't even get halfway through the loading screen, or I'd have to cut out at least half of the mods I use and then trim them for parts on top of that. As a result, I recommend this mod to pretty much everyone who I interact with on the forums. I may have made some comments about memory usage, but I'm only trying to give what little input I can to help Faark get to the most optimal configuration.

As for the glitches, I'd say they will be gone soon enough. And anyway, as you said, this is a default part of pretty much every other game, so the fact that Faark has produced this almost single handedly raises KSP to a level with many other games that have had much more money thrown into their development.

[dRuPpI]

well deleted Partcatalog and converted all tgas to png, runs very good now even in VAB

and converting also saved 400 mb GameData folder size from ~1.8 gb to ~1.4 gb

Edited June 1, 2014 by dRuPpI

[raxo2222]

I wouldn't say that ATM is obsolete now.... These two mods work extremely well together, and working together, I consider them to be must have add-ons. With this and ATM in aggressive mode, I'm able to run with a 3.2GB GameData folder, and with this newest version, do it at about 90%-95% stability. Without these mods, I wouldn't even get halfway through the loading screen, or I'd have to cut out at least half of the mods I use and then trim them for parts on top of that. As a result, I recommend this mod to pretty much everyone who I interact with on the forums. I may have made some comments about memory usage, but I'm only trying to give what little input I can to help Faark get to the most optimal configuration.

As for the glitches, I'd say they will be gone soon enough. And anyway, as you said, this is a default part of pretty much every other game, so the fact that Faark has produced this almost single handedly raises KSP to a level with many other games that have had much more money thrown into their development.

Forums crashed >.>

Well ATM broke warp effect in interstellar and scansat buttons on toolbar.

I can have open browser, while playing game too

Also I can do just fine without ATM.

Edited June 1, 2014 by raxo2222

[Badabooooom]

First: Super great mod! This is something that Kerbal should have build in.

I did run some test with ATM 3-1 and Loading on Demand 2.2. It seems like that LoD is not loading ATM's textures properly (easily to see when ATM is running in aggressive mode). Instead of using ATM's textures, it is loading textures directly from mod's resources. This is not normally issue, because LoD and ATM aggressive performance are almost equal - memory usage difference was around 1-2% in my tests. ATM aggressive is better, if you have large builds with lot's of different textures, but in other hand LoD builds are using high quality textures, LoD game is launching much more faster and LoD enabled game seems to be able to run much more longer because memory of unactive textures are quickly released (thus allowing small memory leaks that are in game/Mono build bit longer). Only reason why someone should use ATM aggressive instead of LoD is to reduce texture size so that their computer can run graphics easier.

I hope that you can implement someday active or maybe dynamic texture reduction (change to lower resolution textures after certain number of active textures) to your great plugin! Thank you!

PS: If you need developement help with your plugin - please PM to me. I am pretty busy in work related programming, but maybe I can find tiny amount of time for some fun programming!