[WIP] Loading textures only as required

[Faark]

This version seems to use much more memory while in play. A rocket that used to have a working memory of about 2.8gb on the pad now uses about 3.3gb with a commit of 3.4gb. It is fairly complex with about 110 parts, but this could mean trouble for complex structures like stations.

Okay, i spend way to much time today testing with a 85 and 250 [lod-]textures launch pad scenario.

V SpaceCenter LaunchPad (WorkingSet/Commit)

84tex

v3 1g121/1g267 1g450/1g665

v2 1g140/1g215 1g500/1g600

v3 1g155/1g272 1g450/1g680

v2 1g141/1g220 1g500/1g592

250tex

v2 1g154/1g226 1g757/1g842

v3 1g144/1g267 1g783/1g914

v2 1g154/1g234 1g737/1g841

v3 1g155/1g279 1g800/1g942

v3 1g156/1g271 1g773/1g972(f)

v3 1g155/1g268 1g790/1g890(n)

v3 1g151/1g261 1g855/1g950(f)

v3 1g149/1g270 1g780/1g908(n)

v3 1g016/1g268 1g660/1g892(n)

v3 1g157/1g269 1g941/2g036(f)

are as expected v3 required about ~60-80mb in total. Measuring it is kinda complicated, since everything can have an impact. For example an focused app during SC=>LaunchPad had 40-150mb more memory committed. But stuff like that is in the nature of multi-threaded applications. I cannot reproduce such a significant increase in memory usage.

There are some fixes that might relate to your issue. I changed some handling in case there were 2 similar named files with different extensions. Do you have stuff like that in your GameData? Did the amount of textures changed between version in the first place? (You can re-enable the debug UI via cfg)

[..]

So far this kind of errors was always caused by a missing dependency, so please try to re-install all dependencies listed in the first post, including DirectX. Btw: nvd3dum.dll is the file used by KSP, not the one required for LOD.

I did run some test with ATM 3-1 and Loading on Demand 2.2. It seems like that LoD is not loading ATM's textures properly (easily to see when ATM is running in aggressive mode). Instead of using ATM's textures, it is loading textures directly from mod's resources.

Thats exactly what LOD is supposed to do. It currently does not offer any kind of "low-res" textures and i hope this will never be necessary (apart from DXT compression, what will be one of the next features and should help a lot). LOD tries to always have enough memory available by not loading textures unless they are required, but itself requires logic to know when those textures are required. It currently only has those logic for part textures. ATM instead just scales down textures, thus is a perfect to handle textures LOD can't.

Help would be appreciated, ofc!

[Alewx]

Since the mipmap disabling in TR, I can have mutliple launches and revertings to the VAB, depending on the part count, 4-5 times till it crashes with oom exception. So I can't report any problems from this weekend.

[smart013]

Whoa!

Ok, you were right, there *was* a error on my side, and now everything seems to run just fine. And fast!

Great mod, really hoping you keep up the fantastic work.

Edited June 1, 2014 by smart013

[SpacedInvader]

Okay, i spend way to much time today testing with a 85 and 250 [lod-]textures launch pad scenario.

There are some fixes that might relate to your issue. I changed some handling in case there were 2 similar named files with different extensions. Do you have stuff like that in your GameData? Did the amount of textures changed between version in the first place? (You can re-enable the debug UI via cfg)

V SpaceCenter LaunchPad (WorkingSet/Commit)

84tex

v3 1g121/1g267 1g450/1g665

v2 1g140/1g215 1g500/1g600

v3 1g155/1g272 1g450/1g680

v2 1g141/1g220 1g500/1g592

250tex

v2 1g154/1g226 1g757/1g842

v3 1g144/1g267 1g783/1g914

v2 1g154/1g234 1g737/1g841

v3 1g155/1g279 1g800/1g942

v3 1g156/1g271 1g773/1g972(f)

v3 1g155/1g268 1g790/1g890(n)

v3 1g151/1g261 1g855/1g950(f)

v3 1g149/1g270 1g780/1g908(n)

v3 1g016/1g268 1g660/1g892(n)

v3 1g157/1g269 1g941/2g036(f)

are as expected v3 required about ~60-80mb in total. Measuring it is kinda complicated, since everything can have an impact. For example an focused app during SC=>LaunchPad had 40-150mb more memory committed. But stuff like that is in the nature of multi-threaded applications. I cannot reproduce such a significant increase in memory usage.

I will spend time looking through my gamedata folder to see if any mods have left similarly named files like that sitting around. That being said, I'm a little confused by your numbers. It looks to me like you're getting the same 400-600 MB jump in memory usage that I am experiencing, except you're getting it for both V2 and V3, where I am only getting it with V3. Perhaps the issue is that I was getting better usage stats with the previous version through some quirk of my install?

EDIT: Is it possible that the extra texture memory might be at least slightly additive when you have more textures? I've been testing (unscientifically) what happens when I reduce the overall number of textures and the results seem to show that being the case, at least to a small degree. I had about 2200 textures, which would have me load into the VAB at the aforementioned 3.3GB, then I tried removing a few hundred textures, which resulted in the same ship loading at about 3.15GB without losing any of its textures.

Edited June 2, 2014 by SpacedInvader

[Alewx]

I will spend time looking through my gamedata folder to see if any mods have left similarly named files like that sitting around. That being said, I'm a little confused by your numbers. It looks to me like you're getting the same 400-600 MB jump in memory usage that I am experiencing, except you're getting it for both V2 and V3, where I am only getting it with V3. Perhaps the issue is that I was getting better usage stats with the previous version through some quirk of my install?

EDIT: Is it possible that the extra texture memory might be at least slightly additive when you have more textures? I've been testing (unscientifically) what happens when I reduce the overall number of textures and the results seem to show that being the case, at least to a small degree. I had about 2200 textures, which would have me load into the VAB at the aforementioned 3.3GB, then I tried removing a few hundred textures, which resulted in the same ship loading at about 3.15GB without losing any of its textures.

Here I can confirm similar behaviour with entering the VAB, now as I got my first welded parts. I have almost the same load in memory but I have currently only 1012 textures handled by LoD.

@SpaceInvader: How big is your persistant save file that you use for testing? Mine is something around 9mb; 145 debris, 1 Station, 1 ship, around 20 landing flags and about 15 sattelites.

[SpacedInvader]

Here I can confirm similar behaviour with entering the VAB, now as I got my first welded parts. I have almost the same load in memory but I have currently only 1012 textures handled by LoD.

@SpaceInvader: How big is your persistant save file that you use for testing? Mine is something around 9mb; 145 debris, 1 Station, 1 ship, around 20 landing flags and about 15 sattelites.

You're confirming the possible additive memory, or the large-ish increase in memory going from VAB to pad? Sorry, just want to be sure of what you're saying here. Also, my persistent save is much smaller at about 1.4MB.

[Alewx]

My average memory useage with VAB entering from fresh gamestart is 2.8GB and then going to launchpad it is 3.1GB, it only gets changed a little depending on the complexity of the craft.

But as I have only half the number of textures from you, I assume that the memory is used by something else and not by textures otherwise my numbers would have to be much lower than yours.

[Badabooooom]

Something is going wrong with TGA images:

2.6.2014 11:18:48: Loaded Texture 534, informing KSP runtme
2.6.2014 11:18:48: Unhandled exception!
2.6.2014 11:18:49: System.Exception: Failed to load TGA file: C:\Kerbal\Game\GameData\ProceduralFairings\thrustPlate1.tga ---> System.ArgumentException: Virheellinen parametri. (Invalid Argument)
   kohteessa System.Drawing.Image.get_RawFormat()
   kohteessa System.Drawing.Graphics.DrawImage(Image image, Int32 x, Int32 y, Int32 width, Int32 height)
   kohteessa System.Drawing.Bitmap..ctor(Image original, Int32 width, Int32 height)
   kohteessa System.Drawing.Bitmap..ctor(Image original)
   kohteessa FormatDatabase.RecognizeTGA(FileInfo file, Byte[] data)
   --- Sisäisen poikkeuksen pinon jäljityksen loppu ---
   kohteessa FormatDatabase.RecognizeTGA(FileInfo file, Byte[] data)
   kohteessa FormatDatabase.AddRecognitionScope`1.Recognize(FileInfo file, Byte[] bytes)
   kohteessa FormatDatabase.Recognize(String file, Byte[] data)
   kohteessa TextureManager.StartLoadHighResTextureScope.ProcessLoadedData(Byte[] loaded_data)
   kohteessa Disk.RequestFileScope.RunProcessor()
   kohteessa Work.WorkThread()
   kohteessa System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   kohteessa System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   kohteessa System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   kohteessa System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   kohteessa System.Threading.ThreadHelper.ThreadStart()

This doesn't seems to happen always. After quickly looking code, it seems like that static LoadTargaImage method is called in TargaImage.h during FormatDatabase.RecognizeTGA methods execution (LoadTargaImage is inlined - so it is not visible in stacktrace). LoadTargaImage seems to construct first it's own targa image object and then gives this image object for Syste.Drawing Bitmap constructor, which should create a new bitmap object. According this stack overflow discussion, it is likely that targa image is invalid somehow.

[Subcidal]

Trying this mod again seeing it's been updated a few times now. I keep crashing on the last loading screen before the main menu, I'm unsure why. I've disabled TR's mipmap and compression options, Not sure what else it could be. I have a pretty heavily modded game, So i'm unsure as to where to start. And the varying amount of crash reports in this thread isn't helping me pinpoint it much either.

Anybody have any insight? Would love to use LoD, Starting to become essential at this point. Everytime I make a long flight to the tundra (Using RSS) I usually crash when i come down to land =( hour long flights down the drain, hard to deal with.

Crash logs::

https://www./?x8rd1d1i818218r

[BananaDealer]

Oooh... New version!

Edit: Just wanna say- the info thing got gone when I updated to LoDv3. Also some parts aren't getting their textures loaded properly. Some are high-res, others would be as if you're loading with quarter/eight-res on. That's after the thing's supposedly done preparing...

Edited June 3, 2014 by BananaDealer

[SpacedInvader]

My average memory useage with VAB entering from fresh gamestart is 2.8GB and then going to launchpad it is 3.1GB, it only gets changed a little depending on the complexity of the craft.

But as I have only half the number of textures from you, I assume that the memory is used by something else and not by textures otherwise my numbers would have to be much lower than yours.

Hmm... Now I'm kinda wondering where all of your memory is going if I've got twice the textures you've got and you're getting the same memory usage that I'm getting.

[raxo2222]

ATM seems to not work at all - I get lag with 300 part ship. Well, Dynamic Time or how its called is set to keep 10 FPS, but time itself slows down to 1/3 or 1/4 of normal. So how I can play at normal speed with 300 part ship?

Also it doesn't let ATM's compressed textures in.

Or my computer is too slow for game with 60 mods?

Edit: Stock settings to reduce texture quality doesn't work too.

Edited June 3, 2014 by raxo2222

[Alewx]

Hmm... Now I'm kinda wondering where all of your memory is going if I've got twice the textures you've got and you're getting the same memory usage that I'm getting.

That is exactly my question. We should share Mod lists. But I doubt that there will be too much difference.

With the latest Universal Storage update, now I have 1015 textures in LoD.

[Faark]

Just wanna say- the info thing got gone when I updated to LoDv3. Also some parts aren't getting their textures loaded properly. Some are high-res, others would be as if you're loading with quarter/eight-res on. That's after the thing's supposedly done preparing...

Debug UI is intentionally gone, you can re-enable it via the cfg. Details about the other issues incl instructions on how to reproduce it would be nice. Please also try to delete the old LOD directory and report whether this makes any difference.

Crash logs::

https://www./?x8rd1d1i818218r

There is definitively some mod modifying LOD textures, either an old ATM version or there might be other TR configs that re-enable this.

Something is going wrong with TGA images:

[..]

This doesn't seems to happen always. After quickly looking code, it seems like that static LoadTargaImage method is called in TargaImage.h during FormatDatabase.RecognizeTGA methods execution (LoadTargaImage is inlined - so it is not visible in stacktrace). LoadTargaImage seems to construct first it's own targa image object and then gives this image object for Syste.Drawing Bitmap constructor, which should create a new bitmap object. According this stack overflow discussion, it is likely that targa image is invalid somehow.

I had quite a few "Parameter not valid" exceptions in situations where the game was running out of memory, even though it should give an OutOfMemoryException. Have you verified that this wasn't the cause? Since so far i couldn't reproduce this kind of error any other way.

Hmm... Now I'm kinda wondering where all of your memory is going if I've got twice the textures you've got and you're getting the same memory usage that I'm getting.

I guess the diff are textures that are handled by LOD and not loaded, thus don't require any memory.

[BananaDealer]

Debug UI is intentionally gone, you can re-enable it via the cfg. Details about the other issues incl instructions on how to reproduce it would be nice. Please also try to delete the old LOD directory and report whether this makes any difference.

I'm not sure how to replicate it... Got about 2170 textures so maybe that's the "issue". It just happens on part by part basis. Sometimes it's even only parts of a part's texture being lower res, like say with SumGhai's parachute-sporting clamp-o-tron. The parachute on the outside is high-res, while the clamp-o-tron is lower res.

I'll provide pictures as soon as the bug shows up again...

[Senshi]

We really need a KSP Memory Profiler, telling us which part/model/teture hogs how much memory...I only have 1029 textures, all cramped and compressed with ATM and at 1/4 resolution...starts with 3.0GB memory on the launchpad, then slowly creeps up to the inevitable 3.6GB crash about 30-45min later...

[BananaDealer]

We really need a KSP Memory Profiler, telling us which part/model/teture hogs how much memory...I only have 1029 textures, all cramped and compressed with ATM and at 1/4 resolution...starts with 3.0GB memory on the launchpad, then slowly creeps up to the inevitable 3.6GB crash about 30-45min later...

You might want to consider this mod then... Also perhaps switching to aggressive ATM, if you haven't already...

[Badabooooom]

I had quite a few "Parameter not valid" exceptions in situations where the game was running out of memory, even though it should give an OutOfMemoryException. Have you verified that this wasn't the cause? Since so far i couldn't reproduce this kind of error any other way.

True. I did try demo from Targa Image Reader that you are using in your code - it managed to read TGA without any problems. So - at least it is not Targa Image Reader's fault. Going to try some texture reduction packs and see is that going to help - or maybe I will write own tool that allows to select what textuers are going to reduced (ATM is not going to fix problem, because I want to use LoD).

[dRuPpI]

ATM seems to not work at all - I get lag with 300 part ship. Well, Dynamic Time or how its called is set to keep 10 FPS, but time itself slows down to 1/3 or 1/4 of normal. So how I can play at normal speed with 300 part ship?

that's normal and has nothing to do with LOD you need 6 GHZ CPU to play a 300 Part ship with out getting a yellow timer

Note: if you use more Mods (Plugins) the performance of KSP goes down and down and down.

Well, Dynamic Time or how its called is set to keep 10 FPS, but time itself slows down to 1/3 or 1/4 of normal. So how I can play at normal speed with 300 part ship?

That's what Time Control do is slows down time to keep FPS

[BananaDealer]

More on the bug front. Pictures:

Javascript is disabled. View full album

In the first and second pictures, the KSO docking ports were loaded at a lower resolution than the rest of the parts (game's set to load Half-Res at 8xAA). Same can be said for the battery packs on the left and the Stretchy Tank and KW engine in the third image. The Procedural Fairing base also experiences the same issue.

[raxo2222]

that's normal and has nothing to do with LOD you need 6 GHZ CPU to play a 300 Part ship with out getting a yellow timer

Note: if you use more Mods (Plugins) the performance of KSP goes down and down and down.

That's what Time Control do is slows down time to keep FPS

So lowering resolution of textures won't help? Well, so I have to play in slow motion, when my lander is docked to rest of ship.

I thought it was GPU having too much stuff in its hand o.o

EDIT: So I guess I will be using MechJeb for navigation

Also my lander itself has 150 parts or so and works fine alone - no slow motion.

Edited June 3, 2014 by raxo2222

[frash23]

Do you think an openGL port possible?

I'm not asking for it right now, just if possible :b

[Alewx]

Do you think an openGL port possible?

I'm not asking for it right now, just if possible :b

I think not as it sounds like Fraak wrote it do directly interact with the DX compnents to do his magic. So it's win-exclusive currently, maybe in a distant future it might be possible. But that is just me, damn win 7 user

[SpacedInvader]

That is exactly my question. We should share Mod lists. But I doubt that there will be too much difference.

With the latest Universal Storage update, now I have 1015 textures in LoD.

Is there a program or mod that allows you to easily generate a mod list? I do have 80ish mods and I'm not really that interested in writing them all out...

I guess the diff are textures that are handled by LOD and not loaded, thus don't require any memory.

Does LOD handle MBM files at all? I've noticed several of my largest part mods are mostly MBM and removing them from the game can free up hundreds of MB of memory still, even with LOD + ATM installed, so I'm wondering if this might be where all the extra memory is being eaten up?

We really need a KSP Memory Profiler, telling us which part/model/teture hogs how much memory...I only have 1029 textures, all cramped and compressed with ATM and at 1/4 resolution...starts with 3.0GB memory on the launchpad, then slowly creeps up to the inevitable 3.6GB crash about 30-45min later...

I think this is the next question we need to solve. This mod has allowed most of us to install way more mods than would previously have been possible by cutting out most of the texture memory (I know for a fact that I don't even get to the main menu without it with my current setup), but now we're left with the question of where all that memory is still being used. It's pretty clear that its something that we all have in common, or at least in similar, as a load of 2.8 - 3.0 GB with an increase moving into play mode seems very common.

[Faark]

Going to try some texture reduction packs and see is that going to help - or maybe I will write own tool that allows to select what textuers are going to reduced (ATM is not going to fix problem, because I want to use LoD).

Yea, that should work for now. But i hope that LoD loading images compressed via DXT will save enough to not having to care about sth like that anymore. If it doesn't then this project will need some other kind of reduction, ofc.

I'm not sure how to replicate it... Got about 2170 textures so maybe that's the "issue". It just happens on part by part basis. Sometimes it's even only parts of a part's texture being lower res, like say with SumGhai's parachute-sporting clamp-o-tron. The parachute on the outside is high-res, while the clamp-o-tron is lower res.[..]

Thanks for the report and pics. Especially that "sometimes" part is worrying. Load requests cannot be "dropped" or sth like that, at least not without leaving those x/y status screen open forever. Is there any interesting in the logs? Also it would be interesting whether the necessary textures of those parts are actually referenced at that time according to DebugUI => Info (that button does write that stuff into the ksp log).

So far at least i haven't experienced anything like that by myself

So lowering resolution of textures won't help?

It highly depends on what is slowing down KSP. If its to much stuff to render smaller textures can help, but lots of parts usually mean slow frame rate because of badly scaling physic calculations.

Do you think an openGL port possible?

Possible? Yes. Likely? No, to be honest. For sure not anytime "soon" (valve time, if you know what i mean), as slow as the project is currently progressing...

Does LOD handle MBM files at all? I've noticed several of my largest part mods are mostly MBM and removing them from the game can free up hundreds of MB of memory still, even with LOD + ATM installed, so I'm wondering if this might be where all the extra memory is being eaten up?

Yes, MBM is a fully supported image format for this mod, just like TGA or PNG. But LOD can only handle images (of whatever format) that it can associate with a part, so that might be a cause. Images handled by LOD are listed in both the log files and its config.

But again the best way to say anything about some issue (or in your case observation) would be to get fool proof instructions on how to reproduce it here, preferably with a near-stock installation. I guess you didn't just delete some random MBM files?