[WIP] Loading textures only as required

[TomatoSoup]

I'm not sure how possible this is, but could there be a companion program to this that the unused textures are loaded into such that your plugin can ask it for the textures? Basically, a hacky way to keep them in RAM without exceeding 3.5gb on KSP's thread so it doesn't keep accessing the harddrive.

[sidfu]

sigh still don get it its not picking up all the textures for example with the mods u have installed its only catching 800 of 1700 textures.

[rbray89]

I'm not sure how possible this is, but could there be a companion program to this that the unused textures are loaded into such that your plugin can ask it for the textures? Basically, a hacky way to keep them in RAM without exceeding 3.5gb on KSP's thread so it doesn't keep accessing the harddrive.

Could be done easily: http://forum.unity3d.com/threads/17488-Start-a-external-process

[BananaDealer]

sigh still don get it its not picking up all the textures for example with the mods u have installed its only catching 800 of 1700 textures.

Huh? The purpose of this is to only load textures when and if they are required...

So yes, if you have mods that add up to 1700 textures and you have a craft that requires only 800 of those to be loaded, the mod's gonna load only those 800...

[sidfu]

what i mean is that the mod is only seeing 800 textures where as my total is 1700.

when u start game it should be 0/(ever how many textures u have) for me it should be 0/1700 but its not its 0/871 which means the mod isnt seeing all the textures

[Faark]

@Non-Windows

Yes, one bigger problem is DirectX and the way its currently written doesn't make it easy to provide an OpenGL version as well. I hope there will be time to do so, but last time i checked Steams HW stats (currently down) Windows was by far the most frequent used gaming system, so thats where my priorities are for now. Btw, next version will use net4.0 instead of 4.5, to support WinXP as well.

@sidfu / LOD does not manage all textures / non-part-textures like planets

It only handles part textures! To load and unload textures this mod have to know when textures are used. It currently does so by tracking vessels in flight / parts in editor. For anything else we would need logic to load and unload those textures. For parts it currently uses a not-that-smart directory based heuristic.

Feel free to share your KSP.log, if you think LOD should manage more textures. It currently logs all textures it doesn't managed (search for "REM_IMG:") so we can check what it misses and what we could/should improve.

@changing texture quality

Never used that feature, thus never considered it when working on LOD. Will add it to the todo list, but with a lower priority.

Holy Crud! You just gave me a BRILLIANT idea! All this time, I've been re-loading textures already loaded by KSP. There is no reason why I couldn't load up the TextureDB with my cached textures first right!? This could speed up loading time tremendously!

Thievery! But seriously, it will be interesting how you do that without making the game appear to be crashed for a long time^^

@BananaDealer

Any idea what finally solved your problem? In case others have the same issue...

@Bloodbunny

Thanks, il'll have a look.

@TomatoSoup

Do you have problems with this mod accessing your harddrive? It shouldn't influence your game and only take slightly longer, so creating sth like that doesn't really seem worth the effort for me for now. Apart from that use current window versions (8 for sure, duno about 7) unused RAM to cache frequently used file, so the physical drive might not be used anyway.

Edited March 25, 2014 by Faark

[MK3424]

Thievery! But seriously, it will be interesting how you do that without making the game appear to be crashed for a long time^^

Yeah, thought of that as well, then he should at least credit your work.

[BananaDealer]

@BananaDealer

Any idea what finally solved your problem? In case others have the same issue...

Not a clue... Munfry is tossing around his crazy space noodle theories again...

[rbray89]

@Non-Windows

Yes, one bigger problem is DirectX and the way its currently written doesn't make it easy to provide an OpenGL version as well. I hope there will be time to do so, but last time i checked Steams HW stats (currently down) Windows was by far the most frequent used gaming system, so thats where my priorities are for now. Btw, next version will use net4.0 instead of 4.5, to support WinXP as well.

@sidfu / LOD does not manage all textures / non-part-textures like planets

It only handles part textures! To load and unload textures this mod have to know when textures are used. It currently does so by tracking vessels in flight / parts in editor. For anything else we would need logic to load and unload those textures. For parts it currently uses a not-that-smart directory based heuristic.

Feel free to share your KSP.log, if you think LOD should manage more textures. It currently logs all textures it doesn't managed (search for "REM_IMG:") so we can check what it misses and what we could/should improve.

@changing texture quality

Never used that feature, thus never considered it when working on LOD. Will add it to the todo list, but with a lower priority.

Thievery! But seriously, it will be interesting how you do that without making the game appear to be crashed for a long time^^

@BananaDealer

Any idea what finally solved your problem? In case others have the same issue...

@Bloodbunny

Thanks, il'll have a look.

@TomatoSoup

Do you have problems with this mod accessing your harddrive? It shouldn't influence your game and only take slightly longer, so creating sth like that doesn't really seem worth the effort for me for now. Apart from that use current window versions (8 for sure, duno about 7) unused RAM to cache frequently used file, so the physical drive might not be used anyway.

Yeah, I was looking at that. I bet I can draw to the graphics buffer, We'll have to see.

Yeah, thought of that as well, then he should at least credit your work.

Haha of course! Though it isn't code I'm borrowing, it is the concept.

[blackheart612]

Hmmm, I tried using this w/o Texture Management. The major mods that eats has most appetite for ram I have are KW Rocketry, KSO and Visual Enhancements. It crashes though. I also have a lot of extra mods such as DRe, FAR, KAS, etc.

[Bloodbunny]

Hmmm, I tried using this w/o Texture Management. The major mods that eats has most appetite for ram I have are KW Rocketry, KSO and Visual Enhancements. It crashes though. I also have a lot of extra mods such as DRe, FAR, KAS, etc.

When are you getting the crashes? In the VAB/SPH? That is where 100% of my crashes occurred; seemingly at random.

[blackheart612]

The common crash in the first loading (before the title screen) it feels like it has no improvements at all. I wonder...

[Faark]

I would expect one of the 3 most common causes:

- Not properly installed / working

- To many non-part Textures and thus not enough memory

- Some kind of internal issues (incompatibility, crazy file, whatever)

Logs would be nice to know what is actually going on.. Especially GameData\LoadOnDemand\LoadOnDemand.log and KSP_Data\output_log.txt, pls.

[sidfu]

bloodbunny i tried it again but removed all mods but the ones u showed in your ss still same thing was at 2.5gb usage and the mods are not even full modpack. ill mess around with it more later maybe something ive changed with my mods causing problems. but your mod does work i just cant get to work s well as u lol.

some things to improve on it. the window to show the files need to make it stay open when u looking at it. its praticaly impossible to verfy which textures its missing when the window dont say open.

also having a button for it to make a txt fil listing all the textures if sees would be nice.

to make it less intrusive make it so its a square box with just the number 0/number of textures. and when u click on it it open to the veiw of it now.

[Bloodbunny]

@ sidfu: it is Faark's mod, not mine. That said, the mod didn't work for me when I first tried it; make sure you have all the required dependencies, otherwise it WILL NOT work. Are you using TextureReplacer (it was in my screenshot)? Make sure you open the config file and turn off texture compression or it will interfere with the mod.

@ blackheart: if it is crashing before you get to the main menu, then most likely there is some kind of mod conflict at work, see if it works with a 100% stock setup

Edited March 25, 2014 by Bloodbunny

[blackheart612]

Yeah, it works like a charm on stock setup.

But crashes after all the initial loading even if it has no errors

========== OUTPUTING STACK TRACE ==================

(0x77194B32) (KERNELBASE): (filename not available): RaiseException + 0x49
(0x74AC2A42) (clr): (filename not available): CorExeMain + 0x1072d
(0x74A7D22D) (clr): (filename not available): DllGetClassObjectInternal + 0x1fd60
(0x112DE050) ((module-name not available)): (filename not available): (function-name not available) + 0x0
(0x112DDFBA) ((module-name not available)): (filename not available): (function-name not available) + 0x0
(0x106BBA3C) ((module-name not available)): (filename not available): (function-name not available) + 0x0
(0x3C8332F8) (Mono JIT code): (filename not available):  (wrapper managed-to-native) LoadOnDemand.NativeBridge:NlodRequestedUpdate (void*) + 0x28 (3C8332D0 3C83331B) [03D44E70 - Unity Root Domain] + 0x0
(0x3C8332B5) (Mono JIT code): (filename not available):  LoadOnDemand.NativeBridge:<OnCallbackFromKspThreadRequested>b__3 () + 0x2d (3C833288 3C8332C3) [03D44E70 - Unity Root Domain] + 0x0
(0x04475BD3) (Mono JIT code): (filename not available):  LoadOnDemand.Logic.WorkQueue:Update () + 0xa3 (04475B30 04475CC8) [03D44E70 - Unity Root Domain] + 0x0
(0x043BE629) (Mono JIT code): (filename not available):  (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr) + 0x41 (043BE5E8 043BE67D) [03D44E70 - Unity Root Domain] + 0x0
(0x100EFCF6) (mono): (filename not available): mono_set_defaults + 0x22bf
(0x1005D603) (mono): (filename not available): mono_runtime_invoke + 0x51
(0x00F6BD88) (KSP): (filename not available): RaiseNullExceptionObject + 0xa888
(0x00F6BA76) (KSP): (filename not available): RaiseNullExceptionObject + 0xa576
(0x00F6B625) (KSP): (filename not available): RaiseNullExceptionObject + 0xa125
(0x00F608E8) (KSP): (filename not available): GetMonoBehaviourInConstructor + 0x1758
(0x00F60951) (KSP): (filename not available): GetMonoBehaviourInConstructor + 0x17c1
(0x00F60967) (KSP): (filename not available): GetMonoBehaviourInConstructor + 0x17d7
(0x00F38143) (KSP): (filename not available): QuaternionToEuler + 0x31333
(0x00FCEA9D) (KSP): (filename not available): PlayerMainWndProc + 0x5ad
(0x00FD01FB) (KSP): (filename not available): PlayerWinMain + 0x99b
(0x011174E8) (KSP): (filename not available): Object::SetInstanceID + 0x7d728
(0x01148E10) (KSP): (filename not available): Object::SetInstanceID + 0xaf050
(0x76C6850D) (KERNEL32): (filename not available): BaseThreadInitThunk + 0xe
(0x77DBBF39) (ntdll): (filename not available): RtlInitializeExceptionChain + 0x85
(0x77DBBF0C) (ntdll): (filename not available): RtlInitializeExceptionChain + 0x58

========== END OF STACKTRACE ===========

[BananaDealer]

Yeah, I've been getting all sorts of crashes, even hard CTDs with no log output with this thing running...

Mostly in VAB, it seems to be quite stable while on a vessel. The only problem then is having it load textures and parts for multiple vessels (i.e when they rendezvous), that can get it choking but it usually works itself out...

Something weird that's been occurring lately when using this- KSP sometimes crashes but doesn't! Like, the "Game has crashed, have a log" pops up but KSP continues to run in the back...

As for what your next point of development should be- Probably look into having the textures (both thumbs and actual ones) compressed and maybe getting those pesky crashes in VAB while browsing the catalog fixed. That's what I think is the main problem right now as sometimes my game crashes almost immediately after entering VAB, loading up a craft and turning a page or two of the catalog (a total of say 4~5 minutes between loading screen finish and crash). Maybe also add a button to just unload all catalog thumb textures (or just have them unloaded between page turns and then have the new ones loaded, instead of doing the two at the same time)...

[rbray89]

So I was able to put a "pre-loader" together, and it is a LOT faster. Downside, is as you predicted, there is no way to push a frame update to the screen, so KSP looks like it freezes while it is loading now.

[Faark]

...

That stack trace tells me that the interesting data is in GameData\LoadOnDemand\LoadOnDemand.log (crash/exception details) and the remaining parts of KSP.log / output_log.txt (general info about the mods behavior).

...

Beside of mod incompatibility stuff i am currently "know" about 2 problems. Especially in VAB can the mod attempt to load more textures then memory is available. My test version does already release textures asap, so i consider that solved. But there appear to be real "random" crashes and i'm afraid those are caused by one of my more dangerous design decisions leading to corrupted memory (very bad). Tracking that might take a while, i hope to have more information after next weekend.

(i.e when they rendezvous), that can get it choking but it usually works itself out...

You mean the game does freeze for a short time? Is it really worse than stock KSP? (i always hated that freeze/lag/whatever when approaching my larger space stations in stock KSP)

KSP sometimes crashes but doesn't! Like, the "Game has crashed, have a log" pops up but KSP continues to run in the back...

Interesting. Do you get a log? It might be caused by one of my threads crashing, though that usually just silently disabled my mods functionality and should just barely be possible anyway. Does LOD still load textures after that?

So I was able to put a "pre-loader" together, and it is a LOT faster. Downside, is as you predicted, there is no way to push a frame update to the screen, so KSP looks like it freezes while it is loading now.

Please leave me a message in case you encounter any issues i might have missed and are caused by removing those texture files from KSPs internal directory listing.

[BananaDealer]

You mean the game does freeze for a short time? Is it really worse than stock KSP? (i always hated that freeze/lag/whatever when approaching my larger space stations in stock KSP)

Yes, it freezes like "normal" but for a lot longer and sometimes just stays like that until it eventually crashes (which can be like 5 minutes after it froze). It usually works itself out and loads the textures, even if they are initially low res...

Interesting. Do you get a log? It might be caused by one of my threads crashing, though that usually just silently disabled my mods functionality and should just barely be possible anyway. Does LOD still load textures after that?

Sometimes I get a log, other times I don't even if the pop up states there's a log...

LOD continues to work, along with the game for awhile after the "crash" but as soon as I try to change scenes it just crashes to desktop without any notice. Note that this seems to happen only in the editor buildings (VAB/SPH).

[SpacedInvader]

This mod really has some serious potential. Ever since I started playing KSP (all of 3 weeks ago, but still), I've been wondering why the heck it loads all of the textures at the beginning when they could all be loaded and unloaded on an as needed basis. Installing this has dropped my RAM usage by about 1GB and is holding steady at about 2.4GB even with relatively fast VAB browsing. This is uncompressed and compared to aggressive mode of the active texture management mod. The only issue I've noticed (other than the ones mentioned in the initial post and later in the thread) is that on loading parts in the VAB, they sometimes load an extremely low-rez / blurry texture which then updates after 10-30 seconds. That being said, I'd rather have a short delay on loading than to have to bend over backwards to trim out parts and mods to keep RAM usage under 3.5GB (my crash point).

Thank you for this!

PS: Eagerly awaiting a version that fixes StretchySRB textures.

Edited March 27, 2014 by SpacedInvader

[blackheart612]

That stack trace tells me that the interesting data is in GameData\LoadOnDemand\LoadOnDemand.log (crash/exception details) and the remaining parts of KSP.log / output_log.txt (general info about the mods behavior).

This?

Logging Started at 3/26/2014 11:03:14 AM (Async)
3/26/2014 11:04:34 AM: Thumbnail cache is invalid: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\HF0DAZKR8YXOXP5T.THUMB
Error: System.FormatException: Loaded format (32x32, FMT21) does not match expected format (32x32, FMT894720068)
   at TextureInitialization.ThumbTryLoadFromDisk_OnLoaded(Byte[] loaded_data)
3/26/2014 11:04:34 AM: Thumb cache hit, loading D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\NHWFXPJGD3HDDE19.THUMB to 1
3/26/2014 11:04:34 AM: Texture prepared: 1
3/26/2014 11:04:34 AM: Thumbnail cache is invalid: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\6CWSXKZUVPCRQM1V.THUMB
Error: System.FormatException: Loaded format (32x32, FMT21) does not match expected format (32x32, FMT894720068)
   at TextureInitialization.ThumbTryLoadFromDisk_OnLoaded(Byte[] loaded_data)
3/26/2014 11:04:34 AM: Writing cache: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\HF0DAZKR8YXOXP5T.THUMB (Texture)
3/26/2014 11:04:34 AM: Texture prepared: 0
3/26/2014 11:04:34 AM: Thumbnail cache is invalid: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\8HX7WHDK1JCO7ESI.THUMB
Error: System.FormatException: Loaded format (32x32, FMT21) does not match expected format (32x32, FMT894720068)
   at TextureInitialization.ThumbTryLoadFromDisk_OnLoaded(Byte[] loaded_data)
3/26/2014 11:04:34 AM: Writing cache: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\OT270VMP8ZSRPLA4.THUMB (Texture)
3/26/2014 11:04:34 AM: Texture prepared: 4
3/26/2014 11:04:34 AM: Thumb cache hit, loading D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\73Q5GB8BXZD2XW8N.THUMB to 5
3/26/2014 11:04:34 AM: Texture prepared: 5
3/26/2014 11:04:34 AM: Thumbnail cache is invalid: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\ILQIB8IJAC74G729.THUMB
Error: System.FormatException: Loaded format (32x32, FMT21) does not match expected format (32x32, FMT894720068)
   at TextureInitialization.ThumbTryLoadFromDisk_OnLoaded(Byte[] loaded_data)
3/26/2014 11:04:34 AM: Writing cache: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\KUNE8FZQIXOBZIQN.THUMB (Texture)
3/26/2014 11:04:34 AM: Texture prepared: 7
3/26/2014 11:04:34 AM: Thumb cache hit, loading D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\VPH1B2WV70F5EU57.THUMB to 8
3/26/2014 11:04:35 AM: Texture prepared: 8
3/26/2014 11:04:35 AM: We got a AF match, a: 32x32@D3DFMT_A8R8G8B8, b: 32x32@D3DFMT_A8R8G8B8
3/26/2014 11:04:35 AM: success reporting?
3/26/2014 11:04:35 AM: No AF match, a: 32x32@D3DFMT_DXT5, b: 32x32@D3DFMT_A8R8G8B8
3/26/2014 11:04:35 AM: DEBUG: HORRIBLE SYNC CONVERSION IN PLACE. REMOVE THAT ASAP ONCE DEBUGGING IS DONE!
3/26/2014 11:04:36 AM: System.Exception: Unrecognized format: D3DFMT_DXT5
   at AssignableFormat.PixelFormatFromD3DFormat(_D3DFORMAT from)
   at FormatDatabase.GetConverterJobScope.RunConvertion(AssignableFormat expectedFormat)
   at ThumbnailTexture.GetAssignableData(AssignableFormat assignableFormat, Func`2& delayedCb)
   at GPU.AssignDataToThumbnailAsyncScope.Run(IntPtr devicePtr)
   at KSPThread.Update(IDirect3DTexture9* devicePtrTexture)
   at NlodRequestedUpdate(Void* deviceRefTexturePtr)
3/26/2014 11:04:38 AM: Writing cache: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\U9K2NP68YI9YQ8LZ.THUMB (Texture)
3/26/2014 11:04:38 AM: Texture prepared: 9
3/26/2014 11:04:38 AM: Thumbnail cache is invalid: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\4J03IYFS02XA5AGL.THUMB
Error: System.FormatException: Loaded format (32x32, FMT21) does not match expected format (32x32, FMT894720068)
   at TextureInitialization.ThumbTryLoadFromDisk_OnLoaded(Byte[] loaded_data)
3/26/2014 11:04:38 AM: Thumbnail cache is invalid: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\3XDRBW51Q4HAERC5.THUMB
Error: System.FormatException: Loaded format (32x32, FMT21) does not match expected format (32x32, FMT894720068)
   at TextureInitialization.ThumbTryLoadFromDisk_OnLoaded(Byte[] loaded_data)
3/26/2014 11:04:38 AM: Thumbnail cache is invalid: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\8CTYKM3AI4PX7H6G.THUMB
Error: System.FormatException: Loaded format (32x32, FMT21) does not match expected format (32x32, FMT894720068)
   at TextureInitialization.ThumbTryLoadFromDisk_OnLoaded(Byte[] loaded_data)
3/26/2014 11:04:40 AM: Writing cache: D:\Games\Kerbal Space Program Ultra 23 KSO LOD\GameData\LoadOnDemand\G1LS7XDEDEA8T0H8.THUMB (Normal)

It's a separate ksp install to test it.

[Mekan1k]

So... Why has squad not done this already?

[SpacedInvader]

So... Why has squad not done this already?

Perhaps it's on their list, but they have such a long dev list that they just haven't gotten around to it yet.

[SpacedInvader]

So, out of curiosity, how do you want to handle crash reports? I don't want to just start sending you crash logs for fear that you're going to get inundated...