[WIP] Loading textures only as required

[rottielover]

PM headed your way with download link for DWI file.

[rottielover]

I have no idea how exactly, but now it's working....

The only thing I can think of, is that I completely removed all redistribute anything listed in my add/remove programs. rebooted, installed the x86 c++ one and rebooted, then clean installed KSP again, and re-added the dll.

I suspect the real culprit to be the AVG installed redistribs, as to completely remove AVG I had to download a cleaner program...

[BananaDealer]

I have no idea how exactly, but now it's working....

The only thing I can think of, is that I completely removed all redistribute anything listed in my add/remove programs. rebooted, installed the x86 c++ one and rebooted, then clean installed KSP again, and re-added the dll.

I suspect the real culprit to be the AVG installed redistribs, as to completely remove AVG I had to download a cleaner program...

Like I said, you only ever need one version of the C++ redist (either 32bit (x86) or 64bit; versions from different years notwithstanding).

I'm not excluding AVG preventing things from being ran in your KSP folder, even when whitelisted (as anti-virals can be weird sometimes).

I do hope you're not now running without an anti-virus however...

[rottielover]

I do hope you're not now running without an anti-virus however...

I'm trying out Vipre at the moment.

[biohazard15]

Damn, that's a great plugin. Saved about 1Gb - sure, that low-to-hi-res loading looks strangely reminiscent of some Unreal Engine-powered console games, but anything for some RAM.

I've got one question\idea: can you load textures in RAM, but do it OUTSIDE of main KSP process, something like RAM drive, and then tell KSP to load on demand? That will probably require some serious programming skills (basically this needs to be a software RAM disk application, that must be launched before KSP), 64-bit system and lots of free RAM, but in theory that could be used to effectively circumvent 3.5Gb limit.

[BananaDealer]

Damn, that's a great plugin. Saved about 1Gb - sure, that low-to-hi-res loading looks strangely reminiscent of some Unreal Engine-powered console games, but anything for some RAM.

I've got one question\idea: can you load textures in RAM, but do it OUTSIDE of main KSP process, something like RAM drive, and then tell KSP to load on demand? That will probably require some serious programming skills (basically this needs to be a software RAM disk application, that must be launched before KSP), 64-bit system and lots of free RAM, but in theory that could be used to effectively circumvent 3.5Gb limit.

I think something like that was suggested before. Not sure on whether it went anywhere...

[biohazard15]

Not sure where this should go, so I'll post it in both topics.

There is an interesting visual glitch with DMagic Mag Boom and LoadOnDemand plugin:

It looks like that LoadOnDemand doesn't load alpha texture for boom (the rest is OK). Launch clamps looks the same when its texture isn't loaded - but it loads correctly.

[SpacedInvader]

Not sure where this should go, so I'll post it in both topics.

There is an interesting visual glitch with DMagic Mag Boom and LoadOnDemand plugin:

http://i.imgur.com/RsbAdPk.jpg

It looks like that LoadOnDemand doesn't load alpha texture for boom (the rest is OK). Launch clamps looks the same when its texture isn't loaded - but it loads correctly.

I didn't even know that was an artifact of this mod... for some reason I thought that was an issue with the part itself ...

[KvickFlygarn87]

How crashy is this? I'm considering getting it.

[BananaDealer]

How crashy is this? I'm considering getting it.

Depending on how many textures you're running with...

On my end, with 2198 textures- it crashes fairly often in VAB. But it's stable while in-flight.

[Guest]

I have an idea that could reduce VAB crashes. Would that be possible to load textures after a part is picked up and placed in VAB? Currently, a lot of textures uselessly loads when browsing tabs in VAB, and this leads to crashes. If left tumbnails on the side display and loaded only parts that were actually placed, the issue should go away.

[SpacedInvader]

I have an idea that could reduce VAB crashes. Would that be possible to load textures after a part is picked up and placed in VAB? Currently, a lot of textures uselessly loads when browsing tabs in VAB, and this leads to crashes. If left tumbnails on the side display and loaded only parts that were actually placed, the issue should go away.

I don't actually think this would help all that much... I always run KSP with my system memory monitor open to keep an eye on use as I play and have noticed that memory does not increase while loading a page of textures while browsing the VAB. I think that that window may be describing the page full of placeholder textures rather than the full sized examples.On the other hand, the issue for me always seems to be reentering the VAB after a failed launch. Each time I return to the VAB, KSP holds onto between 100MB-250MB more than when I left it in the first place and that memory isn't released at all. That being said, I really think the best possible place for additional overhead is still planetary textures because they are always loaded, no matter whether or not you're looking at them. If we could have a way that would only load placeholder textures while not looking directly at them, or even when looking from a distance, and then load higher resolution textures as you close in, I feel like it would save a good chunk of wasted memory.

Edited May 4, 2014 by SpacedInvader

[Guest]

It's not a memory crash. It's a crash caused by swapping textures around too quickly (also it seems that after a certain number of swaps, it crashes anyway). Don't look at memory monitor, look at the LOD monitor in-game. If we stuck to thumbnails in VAB window, there would be little on-the-fly loading in VAB, and that would reduce crashes.

[Faark]

I have an idea that could reduce VAB crashes. Would that be possible to load textures after a part is picked up and placed in VAB? Currently, a lot of textures uselessly loads when browsing tabs in VAB, and this leads to crashes. If left tumbnails on the side display and loaded only parts that were actually placed, the issue should go away.

I don't really like that since some parts rly look ugly and more importantly it doesn't fixes the problem. There rly should be enough memory to load a dozen additional parts beside the current rocket, especially since it is usually the same situation on the launchpad anyways. Because of that i never even considered this.

But it is an extremely easy workaround for now for a bunch of concepts that don't rly work out as good i had hoped and have to be improved especially for fast VAB browsing. The next version will contain such a configuration flag that will even default to DontLoadEditorPages = true, for now.

It's a crash caused by swapping textures around too quickly[...]

Those should be memory crashes nevertheless. As mentioned does LOD currently can use quite some RAM while loading images... and verifying it is indeed memory related isn't that easy. The safest way to do so is via crash dumps, but i definitively won't let anyone upload more than a GB of data. Sharing a save VMMap data file for the crashed KSP.exe might be the best way to do so.

Not sure where this should go, so I'll post it in both topics.

There is an interesting visual glitch with DMagic Mag Boom and LoadOnDemand plugin:

[...]

It looks like that LoadOnDemand doesn't load alpha texture for boom (the rest is OK). Launch clamps looks the same when its texture isn't loaded - but it loads correctly.

Thanks for the report, i'll look into it! The upcoming version will support disabling LOD for individual textures, so that might help until there is an ultimate solution.

I've got one question\idea: can you load textures in RAM, but do it OUTSIDE of main KSP process, something like RAM drive, and then tell KSP to load on demand? That will probably require some serious programming skills (basically this needs to be a software RAM disk application, that must be launched before KSP), 64-bit system and lots of free RAM, but in theory that could be used to effectively circumvent 3.5Gb limit.

No, texture objects have to be within KSP/Unitys 4GB address space. Yes, in theory a different process could be used to preload & prepare textures. It wouldn't speed up Disk IO very much, since a modern OS like Win8 already caches frequently used files like maybe KSP textures in unused RAM. Yes, it might still would allow a lot more preparation and would make it simpler to do so. But probably still not worth the effort and should require System.Diagnostics.Process, that is outright banned for KSP modding afaik.

[Proot]

Will be the next version fully compatible with the texture compression/replacement?, or will have the next version some type of texture management?

I love this concept but can't use it if this don't handle well ATM and/or TR.

[Faark]

Will be the next version fully compatible with the texture compression/replacement?, or will have the next version some type of texture management?

The currently available version of this mod should already be fully compatible with ATM (as long as LOD is installed properly and you have ATM 3.0 or newer). It is also compatible with TextureReplacer, as long as TR's compression stuff is disabled by either having ATM installed or manually disabling it. I might make LOD automatically disable TR's compression with a future version, if that isn't to much of a hassle.

[Proot]

The currently available version of this mod should already be fully compatible with ATM (as long as LOD is installed properly and you have ATM 3.0 or newer). It is also compatible with TextureReplacer, as long as TR's compression stuff is disabled by either having ATM installed or manually disabling it. I might make LOD automatically disable TR's compression with a future version, if that isn't to much of a hassle.

I'm afraid in my case is not (I have all the dependencies)...

My LOD log (as least now I have logs!) : https://mega.co.nz/#!soMxTQ7C!vIsKcfIrVaQiqOXeidU-gjgdSLTwozJC1sdNjKwl8ZE

Most of textures appears as pink/black pattern, randomly some of them loads well, but the normal maps are wrong too.

BTW, I have now a huge amout of .THUMB files...

[Yarbrough08]

I'm afraid in my case is not (I have all the dependencies)...

My LOD log (as least now I have logs!) : https://mega.co.nz/#!soMxTQ7C!vIsKcfIrVaQiqOXeidU-gjgdSLTwozJC1sdNjKwl8ZE

Most of textures appears as pink/black pattern, randomly some of them loads well, but the normal maps are wrong too.

BTW, I have now a huge amout of .THUMB files...

I'm having the exact same issue (pink/black spotted textures at horrible resolutions). TextureReplacer is the problem, even with compression and mipmap generation (b/c I thought just maybe...) turned off. Removing TR and deleting the load on demand folder (to re-cache all the .THUMB files) resolves the issue (yes! freed up 1.5 Gb RAM, outstanding mod).

@Faark: I received no error in the log with TR installed, but it only cached about half of my textures. LOD said I had 688 textures, but only 368 when it was creating the cache. If I load all of the cached textures, it stops at 325 and CTD (or just freezes) with no error in the log. Uninstalling TR corrected this and cached the entire 688.

Also, what does this do with normal maps? It seems like they are not being recognized as normals anymore (more like a black diffuse applied on top of the rest). Might need to handle those textures differently, or exclude them altogether.

Edit: My apologies; it may not be the normal, could be something with the alpha layer... 0 Alpha(fully transparent) = black

I may just have to go without TR for now, this mod is incredible, thank you so much..

Edited May 5, 2014 by Yarbrough08

[Proot]

Sadly I want this precisely as TR complement, to use it in KSPRC. So I'm stucked at the same point as always...

[Guest]

Those should be memory crashes nevertheless. As mentioned does LOD currently can use quite some RAM while loading images... and verifying it is indeed memory related isn't that easy. The safest way to do so is via crash dumps, but i definitively won't let anyone upload more than a GB of data. Sharing a save VMMap data file for the crashed KSP.exe might be the best way to do so.

Strange enough, memory usage doesn't seem to go up when using LOD, in fact, it's quite a way down from normal. It crashes around 64% usage at times. I'll try VMMap, though.

[Faark]

I'm afraid in my case is not (I have all the dependencies)...

My LOD log (as least now I have logs!) : https://mega.co.nz/#!soMxTQ7C!vIsKcfIrVaQiqOXeidU-gjgdSLTwozJC1sdNjKwl8ZE

Most of textures appears as pink/black pattern, randomly some of them loads well, but the normal maps are wrong too.

BTW, I have now a huge amout of .THUMB files...

Hm, thats unfortunate, i was pretty sure someone reported TR as working. This log didn't contain anything unusual, i'm afraid (*edit* actually it did, i just didn't notice since for some reason it didn't end up in an error as it did on my test install). Setting up a test install with KSPRC right now.

Update: Currently TR is only kinda supported. You currently cant use TR to replace part textures and you also have to disabled mipmap generation. Yes, thats not rly what you want and i'll think about how to improve compatibility. But so far even without the crash LOD cannot just simply load TRs textures, but i guess thats what it should aim for.

Also, what does this do with normal maps? It seems like they are not being recognized as normals anymore (more like a black diffuse applied on top of the rest). Might need to handle those textures differently, or exclude them altogether.

Edit: My apologies; it may not be the normal, could be something with the alpha layer... 0 Alpha(fully transparent) = black

LOD has some problems with transparency, but usually only on those "unloaded" thumbnails textures. If its TR related i'll might find it when testing KSPRC, otherwise some more information on exactly how to reproduce it would be helpful...

Edited May 5, 2014 by Faark

[Guest]

Now that I think of it, would it be possible to include TR functionality with LOD? Something like a config to redirect the loader from one texture to another. I have no idea how difficult would that be, but you're already handling texture loading, so it seems that it should be possible to redirect the loader on "preparing textures" step.

Edited May 5, 2014 by Guest

[Yarbrough08]

LOD has some problems with transparency, but usually only on those "unloaded" thumbnails textures. If its TR related i'll might find it when testing KSPRC, otherwise some more information on exactly how to reproduce it would be helpful...

This was not TR related.. To reproduce just install ALCOR; the top, bottom, hatch, and small ring along the bottom are all black. Easiest way to reproduce...

Edited May 5, 2014 by Yarbrough08

[Faark]

@Dragon01

Possible for sure, but we'll better keep the scope of this project on loading textures. I think the best solution for now is to create an LOD API other mods could use (sth i would have done anyway at some point) and provide a lod compatible fork of TR.

@Yarbrough08

Thanks, i'll have a look at it.

[Guest]

I'm really looking forward to the next version, Faark. It's absolutely essential for me now. No more running out of memory even when running with half the Spaceport installed at the same time.