[WIP][1.9.x-1.12.x] Scatterer-atmospheric scattering (0.0838 - 14/08/2022) Scattering improvements, in-game atmo generation and multi-sun support

[Oddible]

12 hours ago, blackrack said:

Switch to dx11 or disable terrain shadows.

Thanks that did it.  

Next question:

What other mods work great with scatterer for the best visuals - I've seen some really fantastic screenshots and mine look nothing like that.  

[DMagic]

I didn't look too carefully, but I'm assuming that the large arrays causing the memory issues are the big Color arrays used to set some texture.

There are a few things that might be of interest regarding Color vs Color32:

Using Color32 rather than Color saves a lot of memory. Since a Color object uses 4*32 bit floats, it takes up four times the memory of a Color32 object, which uses 4*8 bit bytes.

If the end result is to apply this Color array to a texture then there is no difference in the resulting texture colors as Unity converts all Color references to Color32.

Using Texture.SetPixels is considerably slower than Texture.SetPixels32; that internal Color -> Color32 conversion is actually quite slow, so cutting out that step can speed things up. The only drawback is that in Unity 4.x you have to set the entire texture's colors when using SetPixels32, whereas SetPixels allows for setting any arbitrary number of pixels; this is fixed in Unity 5; it looks like this isn't a problem here.

The only issue that I can think of is about the source of the colors in the first place. If they are all float Color values then you would have to convert them to Color32 and so would lose some of the speed advantage.

[sDaZe]

New eclipses looking good  

[blackrack]

1 hour ago, DMagic said:

I didn't look too carefully, but I'm assuming that the large arrays causing the memory issues are the big Color arrays used to set some texture.

There are a few things that might be of interest regarding Color vs Color32:

Using Color32 rather than Color saves a lot of memory. Since a Color object uses 4*32 bit floats, it takes up four times the memory of a Color32 object, which uses 4*8 bit bytes.

If the end result is to apply this Color array to a texture then there is no difference in the resulting texture colors as Unity converts all Color references to Color32.

Using Texture.SetPixels is considerably slower than Texture.SetPixels32; that internal Color -> Color32 conversion is actually quite slow, so cutting out that step can speed things up. The only drawback is that in Unity 4.x you have to set the entire texture's colors when using SetPixels32, whereas SetPixels allows for setting any arbitrary number of pixels; this is fixed in Unity 5; it looks like this isn't a problem here.

The only issue that I can think of is about the source of the colors in the first place. If they are all float Color values then you would have to convert them to Color32 and so would lose some of the speed advantage.

I create a big color array and a bunch of other float arrays as a preliminary step to loading a floating point texture from disk into a rendertexture. Since unity 4 doesn't support 32-bit per channel floating point textures outside of rendertextures the only way to do this is to read the file and encode it into 4 separate 8-bit per channel texture2D textures and then use a shader to read everything and decode it back to a 32-bit rendertexture. This is only done when switching scenes and after that I need all the arrays and texture2D textures gone ASAP. The issue is that the collector never gets rid of the Color and float arrays so although Color32 arrays would help it would still leak a bunch of memory. To get around this I used the Marshal class from System.Runtime.InteropServices to allocate some memory and then manage and free it manually, it seems to work quite nicely and saves up a lot of memory.

Although using a Color32 array could limit the size of the "memory spike" during loading by a bit so I'll do this. For your tip about SetPixels32, I would indeed need to convert float values to Color32 (or 1 byte floating point number, what's that called? a minifloat?) so in the end probably like you said there wouldn't be a speed advantage. To clarify the source is not Color values but rather their float components, calculated by the encode function.

2 hours ago, Oddible said:

Thanks that did it.  

Next question:

What other mods work great with scatterer for the best visuals - I've seen some really fantastic screenshots and mine look nothing like that.  

You could download EVE which adds clouds or get one of the visual packs (e.g SVE, KSPRC) which include EVE with custom cloud configs and other custom textures and miscellaneous visual mods.

52 minutes ago, sDaZe said:

New eclipses looking good  

Finally, a screenshot with something other than lens flare, thank you

Edited March 2, 2016 by blackrack

[sDaZe]

20 minutes ago, blackrack said:

 

Finally, a screenshot with something other than lens flare, thank you

hehe np  

[selfish_meme]

These parts I just decoupled I think are leaving god rays

I shouldn't have F2'd before taking the screenshot or you would see the debris markers, so there's some unnecessary calculation going on there

Edited March 3, 2016 by selfish_meme

[Brigadier]

2 hours ago, selfish_meme said:

These parts I just decoupled I think are leaving god rays

To be honest, I've been getting these, too, lately.  As I'm testing the new .dll for memory usage, I'll see if I can capture the effect.  Is there something in particular you want to see tested, or just memory consumption according to GCMonitor?

Edit: Test Results

KSP 1.0.5 32-bit, Win7 64-bit, 12GB RAM, Scatterer options enabled: Lens Flare, Eclipse, Godrays.  Ocean shaders don't work well for me and my older gfx card.

KSP commandline: -popupwindow -single-instance -force-opengl

The game save was restored between the tests so it was the same starting point.  I started the game, went to the KSC, VAB, KSC, TS and a craft in Kerbin orbit, in sequence.

dll Version	Start	KSC	VAB	KSC	TS	Craft
1.0.5900.13398	2351	2823	2906	2944	2904	2996
1.0.5905.12084	2342	2714	2694	2855	2815	2892

The only anomaly was after I replaced the .dll and started the game, the Universal Storage mod parts were not recognized on getting to the KSC screen.  This happens to me occasionally when I update mods.  When it does, I clear the temporary Module Manager files to force it to rebuild its database.  The numbers above for the new dll run are for a post-rebuild game start.

Screen changes to enter the KSC (from a craft or the VAB) and to load a craft from the TS seemed longer but I didn't time them.

Finally, Scatterer doesn't show up on the KSP-AVC installed mods list.  Never noticed that before.  Of course, it appears in CKAN.

Any memory saving is good :-)

Edited March 3, 2016 by Brigadier
Added test results

[blackrack]

12 hours ago, selfish_meme said:

These parts I just decoupled I think are leaving god rays

I shouldn't have F2'd before taking the screenshot or you would see the debris markers, so there's some unnecessary calculation going on there

11 hours ago, Brigadier said:

To be honest, I've been getting these, too, lately.  As I'm testing the new .dll for memory usage, I'll see if I can capture the effect.  Is there something in particular you want to see tested, or just memory consumption according to GCMonitor?

Edit: Test Results

KSP 1.0.5 32-bit, Win7 64-bit, 12GB RAM, Scatterer options enabled: Lens Flare, Eclipse, Godrays.  Ocean shaders don't work well for me and my older gfx card.

KSP commandline: -popupwindow -single-instance -force-opengl

The game save was restored between the tests so it was the same starting point.  I started the game, went to the KSC, VAB, KSC, TS and a craft in Kerbin orbit, in sequence.

dll Version	Start	KSC	VAB	KSC	TS	Craft
1.0.5900.13398	2351	2823	2906	2944	2904	2996
1.0.5905.12084	2342	2714	2694	2855	2815	2892

The only anomaly was after I replaced the .dll and started the game, the Universal Storage mod parts were not recognized on getting to the KSC screen.  This happens to me occasionally when I update mods.  When it does, I clear the temporary Module Manager files to force it to rebuild its database.  The numbers above for the new dll run are for a post-rebuild game start.

Screen changes to enter the KSC (from a craft or the VAB) and to load a craft from the TS seemed longer but I didn't time them.

Finally, Scatterer doesn't show up on the KSP-AVC installed mods list.  Never noticed that before.  Of course, it appears in CKAN.

Any memory saving is good :-)

For the godrays it's a limitation of the current method, check the first entry in the known issues section in the OP.

Thanks for the test results.

 

Edited:

I pushed the memory optimizations even further, I believe the numbers speak for themselves.

As with the previous one, this might be a bit dodgy so please test it out and report on whether it works or not and what kind of memory savings you get from it. If you have the 64bit workaround on windows also do test with it.

https://github.com/LGhassen/Scatterer/raw/improvedMemoryManagement/scatterer/bin/Release/scatterer.dll

Paging the people who tested the previous version: @Brigadier  @SkyKaptn @selfish_meme

I believe the loading process should be near optimal now, there might still be some "per-frame" leaks which I will check out next.

If this works for everyone this should be great news for people with a memory budget and people on 32bit.

Edited March 3, 2016 by blackrack

[JDarksword]

for some reason (idk why probably the amount of mods i have though) i crash on startup with this mod, it really makes me sad

ill try to link a error report if possible.

[Paul_Sawyer]

Windows 8 64bit, KSP 1-0-5 32bit
OpenGL forced
This is what GCmonitor shows to me:

	Original dll	Newest dll
KSC	3217	3143
Runway	3513	3214
In orbit	3756	3294

Haven't noticed any extra loading time. I pretty much like it

EDIT: And also, as I have already mentioned, I like the Sun...

Edited March 3, 2016 by Paul_Sawyer

[MR L A]

Love the work!

I've been using scatterer for a while now, stuck with it when the ocean was a Z-fighting (no not the Goku kind) mess too.
Latest release is simply beautiful too. I haven't noticed any bugs additional to the ones already reported.. though I was wondering, if it is possible for you to tell, what the release of 1.1 will have on the mod? Are you expecting a million new bugs/unplayable until you update scatterer or will the shift between Unity variants not impact your graphical work so much?

[Brigadier]

Under the same test conditions (exactly the same mods but a later game save - I've moved on).  Previous test result included for comparison:

dll Version	Start	KSC	VAB	KSC	TS	Vessel
1.0.5905.12084	2342	2714	2694	2855	2815	2892
1.0.5609.26962	2324	2579	2600	2737	2698	2748

Nice improvement.

Ideally, I should run this scenario a couple of time to get an average but...well...I actually want to play a game

[blackrack]

2 hours ago, JDarksword said:

for some reason (idk why probably the amount of mods i have though) i crash on startup with this mod, it really makes me sad

ill try to link a error report if possible.

It's probably your memory usage going over the 32bit limit, try to remove all other mods and see if it works, also try the new .dll linked above, it does wonders for the memory usage.

52 minutes ago, MR L A said:

Love the work!

I've been using scatterer for a while now, stuck with it when the ocean was a Z-fighting (no not the Goku kind) mess too.
Latest release is simply beautiful too. I haven't noticed any bugs additional to the ones already reported.. though I was wondering, if it is possible for you to tell, what the release of 1.1 will have on the mod? Are you expecting a million new bugs/unplayable until you update scatterer or will the shift between Unity variants not impact your graphical work so much?

Of course when 1.1 releases nothing will work until I update the mod, but I'm not expecting any significant difficulties in making it run on 1.1

1 hour ago, Paul_Sawyer said:

Windows 8 64bit, KSP 1-0-5 32bit
OpenGL forced
This is what GCmonitor shows to me:

	Original dll	Newest dll
KSC	3217	3143
Runway	3513	3214
In orbit	3756	3294

Haven't noticed any extra loading time. I pretty much like it

EDIT: And also, as I have already mentioned, I like the Sun...

24 minutes ago, Brigadier said:

Under the same test conditions (exactly the same mods but a later game save - I've moved on).  Previous test result included for comparison:

dll Version	Start	KSC	VAB	KSC	TS	Vessel
1.0.5905.12084	2342	2714	2694	2855	2815	2892
1.0.5609.26962	2324	2579	2600	2737	2698	2748

Nice improvement.

Ideally, I should run this scenario a couple of time to get an average but...well...I actually want to play a game

Great, thank you both! Now all I need is some reports that it also works fine on linux and mac.

 

[rbray89]

So what's this I hear about manual buffer alloc/dealloc? I use color arrays in DTL, and may be curious to know if that might have a memory leak from loading textures?

[Nansuchao]

That's pure poetry. Thanks again, Blackrack

 

[blackrack]

1 hour ago, rbray89 said:

So what's this I hear about manual buffer alloc/dealloc? I use color arrays in DTL, and may be curious to know if that might have a memory leak from loading textures?

I use large arrays as a preliminary step to get data from disk into a 32-bit per channel rendertexture. I noticed that large color and float arrays don't seem to be freed after they are destroyed or go out of scope and the collector is called, I made sure there were no remaining references or anything. I only need to keep the 16mb rendertexture after processing but every time the memory usage would balloon up by 300mb or so. I'm not sure what exactly was going on, some people will tell you that the garbage collector is itself garbage and doesn't manage arrays well etc etc... in the end the only thing that helped was using System.Runtime.InteropServices to allocate some unmanaged memory, and then manage it manually as a float or color buffer and free it when I'm done, works like a charm (sorry source code is a bit of a mess right now as I mostly left he old code commented and there are bits and pieces everywhere). I've also had to avoid functions that return arrays in general, for example BitConverter.getBytes() was also causing some leaks along a few other things, when used in loops those were pretty bad.

One peculiar behaviour that I observed is that these leaks don't go on forever with every scene change or operation but seem to top out at 500 mb or so, I've read somewhere that arrays are managed differently in memory, so maybe 500 mb is some kind of limit at which the collector starts doing something for the arrays but 500 mb is still way too much for KSP. Don't know, maybe you could shed some light on this.

You could isolate the color arrays and check their impact on memory, and whether destroying and creating new ones will keep inflating the memory usage. In my case the combined size of temporary arrays was a bit big so having them keep leaking on scene changes was really bad.

48 minutes ago, Nansuchao said:

That's pure poetry. Thanks again, Blackrack

You're welcome

Edited March 3, 2016 by blackrack

[sashan]

Btw, assuming that the body is perfectly circular isn't always working - sun flare is visible through Gilly, for example. Would it be possible to check it against body shape mor eprecisely in later versions? Maybe it isn't needed for all bodies, as large ones can be assumed to be perfect spheres.

[blackrack]

15 minutes ago, sashan said:

Btw, assuming that the body is perfectly circular isn't always working - sun flare is visible through Gilly, for example. Would it be possible to check it against body shape mor eprecisely in later versions? Maybe it isn't needed for all bodies, as large ones can be assumed to be perfect spheres.

Why do you think I'm assuming spheres for the lens flare? It works via raycasting and checking the depth buffer, but I guess the depth buffer isn't rendered for planets that don't have scattering effects, I'll fix it for next updates.

[sashan]

Just now, blackrack said:

Why do you think I'm assuming spheres for the lens flare? It works via raycasting and checking the depth buffer, but I guess the depth buffer isn't rendered for planets that don't have scattering effects, I'll fix it for next updates.

Because that's what it looks like on GIlly - there's an invisible sphere with surface at 0m altitude that blocks the flare, and large hills that stick out don't block it. So, I guess your guess is right. Anyway, I had to report it.

[blackrack]

1 minute ago, sashan said:

Because that's what it looks like on GIlly - there's an invisible sphere with surface at 0m altitude that blocks the flare, and large hills that stick out don't block it. So, I guess your guess is right. Anyway, I had to report it.

Well it works with the mountains on Kerbin so that's not really it  Anyway, I'm guessing that's Just raycasting in scaledSpace working, but not quite matching up to the PQS. I had the same problem on Kerbin before I decided to just use the depth buffer. I'll make it so the depth buffer renders for checking occlusion even on planets without scattering effects, perhaps with a reduced resolution in this case.

[sashan]

8 minutes ago, blackrack said:

Well it works with the mountains on Kerbin so that's not really it

They are too small and rare to notice it easily.
Btw, it works while landed - it's only scaled space and orbit view that are buggy. I'll test it more tomorrow - I have a base on Gilly.

Oh, and you need to change Eve setup - it's kinda broken from certain angles.

It looks great from low altitudes tho:

Edited March 3, 2016 by sashan

[stratochief66]

[12:56] <Theysen> stratochief|away, so the parameters editing he showed result in a much brighter and wider sunflare / spiking on the ground, but the actual problem is that once you pass a certain altitude the sun is just a tiny little dot for me.

@blackrack I had the same experience yesterday, but was too sick/tired to make that connection. above somewhere in 90-140km range the sunflare just disappears with RSS. not that I expect you to fix it, I just wanted you to have a better description of the issue we are seeing. your scaling works well in the atmosphere, but seems to cause no sunflare to be applied when out in space

[blackrack]

5 minutes ago, stratochief66 said:

[12:56] <Theysen> stratochief|away, so the parameters editing he showed result in a much brighter and wider sunflare / spiking on the ground, but the actual problem is that once you pass a certain altitude the sun is just a tiny little dot for me.

@blackrack I had the same experience yesterday, but was too sick/tired to make that connection. above somewhere in 90-140km range the sunflare just disappears with RSS. not that I expect you to fix it, I just wanted you to have a better description of the issue we are seeing. your scaling works well in the atmosphere, but seems to cause no sunflare to be applied when out in space

Does it still work in mapview?

[Theysen]

22 minutes ago, blackrack said:

Does it still work in mapview?

I actually just made some screens during ascent. The first picture shows the default sunflare.cfg values, then I exaggerated it and set them to 0.0011 but it didn't change anything. 

Mapview flaring doesn't work either. I also tried upping the value of sunglare scale in the settings.cfg but this didn't change the size of the sun. 

 

I know your disclaimer says current RSS/Rescale might not work (besides the heavy z-fighting), but maybe this is of help. 

 

Mods used:   Stock + RSS only (so it loads as fast as unmodded install basically) + scatterer 

Log (if needed): https://www.dropbox.com/s/rtg05gwlp5y552k/output_log.txt?dl=0

 

 

 

 

[blackrack]

5 minutes ago, Theysen said:

I actually just made some screens during ascent. The first picture shows the default sunflare.cfg values, then I exaggerated it and set them to 0.0011 but it didn't change anything. 

Mapview flaring doesn't work either. I also tried upping the value of sunglare scale in the settings.cfg but this didn't change the size of the sun. 

I know your disclaimer says current RSS/Rescale might not work (besides the heavy z-fighting), but maybe this is of help. 

Mods used:   Stock + RSS only (so it loads as fast as unmodded install basically) + scatterer 

Log (if needed): https://www.dropbox.com/s/rtg05gwlp5y552k/output_log.txt?dl=0

Does mapview ever work at all? Which version of the mod are you on?