blackrack

[WIP][1.6.1] Scatterer-atmospheric scattering (v0.0540 - 17/03/2019) FIXED TSUNAMI BUG !!!1!1!

Recommended Posts

12 hours ago, blackrack said:

Switch to dx11 or disable terrain shadows.

Thanks that did it.  

Next question:

What other mods work great with scatterer for the best visuals - I've seen some really fantastic screenshots and mine look nothing like that.  

Share this post


Link to post
Share on other sites

I didn't look too carefully, but I'm assuming that the large arrays causing the memory issues are the big Color arrays used to set some texture.

There are a few things that might be of interest regarding Color vs Color32:

Using Color32 rather than Color saves a lot of memory. Since a Color object uses 4*32 bit floats, it takes up four times the memory of a Color32 object, which uses 4*8 bit bytes.

If the end result is to apply this Color array to a texture then there is no difference in the resulting texture colors as Unity converts all Color references to Color32.

Using Texture.SetPixels is considerably slower than Texture.SetPixels32; that internal Color -> Color32 conversion is actually quite slow, so cutting out that step can speed things up. The only drawback is that in Unity 4.x you have to set the entire texture's colors when using SetPixels32, whereas SetPixels allows for setting any arbitrary number of pixels; this is fixed in Unity 5; it looks like this isn't a problem here.

The only issue that I can think of is about the source of the colors in the first place. If they are all float Color values then you would have to convert them to Color32 and so would lose some of the speed advantage.

Share this post


Link to post
Share on other sites
1 hour ago, DMagic said:

I didn't look too carefully, but I'm assuming that the large arrays causing the memory issues are the big Color arrays used to set some texture.

There are a few things that might be of interest regarding Color vs Color32:

Using Color32 rather than Color saves a lot of memory. Since a Color object uses 4*32 bit floats, it takes up four times the memory of a Color32 object, which uses 4*8 bit bytes.

If the end result is to apply this Color array to a texture then there is no difference in the resulting texture colors as Unity converts all Color references to Color32.

Using Texture.SetPixels is considerably slower than Texture.SetPixels32; that internal Color -> Color32 conversion is actually quite slow, so cutting out that step can speed things up. The only drawback is that in Unity 4.x you have to set the entire texture's colors when using SetPixels32, whereas SetPixels allows for setting any arbitrary number of pixels; this is fixed in Unity 5; it looks like this isn't a problem here.

The only issue that I can think of is about the source of the colors in the first place. If they are all float Color values then you would have to convert them to Color32 and so would lose some of the speed advantage.

I create a big color array and a bunch of other float arrays as a preliminary step to loading a floating point texture from disk into a rendertexture. Since unity 4 doesn't support 32-bit per channel floating point textures outside of rendertextures the only way to do this is to read the file and encode it into 4 separate 8-bit per channel texture2D textures and then use a shader to read everything and decode it back to a 32-bit rendertexture. This is only done when switching scenes and after that I need all the arrays and texture2D textures gone ASAP. The issue is that the collector never gets rid of the Color and float arrays so although Color32 arrays would help it would still leak a bunch of memory. To get around this I used the Marshal class from System.Runtime.InteropServices to allocate some memory and then manage and free it manually, it seems to work quite nicely and saves up a lot of memory.

Although using a Color32 array could limit the size of the "memory spike" during loading by a bit so I'll do this. For your tip about SetPixels32, I would indeed need to convert float values to Color32 (or 1 byte floating point number, what's that called? a minifloat?) so in the end probably like you said there wouldn't be a speed advantage. To clarify the source is not Color values but rather their float components, calculated by the encode function.

2 hours ago, Oddible said:

Thanks that did it.  

Next question:

What other mods work great with scatterer for the best visuals - I've seen some really fantastic screenshots and mine look nothing like that.  

You could download EVE which adds clouds or get one of the visual packs (e.g SVE, KSPRC) which include EVE with custom cloud configs and other custom textures and miscellaneous visual mods.

52 minutes ago, sDaZe said:

New eclipses looking good :D 

Finally, a screenshot with something other than lens flare, thank you :P

Edited by blackrack

Share this post


Link to post
Share on other sites
20 minutes ago, blackrack said:

 

Finally, a screenshot with something other than lens flare, thank you :P

hehe np :) 

Share this post


Link to post
Share on other sites

These parts I just decoupled I think are leaving god rays

UyiYGne.png

I shouldn't have F2'd before taking the screenshot or you would see the debris markers, so there's some unnecessary calculation going on there

Edited by selfish_meme

Share this post


Link to post
Share on other sites
2 hours ago, selfish_meme said:

These parts I just decoupled I think are leaving god rays

To be honest, I've been getting these, too, lately.  As I'm testing the new .dll for memory usage, I'll see if I can capture the effect.  Is there something in particular you want to see tested, or just memory consumption according to GCMonitor?

Edit: Test Results

KSP 1.0.5 32-bit, Win7 64-bit, 12GB RAM, Scatterer options enabled: Lens Flare, Eclipse, Godrays.  Ocean shaders don't work well for me and my older gfx card.

KSP commandline: -popupwindow -single-instance -force-opengl

The game save was restored between the tests so it was the same starting point.  I started the game, went to the KSC, VAB, KSC, TS and a craft in Kerbin orbit, in sequence.

dll Version Start KSC VAB KSC TS Craft
1.0.5900.13398 2351 2823 2906 2944 2904 2996
1.0.5905.12084 2342 2714 2694 2855 2815 2892

The only anomaly was after I replaced the .dll and started the game, the Universal Storage mod parts were not recognized on getting to the KSC screen.  This happens to me occasionally when I update mods.  When it does, I clear the temporary Module Manager files to force it to rebuild its database.  The numbers above for the new dll run are for a post-rebuild game start.

Screen changes to enter the KSC (from a craft or the VAB) and to load a craft from the TS seemed longer but I didn't time them.

Finally, Scatterer doesn't show up on the KSP-AVC installed mods list.  Never noticed that before.  Of course, it appears in CKAN.

Any memory saving is good :-)

Edited by Brigadier
Added test results

Share this post


Link to post
Share on other sites
12 hours ago, selfish_meme said:

These parts I just decoupled I think are leaving god rays

I shouldn't have F2'd before taking the screenshot or you would see the debris markers, so there's some unnecessary calculation going on there

11 hours ago, Brigadier said:

To be honest, I've been getting these, too, lately.  As I'm testing the new .dll for memory usage, I'll see if I can capture the effect.  Is there something in particular you want to see tested, or just memory consumption according to GCMonitor?

Edit: Test Results

KSP 1.0.5 32-bit, Win7 64-bit, 12GB RAM, Scatterer options enabled: Lens Flare, Eclipse, Godrays.  Ocean shaders don't work well for me and my older gfx card.

KSP commandline: -popupwindow -single-instance -force-opengl

The game save was restored between the tests so it was the same starting point.  I started the game, went to the KSC, VAB, KSC, TS and a craft in Kerbin orbit, in sequence.

dll Version Start KSC VAB KSC TS Craft
1.0.5900.13398 2351 2823 2906 2944 2904 2996
1.0.5905.12084 2342 2714 2694 2855 2815 2892

The only anomaly was after I replaced the .dll and started the game, the Universal Storage mod parts were not recognized on getting to the KSC screen.  This happens to me occasionally when I update mods.  When it does, I clear the temporary Module Manager files to force it to rebuild its database.  The numbers above for the new dll run are for a post-rebuild game start.

Screen changes to enter the KSC (from a craft or the VAB) and to load a craft from the TS seemed longer but I didn't time them.

Finally, Scatterer doesn't show up on the KSP-AVC installed mods list.  Never noticed that before.  Of course, it appears in CKAN.

Any memory saving is good :-)

For the godrays it's a limitation of the current method, check the first entry in the known issues section in the OP.

Thanks for the test results.

 

Edited:

I pushed the memory optimizations even further, I believe the numbers speak for themselves.

vyu8I08.png

As with the previous one, this might be a bit dodgy so please test it out and report on whether it works or not and what kind of memory savings you get from it. If you have the 64bit workaround on windows also do test with it.

https://github.com/LGhassen/Scatterer/raw/improvedMemoryManagement/scatterer/bin/Release/scatterer.dll

Paging the people who tested the previous version: @Brigadier  @SkyKaptn @selfish_meme

I believe the loading process should be near optimal now, there might still be some "per-frame" leaks which I will check out next.

If this works for everyone this should be great news for people with a memory budget and people on 32bit.

Edited by blackrack

Share this post


Link to post
Share on other sites

for some reason (idk why probably the amount of mods i have though) i crash on startup with this mod, it really makes me sad

ill try to link a error report if possible.

Share this post


Link to post
Share on other sites

Windows 8 64bit, KSP 1-0-5 32bit
OpenGL forced
This is what GCmonitor shows to me:

  Original dll Newest dll
KSC 3217 3143
Runway 3513 3214
In orbit 3756 3294

Haven't noticed any extra loading time. I pretty much like it :)

EDIT: And also, as I have already mentioned, I like the Sun...
yv50EUH.png

Edited by Paul_Sawyer

Share this post


Link to post
Share on other sites

Love the work!

I've been using scatterer for a while now, stuck with it when the ocean was a Z-fighting (no not the Goku kind) mess too.
Latest release is simply beautiful too. I haven't noticed any bugs additional to the ones already reported.. though I was wondering, if it is possible for you to tell, what the release of 1.1 will have on the mod? Are you expecting a million new bugs/unplayable until you update scatterer or will the shift between Unity variants not impact your graphical work so much?

Share this post


Link to post
Share on other sites

Under the same test conditions (exactly the same mods but a later game save - I've moved on).  Previous test result included for comparison:

dll Version Start KSC VAB KSC TS Vessel
1.0.5905.12084 2342 2714 2694 2855 2815 2892
1.0.5609.26962 2324 2579 2600 2737 2698 2748

Nice improvement.

Ideally, I should run this scenario a couple of time to get an average but...well...I actually want to play a game :wink:

Share this post


Link to post
Share on other sites
2 hours ago, JDarksword said:

for some reason (idk why probably the amount of mods i have though) i crash on startup with this mod, it really makes me sad

ill try to link a error report if possible.

It's probably your memory usage going over the 32bit limit, try to remove all other mods and see if it works, also try the new .dll linked above, it does wonders for the memory usage.

52 minutes ago, MR L A said:

Love the work!

I've been using scatterer for a while now, stuck with it when the ocean was a Z-fighting (no not the Goku kind) mess too.
Latest release is simply beautiful too. I haven't noticed any bugs additional to the ones already reported.. though I was wondering, if it is possible for you to tell, what the release of 1.1 will have on the mod? Are you expecting a million new bugs/unplayable until you update scatterer or will the shift between Unity variants not impact your graphical work so much?

Of course when 1.1 releases nothing will work until I update the mod, but I'm not expecting any significant difficulties in making it run on 1.1

1 hour ago, Paul_Sawyer said:

Windows 8 64bit, KSP 1-0-5 32bit
OpenGL forced
This is what GCmonitor shows to me:

  Original dll Newest dll
KSC 3217 3143
Runway 3513 3214
In orbit 3756 3294

Haven't noticed any extra loading time. I pretty much like it :)

EDIT: And also, as I have already mentioned, I like the Sun...

24 minutes ago, Brigadier said:

Under the same test conditions (exactly the same mods but a later game save - I've moved on).  Previous test result included for comparison:

dll Version Start KSC VAB KSC TS Vessel
1.0.5905.12084 2342 2714 2694 2855 2815 2892
1.0.5609.26962 2324 2579 2600 2737 2698 2748

Nice improvement.

Ideally, I should run this scenario a couple of time to get an average but...well...I actually want to play a game :wink:

Great, thank you both! Now all I need is some reports that it also works fine on linux and mac.

 

Share this post


Link to post
Share on other sites

So what's this I hear about manual buffer alloc/dealloc? I use color arrays in DTL, and may be curious to know if that might have a memory leak from loading textures?

Share this post


Link to post
Share on other sites
1 hour ago, rbray89 said:

So what's this I hear about manual buffer alloc/dealloc? I use color arrays in DTL, and may be curious to know if that might have a memory leak from loading textures?

I use large arrays as a preliminary step to get data from disk into a 32-bit per channel rendertexture. I noticed that large color and float arrays don't seem to be freed after they are destroyed or go out of scope and the collector is called, I made sure there were no remaining references or anything. I only need to keep the 16mb rendertexture after processing but every time the memory usage would balloon up by 300mb or so. I'm not sure what exactly was going on, some people will tell you that the garbage collector is itself garbage and doesn't manage arrays well etc etc... in the end the only thing that helped was using System.Runtime.InteropServices to allocate some unmanaged memory, and then manage it manually as a float or color buffer and free it when I'm done, works like a charm (sorry source code is a bit of a mess right now as I mostly left he old code commented and there are bits and pieces everywhere). I've also had to avoid functions that return arrays in general, for example BitConverter.getBytes() was also causing some leaks along a few other things, when used in loops those were pretty bad.

One peculiar behaviour that I observed is that these leaks don't go on forever with every scene change or operation but seem to top out at 500 mb or so, I've read somewhere that arrays are managed differently in memory, so maybe 500 mb is some kind of limit at which the collector starts doing something for the arrays but 500 mb is still way too much for KSP. Don't know, maybe you could shed some light on this.

You could isolate the color arrays and check their impact on memory, and whether destroying and creating new ones will keep inflating the memory usage. In my case the combined size of temporary arrays was a bit big so having them keep leaking on scene changes was really bad.

48 minutes ago, Nansuchao said:

That's pure poetry. Thanks again, Blackrack

You're welcome :P

Edited by blackrack

Share this post


Link to post
Share on other sites

Btw, assuming that the body is perfectly circular isn't always working - sun flare is visible through Gilly, for example. Would it be possible to check it against body shape mor eprecisely in later versions? Maybe it isn't needed for all bodies, as large ones can be assumed to be perfect spheres.

Share this post


Link to post
Share on other sites
15 minutes ago, sashan said:

Btw, assuming that the body is perfectly circular isn't always working - sun flare is visible through Gilly, for example. Would it be possible to check it against body shape mor eprecisely in later versions? Maybe it isn't needed for all bodies, as large ones can be assumed to be perfect spheres.

Why do you think I'm assuming spheres for the lens flare? It works via raycasting and checking the depth buffer, but I guess the depth buffer isn't rendered for planets that don't have scattering effects, I'll fix it for next updates.

Share this post


Link to post
Share on other sites
Just now, blackrack said:

Why do you think I'm assuming spheres for the lens flare? It works via raycasting and checking the depth buffer, but I guess the depth buffer isn't rendered for planets that don't have scattering effects, I'll fix it for next updates.

Because that's what it looks like on GIlly - there's an invisible sphere with surface at 0m altitude that blocks the flare, and large hills that stick out don't block it. So, I guess your guess is right. Anyway, I had to report it. :P

Share this post


Link to post
Share on other sites
1 minute ago, sashan said:

Because that's what it looks like on GIlly - there's an invisible sphere with surface at 0m altitude that blocks the flare, and large hills that stick out don't block it. So, I guess your guess is right. Anyway, I had to report it. :P

Well it works with the mountains on Kerbin so that's not really it :P Anyway, I'm guessing that's Just raycasting in scaledSpace working, but not quite matching up to the PQS. I had the same problem on Kerbin before I decided to just use the depth buffer. I'll make it so the depth buffer renders for checking occlusion even on planets without scattering effects, perhaps with a reduced resolution in this case.

Share this post


Link to post
Share on other sites
8 minutes ago, blackrack said:

Well it works with the mountains on Kerbin so that's not really it

They are too small and rare to notice it easily.
Btw, it works while landed - it's only scaled space and orbit view that are buggy. I'll test it more tomorrow - I have a base on Gilly.

Oh, and you need to change Eve setup - it's kinda broken from certain angles.
7wi6CGY.png
It looks great from low altitudes tho:
12823433_860848314037835_821158690191566

Edited by sashan

Share this post


Link to post
Share on other sites

[12:56] <Theysen> stratochief|away, so the parameters editing he showed result in a much brighter and wider sunflare / spiking on the ground, but the actual problem is that once you pass a certain altitude the sun is just a tiny little dot for me.

@blackrack I had the same experience yesterday, but was too sick/tired to make that connection. above somewhere in 90-140km range the sunflare just disappears with RSS. not that I expect you to fix it, I just wanted you to have a better description of the issue we are seeing. your scaling works well in the atmosphere, but seems to cause no sunflare to be applied when out in space

Share this post


Link to post
Share on other sites
5 minutes ago, stratochief66 said:

[12:56] <Theysen> stratochief|away, so the parameters editing he showed result in a much brighter and wider sunflare / spiking on the ground, but the actual problem is that once you pass a certain altitude the sun is just a tiny little dot for me.

@blackrack I had the same experience yesterday, but was too sick/tired to make that connection. above somewhere in 90-140km range the sunflare just disappears with RSS. not that I expect you to fix it, I just wanted you to have a better description of the issue we are seeing. your scaling works well in the atmosphere, but seems to cause no sunflare to be applied when out in space

Does it still work in mapview?

Share this post


Link to post
Share on other sites
22 minutes ago, blackrack said:

Does it still work in mapview?

I actually just made some screens during ascent. The first picture shows the default sunflare.cfg values, then I exaggerated it and set them to 0.0011 but it didn't change anything. 

Mapview flaring doesn't work either. I also tried upping the value of sunglare scale in the settings.cfg but this didn't change the size of the sun. 

 

I know your disclaimer says current RSS/Rescale might not work (besides the heavy z-fighting), but maybe this is of help. 

 

Mods used:   Stock + RSS only (so it loads as fast as unmodded install basically) + scatterer 

Log (if needed): https://www.dropbox.com/s/rtg05gwlp5y552k/output_log.txt?dl=0

 

 

 

 

Share this post


Link to post
Share on other sites
5 minutes ago, Theysen said:

I actually just made some screens during ascent. The first picture shows the default sunflare.cfg values, then I exaggerated it and set them to 0.0011 but it didn't change anything. 

Mapview flaring doesn't work either. I also tried upping the value of sunglare scale in the settings.cfg but this didn't change the size of the sun. 

I know your disclaimer says current RSS/Rescale might not work (besides the heavy z-fighting), but maybe this is of help. 

Mods used:   Stock + RSS only (so it loads as fast as unmodded install basically) + scatterer 

Log (if needed): https://www.dropbox.com/s/rtg05gwlp5y552k/output_log.txt?dl=0

Does mapview ever work at all? Which version of the mod are you on?

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.