[PLUGIN+PARTS][0.23] SCANsat terrain mapping

[DMagic]

Is it normal that there is no legend for the slope(I was in the big map and, yes, I activated the legend)?

The legend appear without problem for the altitude.

That's normal, same for the biome. There's not much for a legend to show for the slope map; solid puke green, no slope, yellow lines, slope, red lines, steep slope. Maybe it would be a good idea to remove the "legend" button if you're on the slope or biome map.

[Starstrider42]

Feature request for the slope map: can you make it so that, if it's based on low-res altimetry, it gives an average slope between the low-res cells? The current slope map treats the altimetry map as if it were actually a set of perfectly flat cells with infinitely steep edges, which is... not very helpful.

[futrtrubl]

Feature requests:

An option to show vehicles with MapTraq on the maps.

The ability to show the target vessel on the map if it is equipped with a scansat part (including MapTraq).

And a question:

How do you show biome data on the small map?

Edward

[Virtualgenius]

Lets stop asking for feature requests and get a stable workable version going, the guys are doing the best they can but being bombarded by feature requests isnt helping, its clogging the thread so lets cut the requests till version 6 is done dusted and installed

[technogeeky]

I'm happy to announce that it was straightforward for me to build SCANsatRPM myself, and I don't see a reason that I shouldn't just release it with SCANsat.

This version is built against RPM 0.16.

This is the version you want to get now if you use RPM. If it works properly, I will include this in all future SCANsat versions.

SCANsat v6rc2 [download]

SCANsat v6rc2 [source]

I am testing using the pretty damn amazing ALCOR capsule, and it seemed to work fine (though the big map is even slower than I remember! sadface!).

Let me know if I missed something.

[Ratzap]

In SCANsat.SCANsat you have a protected bool called 'scanning'. Could you add a method or something to expose that please? Or tell me if there's another way to see if a SCANsat part is currently using the amount of charge specified by SCANsat.SCANsat.power? Ta

[technogeeky]

Lets stop asking for feature requests and get a stable workable version going, the guys are doing the best they can but being bombarded by feature requests isnt helping, its clogging the thread so lets cut the requests till version 6 is done dusted and installed

If the people asking for these requests don't know, all feature requests have a lower priority than:

* repackaging to make a new official release

* debugging any new features (toolbar support, RPM support, ...)

* performance optimizations

So in that sense, you can feel free to ask (I might even store the request in the TODO.txt), but just know that they are going on some list and there they shall wait.

[technogeeky]

In SCANsat.SCANsat you have a protected bool called 'scanning'. Could you add a method or something to expose that please? Or tell me if there's another way to see if a SCANsat part is currently using the amount of charge specified by SCANsat.SCANsat.power? Ta

Since I recognized what you probably want it for, I've already added this feature. It's now in v6rc2.

[DMagic]

I've been doing some more testing of the big map performance.

For one thing I tested the alternative radial vector calculation using the Quaternion.AngleAxis thing, it made no difference in performance.

For the map caching, I have some concerns about memory usage because the array can be quite large when you start stretching the map out. Unity's garbage collection is apparently fairly lax, so every time you change the map size a new array is created in memory and the old remains. Eventually the GC kicks in and all of those old arrays get deleted (by stretching the map way off screen, I'm guessing to around 4000 pixels wide, you can make the memory usage jump by 200MB or so each time you change it), but I can see this causing problems in some cases. It's also not really consistent in how much gets deleted, I've seen the memory usage drop from around 3.4 to 1.3GB, but I've also seen drop to around 3GB and stay there.

That said, I don't think any of this is really a problem, at least not a new problem. This exact same thing already occurs whenever you resize the big map in v5 (and v6). I assume this is because the texture is also being recreated and taking up a lot of memory. The caching array exacerbates the problem, but I've never heard anyone complain about SCANsat causing KSP to run out of memory before. So if it wasn't causing fatal errors with just the texture taking up space, then I don't think the addition of the array will really change anything (unless you're already right on the line with memory usage, but that's a different issue and doesn't affect the majority of players).

Another reason I'm not too concerned is that for this to really be a problem requires stupidly huge map sizes and constant resizing of the map. When you stay below 1500 pixels or so wide it doesn't have such a huge memory hit. If you just pick a size and only occasionally change it you won't have a problem, just like no one that I know of ever reported having memory problems pre-v6.

There are some cases where I can understand using very large map sizes. I think they can be valuable for exporting high quality maps. Maybe it would be best to create a dedicated map export function, one where you can input a specific pixel width and one that doesn't cache the data. In other cases it might be a good idea to throw a warning (maybe just once) when someone stretches the map out too much or resizes it 20 or so times in less than a minute.

The performance impact is pretty large and I think worth the trade off. For reasonable size maps it can cut the time to rebuild the map, or switch map modes/colors, by about 50%. For larger maps the difference is even bigger, a map of about 1700 pixels wide takes me about 30 seconds to initially build (and this is the same whether I'm caching or not) and about 9 seconds to rebuild.

[Ratzap]

Since I recognized what you probably want it for, I've already added this feature. It's now in v6rc2.

Thanks very much. Yes it's so Fusebox can add in the drain from active SCANsat parts.

[technogeeky]

I've been doing some more testing of the big map performance.

For one thing I tested the alternative radial vector calculation using the Quaternion.AngleAxis thing, it made no difference in performance.

For the map caching, I have some concerns about memory usage because the array can be quite large when you start stretching the map out. Unity's garbage collection is apparently fairly lax, so every time you change the map size a new array is created in memory and the old remains. Eventually the GC kicks in and all of those old arrays get deleted (by stretching the map way off screen, I'm guessing to around 4000 pixels wide, you can make the memory usage jump by 200MB or so each time you change it), but I can see this causing problems in some cases. It's also not really consistent in how much gets deleted, I've seen the memory usage drop from around 3.4 to 1.3GB, but I've also seen drop to around 3GB and stay there.

That said, I don't think any of this is really a problem, at least not a new problem. This exact same thing already occurs whenever you resize the big map in v5 (and v6). I assume this is because the texture is also being recreated and taking up a lot of memory. The caching array exacerbates the problem, but I've never heard anyone complain about SCANsat causing KSP to run out of memory before. So if it wasn't causing fatal errors with just the texture taking up space, then I don't think the addition of the array will really change anything (unless you're already right on the line with memory usage, but that's a different issue and doesn't affect the majority of players).

Another reason I'm not too concerned is that for this to really be a problem requires stupidly huge map sizes and constant resizing of the map. When you stay below 1500 pixels or so wide it doesn't have such a huge memory hit. If you just pick a size and only occasionally change it you won't have a problem, just like no one that I know of ever reported having memory problems pre-v6.

There are some cases where I can understand using very large map sizes. I think they can be valuable for exporting high quality maps. Maybe it would be best to create a dedicated map export function, one where you can input a specific pixel width and one that doesn't cache the data. In other cases it might be a good idea to throw a warning (maybe just once) when someone stretches the map out too much or resizes it 20 or so times in less than a minute.

The performance impact is pretty large and I think worth the trade off. For reasonable size maps it can cut the time to rebuild the map, or switch map modes/colors, by about 50%. For larger maps the difference is even bigger, a map of about 1700 pixels wide takes me about 30 seconds to initially build (and this is the same whether I'm caching or not) and about 9 seconds to rebuild.

FYI: I am reading what you are writing intently; I am just not ready to respond yet. I am playing around with the profiling data and trying to reconcile my guesses with what the profiling data means and what your improvements change.

I also wrote a tool that parses the profiling data (using regular expressions) -- at least, the profiling data that can be useful (I don't parse the UNKNOWN references or file references, since I can't tell what they are talking about). It works OK-ish. I have both the qualified and unqualified name available from the parse output -- so, say, SCANsat.SCANmap.getPartialMap and getPartialMap. I also parse the arguments signature, the % of time, and the # of calls. The next step I wanted to do was to automatically insert the profiling output for a particular function directly into the source code (above the function, say). This is actually tricky to do correctly (and should not be attempted with regexes, because it is not context free), and I didn't want to hack it last night. It also involves inserting unique IDs as placeholders into comments, so they can later be stripped out (remembering line numbers here is not sufficient; I intend to insert things into the files so all of the line numbers after the current function will be wrong).

The overall goal here is to be able to build/run the profiling tool, have it run, save profiling data -> turn that data into textual output -> splice textual output and put in place in source, where we can see it. It seems simple, but it was harder than I thought.

I can think of a few ways to put upper limits (both static and dynamic ones) on the size of the map. I am a habitual mover and resizer of the SCANsat map (both of which delay rendering).

I was going to say something like "export is not a problem; for large map sizes we can simply refuse to cache but export to a file". And there's already an exportPNG function (which I actually changed in a trivial way in my latest commit). I think the cleverer feature will be to reverse this process, and consider the cache you are using to be an intermediate stage. It is almost surely faster (even with slow loading time) to load a huge texture from a file and scale that down rather than to re-generate the map (this only being useful after the map is fully rendered).

BTW: Your judgements on the performance are currently via: monitoring memory and/or FPS and/or wall clock time when rendering the map, right? You are not using the performance tool thing?

[Starstrider42]

I think the cleverer feature will be to reverse this process, and consider the cache you are using to be an intermediate stage. It is almost surely faster (even with slow loading time) to load a huge texture from a file and scale that down rather than to re-generate the map (this only being useful after the map is fully rendered).

IIRC, ISA Mapsat did something very similar, taking an on-disk image as the "true" map and displaying it in-game. The problem was that the image tended to become corrupted, causing lots of trouble if you were doing new scans at the time. Something to watch out for.

[Virtualgenius]

I tried using the new RC2 the scansatrpm.dll doesnt play nicely with mechjebrpm.dll they cancel each other out and nothing works using the 238 dev version of MJ go back to version 5 it all works

[DMagic]

The overall goal here is to be able to build/run the profiling tool, have it run, save profiling data -> turn that data into textual output -> splice textual output and put in place in source, where we can see it. It seems simple, but it was harder than I thought.

I can get results from the profiler in emveepee (I think), but it is a little hard to get any actual information out of it. I can see that getPartialMap, getElevation, gui_bigmap_build, etc... are all very high up on the list when I quickly go in and rebuild the map. So that's reassuring.

I can think of a few ways to put upper limits (both static and dynamic ones) on the size of the map. I am a habitual mover and resizer of the SCANsat map (both of which delay rendering).

Putting an upper limit on map size might not be a bad idea, especially if there is some other way to export larger sizes. Thankfully moving the map, or opening and closing it, doesn't force a reset of the texture and the cache. And as long as you're resizing it between reasonable sizes, not changing from 2500 to 2700 pixels wide for instance, it shouldn't really be a problem.

I was going to say something like "export is not a problem; for large map sizes we can simply refuse to cache but export to a file". And there's already an exportPNG function (which I actually changed in a trivial way in my latest commit). I think the cleverer feature will be to reverse this process, and consider the cache you are using to be an intermediate stage. It is almost surely faster (even with slow loading time) to load a huge texture from a file and scale that down rather than to re-generate the map (this only being useful after the map is fully rendered).

Do you mean making a big texture file of the map that KSP can open when loading? That's a bad idea and will cause the same memory issues that ISA had. Any texture that gets loaded into memory from a file will stay there permanently, taking up space.

If you mean generating a texture from a full coverage map and using that as a base to rebuild the big map might, that might work out better. There might be problems with the sheer number of maps required to make a complete set for your current planet though.

BTW: Your judgements on the performance are currently via: monitoring memory and/or FPS and/or wall clock time when rendering the map, right? You are not using the performance tool thing?

Yeah, these are all made with the stop watch on my phone and by pushing Task Manager into a corner so I can see the various memory usage values while KSP is running (I always run it windowed). I'm watching FPS using a.g.'s plugin. Though I do have a plugin of my own that writes out FPS to a log file, it's intended to be used for my CPU database, but if I could setup a kind of standard test run it might be useful for comparing performance.

[technogeeky]

I tried using the new RC2 the scansatrpm.dll doesnt play nicely with mechjebrpm.dll they cancel each other out and nothing works using the 238 dev version of MJ go back to version 5 it all works

Hm. I wonder how that could happen. I did not include any references to MechJeb at all when building. I did notice that MechJeb things in my ALCOR capsule didn't work, but the SCANsat ones worked fine.

I'm not using dev238, I'm on dev229. But again, I don't see how they would conflict so as to make both not work. I know people in here mentioned recompiling "all three DLLs". I assume this means RPM, SCANsatRPM, and MJRPM. But does MJRPM have a dependency on SCANsatRPM? Why would it? I don't think SSRPM has a dependency on MJRPM (since I didn't build it with one, and it built fine).

Shrug. I hope someone else can elucidate here.

[technogeeky]

I can get results from the profiler in emveepee (I think), but it is a little hard to get any actual information out of it. I can see that getPartialMap, getElevation, gui_bigmap_build, etc... are all very high up on the list when I quickly go in and rebuild the map. So that's reassuring.

Putting an upper limit on map size might not be a bad idea, especially if there is some other way to export larger sizes. Thankfully moving the map, or opening and closing it, doesn't force a reset of the texture and the cache. And as long as you're resizing it between reasonable sizes, not changing from 2500 to 2700 pixels wide for instance, it shouldn't really be a problem.

Do you mean making a big texture file of the map that KSP can open when loading? That's a bad idea and will cause the same memory issues that ISA had. Any texture that gets loaded into memory from a file will stay there permanently, taking up space.

If you mean generating a texture from a full coverage map and using that as a base to rebuild the big map might, that might work out better. There might be problems with the sheer number of maps required to make a complete set for your current planet though.

Yeah, these are all made with the stop watch on my phone and by pushing Task Manager into a corner so I can see the various memory usage values while KSP is running (I always run it windowed). I'm watching FPS using a.g.'s plugin. Though I do have a plugin of my own that writes out FPS to a log file, it's intended to be used for my CPU database, but if I could setup a kind of standard test run it might be useful for comparing performance.

Ah. The emveepee output is fine for viewing the first few entries, but the text-based version is better (since it's amenable to grep/sed/awk/regexen). If you're ever interested in a particular method, you can just search the text output from mprof-decoder.exe.

The primary output that you get (in an abstract sense, and assuming a not-yet-written-tool works properly) is a relative, statistical call table (calls made to, calls made by) for every method in your project (plus any calls your project makes to other, known methods). So you can see things like: for every one call to my method blahblah, I make 100 calls to the method derpderp.

Obviously the ones which are high up on the list (the ones you mentioned) are the prime targets for optimization (because they are taking up a lot of clock time). But any methods they ever call optimize them too. And the methods you are optimizing might be calling them.

It's just one route. I think I'm going to adopt the caching you are doing, in one style or another. I wonder if there is an existing data structure out there in C# libraries or Unity libraries that would provide a cache that we could persist (preferentially in chunks).

Also: another route to reducing the memory footprint while sacrificing only a small amount of performance might be to store the cache compressed (just like SCANsat already does with serialize/deserialize). I think the reduction factor is O(10-100) in memory, and I presume that the CPU cost is small enough that it's worth trying.

[Tiron]

Hm. I wonder how that could happen. I did not include any references to MechJeb at all when building. I did notice that MechJeb things in my ALCOR capsule didn't work, but the SCANsat ones worked fine.

I'm not using dev238, I'm on dev229. But again, I don't see how they would conflict so as to make both not work. I know people in here mentioned recompiling "all three DLLs". I assume this means RPM, SCANsatRPM, and MJRPM. But does MJRPM have a dependency on SCANsatRPM? Why would it? I don't think SSRPM has a dependency on MJRPM (since I didn't build it with one, and it built fine).

Shrug. I hope someone else can elucidate here.

There's no cross-dependency that I've seen, but RPM has the same problem with the SSRPM and MJRPM plugins that they have with the SS and MJ Plugins themselves: if they're a different version than the main RPM plugin, it can't use them.

That's why I said you have to do all three: every time I compile them myself, even using the 'release' source, I get a different version number than the master release. If I try to just compile one of the plugins and drop it in, it doesn't work. It needs all three recompiled so they all have matching version numbers.

Like I said, if you somehow got the thing to spit out your recompiled .dll with the same version number as the master, my understanding is that it'd work as a drop-in replacement without doing the other two. I just have no idea how to do that. As far as I can tell, it's the version *number* that causes the problem, not the file being different. So in theory, you could also fix it by just having all the new SCANSat.dlls have the same version number as the one that works, as long as the function didn't change enough to break it. (but that has its own problems)

I've also been having out of memory crashes lately, and I'm not entirely sure why. It happens on scene changes, but only after I've loaded a flight in, generally. I don't know if it's SCANSat or what, because I started using V6R1, RPM 17, and MJ 238 all at the same time, and that's the most recent change I've made. I also picked up IR recently and already had a ton of other plugins and parts (Including a modified EVE, which I've suspected as well.)

Edit:

IIRC, ISA Mapsat did something very similar, taking an on-disk image as the "true" map and displaying it in-game. The problem was that the image tended to become corrupted, causing lots of trouble if you were doing new scans at the time. Something to watch out for.

It actually didn't. It actually used a group of scan 'probe' lines to query the terrain maps and created the texture on the fly with the results. As a result, he actually set it up such that it could support not only changed terrain maps but entirely new planets without requiring an update, starting with X4R1.

The problem it had with the terrain changes in 0.21 was that it has a file called hilo.dat which stores the maximum and minimum height values for each body, used exclusively to set the color scale on the maps. He'd set it up to automatically rebuild it if it found a new planet, but forgot to have it check if the existing planets had changed. So the pre-made hilo.dat it came with was outdated, and resulted in weirdly colored maps due to using the wrong scale. The trick, which I discovered, was to delete the hilo.dat file entirely to force the thing to rebuild it.

He'd done his best to keep it from rebuilding if it didn't need to, due to the fact that rebuilding it was a slow process which made the game seem to hang during loading. He just forgot to include a check for that particular case. Presumably he fixed it in X4R2, since I told him about it on IRC.

The problem with this system is that the map you made was stored purely in the .png it made with the scan results. (There was also an option to spit out a .csv with the raw data alongside, but it wasn't fully re-implemented last I heard.) This meant that even if Hilo.dat was updated when the terrain changed, your stored maps weren't. Which could result in the slightly comedic situation of not only the map being wrong, but the legend for the color scale being different than the one the map was made with. Re-scans overwrite the sections of the stored map you scan, allowing you to fix it without dumping your old data (which is what I'm doing in that image I linked above, except I only had a couple orbits worth of 'bad scale' map done when I figured out how to fix it.)

Edited May 4, 2014 by Tiron

[Rioliki]

I was used SCANsat with pod integration, found somewhere here long time ago (must be outdated now). Suddenly it stops working. It happens right after updating procedural parts v 0.9.8 and Module manager 2.0.9. I don`t really think this is because PP, rather MM. But KER pod integration still works fine… Anyway, if it is exist, please give me link to pod integration cfg. Thank you.

Also to Thank you brave connoisseurs of SCANsat, who took development into their own hands. Only two notes:

- Can you change the description to show scanning altitudes? I learned this to remember, but new users have to seek it in .cfg or lurk over forum.

- Toolbar icons has gray backgrounds. Sometimes it looks not good. Transparent would be great.

[BudgetHedgehog]

Anyway, if it is exist, please give me link to pod integration cfg. Thank you.

This will add the BTDT to pods and probes:

@PART[*]:HAS[@MODULE[ModuleCommand]]
{
	MODULE
	{
		name = SCANsat
		sensorType = 32
		fov = 1
		min_alt = 0
		max_alt = 2000
		best_alt = 0
		power = 0.01
		scanName = BTDT scan
	}
}

That should work fine.

[Rioliki]

Well, thanks, but MM 2.1.0 suddenly solves the problem. It has to be 2.0.9 issue.

[Starstrider42]

The problem with this system is that the map you made was stored purely in the .png it made with the scan results. (There was also an option to spit out a .csv with the raw data alongside, but it wasn't fully re-implemented last I heard.) This meant that even if Hilo.dat was updated when the terrain changed, your stored maps weren't. Which could result in the slightly comedic situation of not only the map being wrong, but the legend for the color scale being different than the one the map was made with. Re-scans overwrite the sections of the stored map you scan, allowing you to fix it without dumping your old data (which is what I'm doing in that image I linked above, except I only had a couple orbits worth of 'bad scale' map done when I figured out how to fix it.)

I think you're referring to a different problem than I was. I encountered problems even when playing with 0.23 exclusively, and without ever deleting Hilo.dat. So there were no terrain changes (real or artificial) to trigger the bug you described.

In any case, we seem to agree that an image is not a good primary representation of the player's scanning progress, though for different reasons.

[Tiron]

I think you're referring to a different problem than I was. I encountered problems even when playing with 0.23 exclusively, and without ever deleting Hilo.dat. So there were no terrain changes (real or artificial) to trigger the bug you described.

In any case, we seem to agree that an image is not a good primary representation of the player's scanning progress, though for different reasons.

Oh, mayhaps. I was extremely tired when I wrote that. I thought you were saying it took a pre-rendered image and revealed it.

It may have had a corruption problem in 0.23 (which would've been version X4R2 I believe.) But I wouldn't know, as I quit playing the middle of 0.22 and only started playing again maybe a month ago. And have been using SCANSat in lieu of Mapsat since I got back.

[Shania_L]

If I update from the latest damny version to the techno/magic version will I lose my currently scanned maps?

Also looking at the changelog, is there a whole lot of difference? My V5 is working fine (afaik) on 0.23.5 Am I going to notice any changes?

[technogeeky]

If I update from the latest damny version to the techno/magic version will I lose my currently scanned maps?

Also looking at the changelog, is there a whole lot of difference? My V5 is working fine (afaik) on 0.23.5 Am I going to notice any changes?

There are not yet any substantive gameplay changes, unless you have the bug where planted flags do not appear on the BigMap.

No, you should not lose your currently scanned maps.

[technogeeky]

I was used SCANsat with pod integration, found somewhere here long time ago (must be outdated now). Suddenly it stops working. It happens right after updating procedural parts v 0.9.8 and Module manager 2.0.9. I don`t really think this is because PP, rather MM. But KER pod integration still works fine… Anyway, if it is exist, please give me link to pod integration cfg. Thank you.

Also to Thank you brave connoisseurs of SCANsat, who took development into their own hands. Only two notes:

I think you might have confused our Bravery and Stupidity bars.

- Can you change the description to show scanning altitudes? I learned this to remember, but new users have to seek it in .cfg or lurk over forum.

You mean in the README.txt? Or in the small map? Or both? Either way, OK.

- Toolbar icons has gray backgrounds. Sometimes it looks not good. Transparent would be great.

Added to todo list with higher priority. Yes, I noticed this the other day when I toggled to the other skin. Yes, I will fix that. It will go along with other support for skins.

Edited May 4, 2014 by technogeeky