[PLUGIN+PARTS][0.23] SCANsat terrain mapping

[Zeroignite]

I still want to figure out which of the devs put that in.

[DeepOdyssey]

I still want to figure out which of the devs put that in.

You mean the

[spoiler below]

smiley face in the ocean?

[spoiler end]

[diomedea]

SCANSAT and resource mapping.

This subject was already mentioned on a number of posts in February, but since then it has subsided. Any effective implementation of resources mapping is still to be achieved with SCANsat.

I am considering planetary resources in general, most notably what is made available by the Open Resources System (ORS), that is the resources system used by the InterStellar mod, and by the Kethane mod.

As with any other localized item, resources could be made to appear in a mapping tool. Map layers with already detected resources may be implemented with SCANsat, just like biome maps currently are.

As it is now, Interstellar only shows resources when a detector is close enough, but there is no memory of what has been detected (scanned).

InterStellar brings a number of png maps in the install, to show the locations and relative density of uranium, thorium, water, alumina on different celestial bodies. The maps dimensions are different, and a config file holds all the required information for management of map data.

SCANsat could use some of the techniques already in place to implement those maps. A section in the savegame about what has been scanned and where on any body, and code to reveal only those parts of a map already scanned. Being the maps readily available makes me guess this feature would require much less effort (from such good programmers as those already taking care of SCANsat) than other features already planned.

As for Kethane data, the system is pretty different (list of locations in savegame where deposits are), so SCANsat would require a method to render those locations on a map to be displayed. Kethane's author showed interest towards this feature (notably, in post #1037), and sure knows how to help.

I hope, now that SCANsat is getting improved again, that the above features may also come into planning.

[Tiron]

Hmmm... that seems like an issue that could affect other plugins as well. Do you have any advice for the "safest" way for modders to set up their plugin versions?

I'm hardly an expert on the subject, you'd probably want to ask Mihara, because everything I know I learned from reading his thread.

Trick being, I gather that it only applies in cases where one plugin is trying to call functions from another. It's not a problem within any particular mod, just when one mod tries to use parts of another.

He said something in one of the posts about being able to use 'reflection' to get around it, which I'd guess is normally what gets used. Problem is, for RPM purposes, SCANSat/Mechjeb *itself* would have to be set up in some specific way in order to use it. If they were, it'd work without the secondary plugins, a-la Vesselview (which has a specific version just for RPM.)

Basically, RPM has a problem with it because Mihara's using a dirty, dirty hack to hook into a couple of mods that weren't really designed to be hooked into at all.

[DMagic]

SCANSAT and resource mapping.

I've never looked at the Open Resources System, but I have spent some time with the Kethane code. I think it stores the data in a somewhat similar manner to SCANsat, basically a publicly accessible list of which cells have been scanned and whether or not they contain Kethane.

SCANsat works entirely with positions described by longitude and latitude, whereas Kethane uses the grid system. It seems relatively straightforward to figure out which cell is below any given point above the surface and that could be translated to something that SCANsat understands.

The issue that needs to be worked out though, is how everything should be displayed. For one, I think the Kethane map should be shown as an optional overlay that can be activated on any map type rather than a separate map, that seems like the most useful method. The other issue is how Kethane's grid should be shown on the map. As far as I can tell there is no granularity below the level of an individual cell, so any map overlay would end up looking like a stretched out version of the Kethane grid (I think even the polar projection would be distorted). I think this would look somewhat awkward, especially on the rectangular map where cells at high latitudes would be very warped, but I don't see any simple way of doing otherwise.

Edited May 9, 2014 by DMagic

[Virtualgenius]

Perhaps lock the Kethane map to one particular format rectangular polar etc the one that gives the most clarity. Is there any fix yet for ScansatRPM when its used inconjunction with MJ

[KerbMav]

For me the big benefit of integration of ressource scanning would be the scan-in-background part and not necessarily ressource maps viewable in SCANsat's windows.

If you say that it would be relatively easy to determine which cell a SCANsat scanner is flying over, would it be equally easy for SCANsat to find and edit the section of the save file used by Kethane to mark scanned cells, so that a SCANsat user could MM-edit the Kethane scanners to use a SCANsat MODULE?

[technogeeky]

For me the big benefit of integration of ressource scanning would be the scan-in-background part and not necessarily ressource maps viewable in SCANsat's windows.

If you say that it would be relatively easy to determine which cell a SCANsat scanner is flying over, would it be equally easy for SCANsat to find and edit the section of the save file used by Kethane to mark scanned cells, so that a SCANsat user could MM-edit the Kethane scanners to use a SCANsat MODULE?

For scanning purposes, there are two methods that we could use:

1. Use the standard Kethane hexagonal grid, and incorporate that into a second fundamental type of scanning.

2. Use the same method that SCANsat uses for scanning and storing data, and use the fact that you can approximate Kethane scanners using a specific FOV/swath width.

The honorable gentleman Psawhn has some Octave/Matlab code that does a rough job of calculating ideal altitudes for scanning Kethane, using the approximate method. Either direct inspection of the code, or evaluation of the output of the code versus the behaviour of Kethane in-game should tell us if the method will work or not.

The general answer to your question is yes: if it's stored in the persistent file, then it is possible to use (and edit) all of the Kethane data.

[technogeeky]

Perhaps lock the Kethane map to one particular format rectangular polar etc the one that gives the most clarity. Is there any fix yet for ScansatRPM when its used inconjunction with MJ

I have read and understood all of the arguments about what to do, and have made the decision to change SCANsat to having fully hardcoded version numbers. However, I do not accept that it is a good idea for me (after having released v6rc1, v6rc2) to manually edit the version number to be identical to that of v5 (the last damny release). First of all, it would be going backwards in version number (which is a horrible idea). But secondly, I have made changes to the part cfgs that I do not want to revert.

I intend to release the official version of SCANsat this weekend. I hope that Mihara will be around to produce a new triplet of RPM, ssRPM, and mjRPM against my new release, before I release it -- so everyone can get the whole package, all at once.

Thereafter, there won't be a problem. I won't be changing the version number on dev builds, so you will be able to use the latest dev build of SCANsat until I change the version number manually (which I will only do when breaking compatibility).

I would ask that someone inform me if part.cfg changes are going to break savegames (or not). I think DMagic should be able to take a guess. I think he mentioned doing something with MM, instead of editing them. I'd be happy to revert the changes and do it that way, if it will guarantee that nobody's saves will get messed up. I suppose I could have tried it out on my career savegame, but I don't have any scanners on any vessels (I did kerbal recovery missions before switching back to development mode).

- - - Updated - - -

Perhaps lock the Kethane map to one particular format rectangular polar etc the one that gives the most clarity. Is there any fix yet for ScansatRPM when its used inconjunction with MJ

I have read and understood all of the arguments about what to do, and have made the decision to change SCANsat to having fully hardcoded version numbers. However, I do not accept that it is a good idea for me (after having released v6rc1, v6rc2) to manually edit the version number to be identical to that of v5 (the last damny release). First of all, it would be going backwards in version number (which is a horrible idea). But secondly, I have made changes to the part cfgs that I do not want to revert.

I intend to release the official version of SCANsat this weekend. I hope that Mihara will be around to produce a new triplet of RPM, ssRPM, and mjRPM against my new release, before I release it -- so everyone can get the whole package, all at once.

Thereafter, there won't be a problem. I won't be changing the version number on dev builds, so you will be able to use the latest dev build of SCANsat until I change the version number manually (which I will only do when breaking compatibility).

I would ask that someone inform me if part.cfg changes are going to break savegames (or not). I think DMagic should be able to take a guess. I think he mentioned doing something with MM, instead of editing them. I'd be happy to revert the changes and do it that way, if it will guarantee that nobody's saves will get messed up. I suppose I could have tried it out on my career savegame, but I don't have any scanners on any vessels (I did kerbal recovery missions before switching back to development mode).

[technogeeky]

From reading Mihara's stuff on the subject, it's actually Squad's fault. Something in the way their plugin loading setup is arranged makes it so that directly calling functions from another .dll only works if it's the exact right version of the .dll, otherwise it doesn't allow it. He's got a couple posts on the subject, talking mostly with Sarbian after a change to Mechjeb's buildbot broke compability with the dev mechjebs.

Here's one

Another

Given what happened with the Mechjeb Dev version and the nature of it, it seems like the pluginloader's going off the version number on the .dll for the purposes of enforcing this.

Basically what happened was this: Mechjeb's version numbers are hard-coded in the code itself, and are only changed when a release is pulled. All the mechjeb dev-builds since version 2.2.1.0 was released (225/226-242 at the moment) if you pull the code down and compile it yourself, will show up as version 2.2.1.0.

They also have an automated build bot set up, which every time there's a commit to the project automatically compiles the new version and hosts it for people to download. At some point here recently they changed how it does so: instead of just compiling it straight up, it now automatically appends the commit # onto the last digit of the version number. So instead of it coming out at 2.2.1.0 like it would if you compiled it yourself, the one the buildbot spits out comes out as 2.2.1.242 (or whatever.)

Judging by the forum posts, before that change to the buildbot, the dev versions of mechjeb worked with RPM without any modification. You could just drop them in and they'd work...apparently because they had the same version number as the last release. I've seen people mentioning a specific commit where the change took place and the compatibility broke.

My understanding is that dropping the version numbering on the .dll wouldn't fix it in and of itself: the new one would still register as 'different', but if the version number didn't change after that, and the functions that were being hooked into didn't either, each subsequent version after that would continue to work after it'd been recompiled once.

In fact, if my understanding is correct, you could make your versions of scansat work as a drop-in on the unmodified RPM simply by setting it up in such a way that your .dlls always came out with a version number of 1.0.5.32015, the version of the release that RPM's been compiled with.

It's not great practice, and it'd remove the ability to tell what version you had by checking the plugin... but it should allow it to work without having to rebuild RPM constantly.

Thanks for this and the follow up post.

I have thought about it, and I see only two routes that will let me develop as actively as I want without breaking compatibility.

1. Strip the versioning from the assembly entirely, effectivelly setting it to "0.0.0.0" for all time. (BAD)
   
2. Hardcode all four components of the version for each release, and store the development build in some other field (which is not checked by the PluginLoader -- like the description or the copyright field). In which case, I might use something like "1.0.6.0"
   

I will not be be officially releasing a DLL with version number "1.0.5.32015" because that seems dangerous and silly. Except: I would like to release such a version to two or three people who are currently straddling SCANsat v5/v6rc2 with RPM, and who wish to help us at SCAN by telling us if indeed that method (hard-coding to the same version number) solves the problem.

If you would (or anyone else) wants to verify that such a scenario works, please chime in and I will give you guys a link to a download in private message. To be safe, I should probably revert to an earlier build than my master, to get rid of part.cfg changes.

[diomedea]

I've never looked at the Open Resources System, but I have spent some time with the Kethane code.

....

Thanks for giving a thought about this (also about your interest on the same subject in the past). Hope you may want to check ORS, I guess it is even simpler to implement in SCANsat than Kethane.

Yes, Kethane stores data (seemingly compressed as SCANsat does) about the areas scanned with each body. But Kethane and Ore deposits coordinates are separately stored, unlike SCANsat (that I guess peeks directly from body texture files).

You are absolutely right any of SCANsat maps would result in distortions of the hex grid used by Kethane. But that is a general problem of topography, and the prime reason there are many different map projections in use. A rectangular map is best to show features close to the equator, and heavily distort anything close to the poles; the opposite with a polar map. We can only accept this fact; the same kind of distortions already occur with the body textures applied to SCANsat maps. With Kethane topographic grid, we also have a different distortion issue, as the cells better follow the contour of the body mesh, while topographic maps are drawn with the assumption the body is an ellipsoid (or a sphere, in KSP).

About granularity, we know Kethane choice to represent only generic cell content, and each cell being 2.25° wide, brings to maps of such apparent limited resolution (160 cells along the equator, horizontal map axis; about 80 cells along a meridian). No way out about this fact, but I would believe an overlay showing those cell contents is really what SCANsat users would like to see, even if at a low resolution.

Yes, map overlays would be the correct approach, as you suggest. Believe you know enough of Geographic Information Systems (GIS), the way those use map overlays to present geo-localized data would be amazing if replicated in SCANsat.

[DMagic]

For me the big benefit of integration of ressource scanning would be the scan-in-background part and not necessarily ressource maps viewable in SCANsat's windows.

If you say that it would be relatively easy to determine which cell a SCANsat scanner is flying over, would it be equally easy for SCANsat to find and edit the section of the save file used by Kethane to mark scanned cells, so that a SCANsat user could MM-edit the Kethane scanners to use a SCANsat MODULE?

Adding another SCANsat scanner type wouldn't be very difficult, and using that to check Kethane data should work fine.

It is possible to read data from Kethane's cache of scanned cells and I think it is possible to write to that as well, so a SCANsat scanner should be able to update both the SCANsat database and the Kethane database (or rather, SCANsat shouldn't need to keep its own record of Kethane data). The simpler solution is not doing any scanning with SCANsat and only reading the existing database directly from Kethane. And while it's possible to also access anything that Kethane stores in the persistent file, I don't think doing so is a good idea; anything that SCANsat uses should probably go through the Kethane code, not go around it and directly access the persistent file.

My problem with this is that I don't really like the idea of working around the very carefully setup method that Kethane uses for scanning. If you really want to change how that works I think it's better to petition Majiir to do that (which probably isn't going to happen) rather than to work around it with another mod.

I would ask that someone inform me if part.cfg changes are going to break savegames (or not). I think DMagic should be able to take a guess. I think he mentioned doing something with MM, instead of editing them. I'd be happy to revert the changes and do it that way, if it will guarantee that nobody's saves will get messed up. I suppose I could have tried it out on my career savegame, but I don't have any scanners on any vessels (I did kerbal recovery missions before switching back to development mode).

The only way to truly screw up a save file with part.cfg changes is to alter the "name = " field, usually found at the top of the file (not to be confused with the "title = " field which controls what the part is called in the VAB). This is what the persistent file and craft files use to identify parts, changing it will result in that part not being found and the craft not loading. It's really easy to check if part.cfg changes break anything because you can reload the database without restarting KSP, you just have to be in the spacecenter to do it; I do this all the time and it's rare that changing a part.cfg does any real harm.

Using Model {} nodes adds some other ways to break a savefile, but SCANsat parts don't use those, so it's not an issue.

There is also some amount of changes that can necessitate re-purchasing a part in the R&D center, I've never been clear on exactly what (besides tech-node position) triggers this, but it won't affect any flights in progress. You won't be able to load a craft file in the VAB until you repurchase the part, but it won't delete anything.

About Module Manager, and this goes for adding SCANsat to Kethane scanners or for adding the MapTraq to command modules, I don't think there's any particular reason why MM is actually required for this. It's not that difficult to dynamically add and remove modules from parts when necessary. You have to be careful about removing the modules and about not saving anything with that module, but that's not really any trouble. xEvilReeperx's Science Alert does this with his transmitter module and I don't think it causes any problems (and that mod seems to have a good method for making sure the module gets properly added and removed). I think it's generally a good idea not to make dependencies unless it's absolutely necessary, even if it is just MM or Toolbar.

That would also open up the possibility of managing these things from the settings menu. Turning off MapTraq for all command pods, or integrating Kethane (or maybe even adding Kethane background scanning) could be controlled through the settings menu so that everyone could configure it how they want.

Edited May 9, 2014 by DMagic

[Zeroignite]

I am really looking forward to having SCANsat, MJ, and RPM play nice again! Definitely hideous that you have to keep a static version number in the file, hardcoded... but hey, workarounds... work.

Honestly I often have an easier time reading scansat via an RPM display (especially in ALCOR) than I do on the big map!

[BudgetHedgehog]

One thought that hasn't been brought up.. why add another projection that can handle the hex layout of Kethane? Rather than squishing either the poles or equator, why not add in some variation of a Goode homolosine or something? Yeah, it might have disadvantages in other areas, but it'd keep the hexes fairly accurate.

[Tiron]

I am really looking forward to having SCANsat, MJ, and RPM play nice again! Definitely hideous that you have to keep a static version number in the file, hardcoded... but hey, workarounds... work.

Honestly I often have an easier time reading scansat via an RPM display (especially in ALCOR) than I do on the big map!

I'm still gonna be doing my own RPM compiles because Mechjeb Dev Version, at least for the moment. I might look at trying again to get Mechjeb itself to compile, since its version numbers are already hardcoded and it's just the stupid buildbot messing things up.

But yeah, same thing with the RPM SCANsat display...mostly. There are two problems with it that I've seen:

It's slower than heck to render, almost as slow as the full bigmap.

It's nigh-useless near the poles, because the RPM map apparently uses either the rectangular or the kavrayskiy projections all the time (I strongly suspect but cannot prove it's using rectangular), with no way I've found to change it. It leads to hugely distorted output in the polar regions and pretty much entirely loses directional alignment.

When I tried to use the zoom-in function on the bigmap instead it seemed to be doing something similar (albeit with MUCH faster rendering but a much smaller window), so I almost wonder if the RPM version isn't calling the function for that little zoom window.

[DMagic]

Say it with me everyone, proof-of-concept...

It's pretty simple actually, I'm just reading from Kethane's database, I'm not doing any scanning with SCANsat. The colors show up a bit better in the grayscale mode, but that can obviously be improved upon. The distortion of the grid isn't too bad until you get to very high latitudes, the rectangular projection is only slightly worse. And this becomes less noticeable the more you have scanned.

But, once again, this isn't a workable solution. I'm just slapping one or the other shade of green on top of the altimetry map if the given cell has been scanned. Ideally the Kethane overlay should be toggleable on any map type and should be more easily visible. Showing the Kethane deposit amount under the mouse would also be a good idea and shouldn't be difficult at all.

[Zeroignite]

I think I love you. And yes, I am saying "proof of concept".

[diomedea]

Yes, DMagic, that is a nice demonstration of the feasibility (really a proof-of-concept) for displaying Kethane. Bringing overlays management into SCANsat would make this an important playable feature. Drooling about the possibility to see this become reality.

Again, thanks.

[PizzaMore]

Say it with me everyone, proof-of-concept...

It's pretty simple actually, I'm just reading from Kethane's database, I'm not doing any scanning with SCANsat. The colors show up a bit better in the grayscale mode, but that can obviously be improved upon. The distortion of the grid isn't too bad until you get to very high latitudes, the rectangular projection is only slightly worse. And this becomes less noticeable the more you have scanned.

http://i.imgur.com/uLTnDRG.png

http://i.imgur.com/Av8eclX.png

But, once again, this isn't a workable solution. I'm just slapping one or the other shade of green on top of the altimetry map if the given cell has been scanned. Ideally the Kethane overlay should be toggleable on any map type and should be more easily visible. Showing the Kethane deposit amount under the mouse would also be a good idea and shouldn't be difficult at all.

I'm totally loving the progress of SCANsat! you'all are all too damn smart!

Quick question and with acknowledgement that this is just a concept exercise; so it is possible to load the Kethane data into SCANsat. Would it also mean that a SCANsat survey would also update the Kethane grid? (not implying that any of the Kethane user experience would occur ie. the Kethane scan tones /variations) From the sounds of it, the data is coming from Kethane, thus the Kethane map would be as current as the SCANsat Kethane data.

Edited May 11, 2014 by PizzaMore

[pantheis]

I absolutely hate Kethane requiring the vessel to be active for the sole purpose of electrical use. There are so many ways to make sure you always have enough power to scan it's absurd that I have to timewarp around a planet to get it scanned. The way Scansat works is much better. If you add Kethane scanning into Scansat, that would be absolutely awesome!

[DMagic]

Does anyone particularly care if the SCANsat map's Kethane overlay actually looks like a grid of hexagons? The performance impact of really accurately matching every pixel to the Kethane grid is significant. Displaying the grid at a lower resolution is very easy to do and reduces the performance impact to almost negligible levels.

Note the "Kethane" button below, next to the "Grid" button, that can be used to turn the overlay on and off, and the Kethane quantity indicator showing the amount in the cell under the mouse pointer. The color of the overlay is a little bit more saturated now and it's easier to see on colored maps; cells that actually have resources under them are emphasized. The overlay works on all map types now, but the polar projection breaks my low resolution grid method, so I'm using a higher resolution (though still not as high as possible) for that.

[pantheis]

I don't care if it looks like a kethane grid or not. Personally I find the kethane overlay on the map view hard to use, except when targeting a landing with mechjeb.

[JeffreyCor]

Don't care about the grid at all. As long as the position of the resource is correct it would be fantastic!

[DMagic]

Of course it's possible to calculate a craft's trajectory relative to the surface, but that's not the problem. As I understand it, the problem stems from the fact that kethane scanning divides the surface of a body into hexagons, and working out exactly which hexagons said trajectory passes through is very much not an easy task.

Actually, an algorithm for solving the generalised version of that problem would probably make a good doctoral thesis for a maths student, if anyone's looking for any ideas.

Basically this. To make performance reasonable while maintaining accuracy is non-trivial. If anyone has a serious proposal for how to solve this, by all means, share it. So far, everything I've seen is along the lines of "just record the trajectory and then turn that into hexes, easy, duh."

So apparently this isn't really difficult at all. Ok, SCANsat already has a method for background scanning built in, but still, it took me literally five minutes to come up with a workable solution. The trick is that SCANsat doesn't actually scan anything in the background, it just records what sensors have flown over what area of the planet. The elevation and biome scanning is all done when the map is drawn. The same thing is possible with Kethane, you just check which areas of the planet have been scanned and which cell those areas correspond to.

The active vessel, with the blue orbit, does not have a Kethane scanner and is orbiting at ~750km. The inactive vessel, in grey, has a small Kethane scanner with a SCANsat module attached to it and is orbiting at ~250km. It took about 15 sec at max time warp to go from the map shown in my last post to this map. The Kethane grid doesn't seem to accept my additions to its database until I reload, but the SCANsat map (note the "Rebuild Kethane" button) works fine.

I added a Kethane sensorType to SCANsat (using sensorType = 4, the old slope scanner number), set the fov = 1 and the best alt = 250000. The rebuild button checks which cells have been scanned and adds them to the Kethane database. I'm guessing you need an active Kethane sensor or part on the vessel for the grid to update without reloading.

So if the reason for not having background scanning really is technical and not philosophical then I don't see any reason why this can't be added to SCANsat. Working out how best to handle the part modules is another question, but that's not really a big deal.

Edit: The code for all of this is available on a branch of my Github page, the base is technogeeky's latest build.

Edited May 11, 2014 by DMagic

[Drew Kerman]

stupid forum wouldn't let me give you more rep This is awesome.