[0.22] Kerbal Maps: Web maps of all bodies. Biomes and text info. (v0.11 Oct 8)

[Drew Kerman]

1 minute ago, seanth said:

Should I keep hacking on the elevation/biome/slope data display in the lower left corner?

I'd say hold off on that, since it would be something that could be affected by the potential upgrade to the latest Leaflet version

[IvanSanchez]

56 minutes ago, seanth said:

Should I keep hacking on the elevation/biome/slope data display in the lower left corner?

I'd say hold on for a bit. First priority is to get the RGB maps (visible colour) maps up and running. Elevation and biome can wait a bit.

I have a rough plan in my head: store the elevation data in a 1-channel, 16-bit TIFF, slice it up in tiles, and let the browser query the pixels directly. Similar thing for the slope. Biomes can hold themselves in a 1-channel 8-bit image nicely.

I mean, UTFgrids were cool back in 2011, but there have been no developments in that tech lately, and handling elevation data directly in the browser opens up the option to do on-the-fly hillshading, which is always cool.

On the code side of things, I'd like to have a simple, documented way to have kerbal maps with Leaflet 1 but minimal bells & whistles. The flashy stuff can be added later.

[seanth]

Ok. I'll hold off, but will finish making all the tiles. That way people can download and have access to a mostly working clone of Kerbal Maps while updates are being worked on

 

[seanth]

I have managed to make the tiles for the slope, color, and sat views for all the bodies. Unfortunately the resulting zip is 2.9gig in size, so I can't upload it to github as a release.

For those with space to spare on their drives, you can download the package and try and run it locally

http://goo.gl/tOk0qD

Edit: I wasn't aware that google's short links won't let you link to zip. Try this:

https://ice-nine.org/seant/KerbalMaps.zip

 

Edited October 14, 2016 by seanth

[HafCoJoe]

On 10/3/2016 at 6:59 PM, Drew Kerman said:

alright, so kerbalmaps.com went down hard at the end of last week. This happened earlier this year and it was back in a few days. It's still down, so I've sent the author an email to see what's up. He and I were working closely and frequently a few months ago on getting the maps updated with the newer biomes and even more zoom levels. Communication stopped from his end at the start of July and haven't been able to raise him since - I hope he just got busy with work and family. If I can't reach him after a week or so, I do have the means to host kerbalmaps.com but it will take me a lot of work to setup because I was never made fully familiar with the process of creating the map tiles. Hopefully he can just get kerbalmaps.com back up and running.

Keep you all posted...

Many thanks, and best of luck Drew. I never knew you where in cahoots on KerbalMaps  - TIL

[Drew Kerman]

Sup.

I've mostly been leaving the heavy-lifting up to @IvanSanchez and his upcoming newer version of the Leaflet.KSP library since he actually knows what he's doing and when I try to fiddle with mapping stuff it's like banging my head against the wall. Which isn't fun. So I've been doing other things, that are fun.

However the plugin Joel (Saik0) passed to me before he went poof might still be of use so I stand ready to assist and hopefully we can get some new map content created still.

So as things stand right now, the maps are still accessible via internet archive, my Flight Tracker works, if the Telemachus dev comes back he'll have almost all he needs to get that plugin working again (the terrain/slope/biome data is probably still broken) and eventually there will be a new library we can all use. Maybe updated maps too.

[ryan andrade]

http://www.kerbalmaps.com/

stupid the site is taken down

[Lunar Sea]

4 hours ago, ryan andrade said:

http://www.kerbalmaps.com/

stupid the site is taken down

[Jiraiyah]

well these days i put a scan sat around any planet and scan the crap out of it then use the big map to get the coordinates i want, that is the only way i have now :/ meaning you would want scan sat mod and a second sand box save for this

[CAKE99]

There's a mod called Waypoint Manager, which is useful for finding coordinates, though it's less precise that Kerbalmaps.

[Drew Kerman]

On 10/29/2016 at 1:09 PM, ryan andrade said:

http://www.kerbalmaps.com/

stupid the site is taken down

Hi there. Welcome to last month. You even bother reading back a few pages?

[damaltor]

Hi everybody,

i made something nice for you here, so you can use kerbalmaps again: http://ksp.deringenieur.net/

This is still based on the original data which was linked in this thread, see below. It might be outdated.

Just to note: i do not want to steal the project from anybody, but i really, really liked kerbalmaps and was sad that it went offline. This version of kerbalmaps is TEMPORARY and will be removed when the original gets back to life, or when drew/joel/ivansanchez/anyone else has a better idea, or whenever i feel that it is time. i do host that on my private hosting ressources, so if i think i need them for something else, i will remove it.

Please feel free to use it as much as you like, but as the server has a somewhat limited upload you might run into trouble when too many people access it simultaneously.

The sat, height, and slope layer should work for any celestial body. The biome layer should work for kerbin and the mun, just as it was with the original kerbalmaps.com. The small overlay in the lower left should show coordinates for every celestial body, by klicking the coordinates you can switch between decimal degrees and dms notation. The "current biome", "height" and "slope" stuff in it do only work on the outermost zoom level for kerbin, more on that in the technical details below.

Although this version can be scrolled indefinitely to the left and to the right, POI and anomalies will only show up in the "center" of the map. This means that you can rotate the celestial body (= scroll to the left/right) as much as you would like, but to see the markers for POI or anomalies you will have to scroll to the center which can be kind of hard to accomplish - the easiest way is to re-load the site. I will try to fix that, but that will take a while.

I also removed the google tracker.

If anybody feels offended that i do now host this, drop me a message. That should not be a problem.

 

Technical info for the interested:

I used the data from the ZIP archive above to make this work. I set up a new blank website, added Leaflet to show the map, recompiled the Leaflet KSP plugin to work with my setup, added proj4leaflet which also wanted to be recompiled and then dropped the data in. I had to change a lot of small bits inside of the Leaflet KSP plugin to make it work as my setup is somewhat different to the original one. i also made changes so that my different directory structure works better, and this should be also good if ever someone else gets this project as there are no more hard-coded urls in the Leaflet KSP plugin. everything now smoothly works from the base dir with subdirectories, without the need for subdomains and such. also, the whole thing could be zipped up and should work locally without trying to reach external servers, which the current zip version does.

The graphic tiles for sat, slope and hight inside the ZIP archive were complete, but the tiles for the biome layer were missing. It worked nevertheless, because these files were still reachable on the original server kerbalmaps.com. With a small shell script i was able to scrape all those missing files from the original server and added them into my structure, so that it will still work if kerbalmaps.com breaks away entirely, which i dont hope. Unfortunately, only the biome tiles for kerbin and the mun are available, as i have a hard time to generate new tiles.

For the small overlay in the lower left of the page, there are special json files which contain the information of which biome is where and which height/slope is in which position. They are not the same as the graphical tiles, but specially-made for the purpose of the overlay. Those json files are missing almost entirely, the only ones i have (kerbin on the outermost zoom level) come from the zip archive. if the rest ever surfaces, i will add them in, but for now they were not in the zip, not reachable on the original server, and not in the web archive - so as of now they are lost.

Anyone else working on this: i would like to help, so please contact me if you need or want something. i have pretty much understood the structure of the data and am somewhat aware of how the json stuff works, but i am unable to generate new maps, let alone json files. Maybe we could work together? I would provide my hosting ressources (if there is not too much traffic) for development purposes, though maybe some day this will have to move somewhere else.

[Drew Kerman]

21 hours ago, damaltor said:

For the small overlay in the lower left of the page, there are special json files which contain the information of which biome is where and which height/slope is in which position. They are not the same as the graphical tiles, but specially-made for the purpose of the overlay. Those json files are missing almost entirely, the only ones i have (kerbin on the outermost zoom level) come from the zip archive. if the rest ever surfaces, i will add them in, but for now they were not in the zip, not reachable on the original server, and not in the web archive - so as of now they are lost.

how well do you understand the whole JSON file setup? I have the ability to create the data needed for slope/elevation in GEOTIFF format but no idea how to make that into the JSON data

[damaltor]

Well, if in understand it correctly, the biggest part of the json file is an array of 256x256 characters. Each of the different characters stands for a different biome (e.g. A=highlands, B=lowlands, &=water). The character array is exactly exactly as big as the image file (I think) So if you generate the image data in pixels, you might also be able to generate a text file with characters in them. We have to find out which character stands for which biome though.

 

This is a guess though, but I think it works like that.

[seanth]

On 11/18/2016 at 9:36 AM, damaltor said:

For the small overlay in the lower left of the page, there are special json files which contain the information of which biome is where and which height/slope is in which position. They are not the same as the graphical tiles, but specially-made for the purpose of the overlay. Those json files are missing almost entirely, the only ones i have (kerbin on the outermost zoom level) come from the zip archive. if the rest ever surfaces, i will add them in, but for now they were not in the zip, not reachable on the original server, and not in the web archive - so as of now they are lost

I never grabbed the rest of the json files because it's a bit tedious to get them all for all the planets and moons. If you are interested, I can start either dling them or explain how to get them from the internet archive.

EDIT: I was just poking around on the internet archive version. I guess nothing but Kerbin had elevation data? It looks like the json elevation files for everything but Kerbin are MIA

On 11/18/2016 at 9:36 AM, damaltor said:

The graphic tiles for sat, slope and hight inside the ZIP archive were complete, but the tiles for the biome layer were missing. It worked nevertheless, because these files were still reachable on the original server kerbalmaps.com. With a small shell script i was able to scrape all those missing files from the original server and added them into my structure, so that it will still work if kerbalmaps.com breaks away entirely, which i dont hope. Unfortunately, only the biome tiles for kerbin and the mun are available, as i have a hard time to generate new tiles.

Yeah. Sorry about that, but I couldn't find high resolution biome maps to turn into tiles, so I didn't make new ones. 

Edited November 20, 2016 by seanth

[damaltor]

You can right - the json files for height, slope, and biome are missing for all bodies but kerbin. Even for kerbin, they are only available for the outermost zoom and the nex zoom level, and even then one is missing. Downloading wou not be a big problem, my script should work for json files as good as it did for the biome tiles, but I gues that they are gone. We might be able to generate the json files from the biome tile though, I want to try that If everything else fails. Might be unaccurate though (but not more unaccurate than the images).

Also, no need for a sorry, you guys put a lot of work into that :)

[Drew Kerman]

On 11/19/2016 at 10:53 AM, damaltor said:

Well, if in understand it correctly, the biggest part of the json file is an array of 256x256 characters. Each of the different characters stands for a different biome (e.g. A=highlands, B=lowlands, &=water). The character array is exactly exactly as big as the image file (I think) So if you generate the image data in pixels, you might also be able to generate a text file with characters in them. We have to find out which character stands for which biome though.

I was talking about the terrain/slope data, not biomes

[damaltor]

Should be the same principle, I guess. I will have a look into that in a few hours, will come back here afterwards. ‎

[damaltor]

Ok, so here are some news.

The height and slope data is made in a similar way to the biome data. my guess with 256x256 characters was wrong, there are actually only 64x64 characters. one character stand for 2x2 pixels. i compared the slopes image of one from the kerbin tiles with the slopes json file of the same tile:

As you can see, the shape of the big dark blue part is similar, and the rest will be too. so i guess we could be able to generate slope/elevation data from the respective images once we decode the characters.

 

What i dont know yet: The json file starts with

{"keys":["0","12","9","2","4","1","3","5","8","11","10","14","7","19","6","15","13","39","22","16","45","17","21","29","18","28","27","31","34","20","43","25","26"],
"data":{},
"grid":[
"                                                                ",
"                                                                ",
"                                                                ",
"!#$$%$&$&&'&&&&&$&$$$'                                          ",
"$&&&%&&$'&  $&&$()&$&&&$$(                                      ",

(...)

]}

i have no idea what the "keys" field means, and what the "data" field means (and if it is important as it is empty anyway). the "grid" field is obviously the real data.

Also, i did find out that moving the mouse generates an ElevationOver, BiomeOver etc. event, which automagically generates an event data field, which already carries the full-text decoded data ("4°" for slopes or "3000m" for height or "Greater Flats" for biome). I was unable to find the actual json parser though for now, i still am searching.

[nascarlaser1]

your a life saver @damaltor! Thank you for the temporary maps!! this will help my circumnavigation of Kerbin!!

 

YAAAYY!! rep giving time!

[Drew Kerman]

hrmm... ok before we move forward any more on this I will try to get an update from @IvanSanchez since I do believe his approach to slope/elevation/biome data was not going to be the same

Will see if the tag notifies him via email or something (he hasn't logged on since last month) if not I know another way to reach him I will use later this week

Edited November 20, 2016 by Drew Kerman

[seanth]

10 hours ago, damaltor said:

i have no idea what the "keys" field means, and what the "data" field means (and if it is important as it is empty anyway). the "grid" field is obviously the real data.

Also, i did find out that moving the mouse generates an ElevationOver, BiomeOver etc. event, which automagically generates an event data field, which already carries the full-text decoded data ("4°" for slopes or "3000m" for height or "Greater Flats" for biome). I was unable to find the actual json parser though for now, i still am searching.

This is probably old news* to the people that do this sort of thing everyday, but I did some digging through @saik0's github history for this project, and then looked closer at the javascript: the json elevation and biome files (that give info in the on-map key) are almost certainly utfgrid files. I'm not a GIS guy, so this is all new to me. Anyone know how to go from the elevation DEMs to the tiled json utfgrid format? I'll poke around tomorrow, but I suspect someone more familiar with gdal might just Know How To Do This. 

Edit: maybe use ogr2ogr gdal_contour to make a .shp file, https://github.com/Ecotrust/create-utfgrids, and then done?

*Yup. I'm an idiot: 

 

Edited November 21, 2016 by seanth

[damaltor]

I decoded it a bit more. if you see this text and this image side by side, (this is biome data btw), you can see that the map is visible pretty good in the text. obviously, the "&" character is water, and the " " (space) character is ice caps, and the "(" character is deserts.

(The text is only the TOP HALF of the image file, not the entire image!)

The biome json file starts with the following:

{
    "keys": [
        "8",
        "3",
        "7",
        "2",
        "1",
        "0",
        "4",
        "5"
    ],
    "data": {
        "0": "Water",
        "1": "Grasslands",
        "2": "Highlands",
        "3": "Shores",
        "4": "Mountains",
        "5": "Deserts",
        "7": "Tundra",
        "8": "Ice Caps"
    },
    "grid": [
(...grid follows)

so my guess is the following:

as the first 32 ascii chars are control sequences, they are not used inside the json files. the space character, 0x20, is the first valid char. so for the ice caps part of the map there is a space character, which is the FIRST valid character. in the "keys" section, the first number is "8", which translates with the "data" section to Ice Caps.

The second valid character is !, which equals 0x21. the second number in the keys section is "3", which translates to shores, and as you can see in the text map there are lots of ! between land and water.

This does not work out for water '&' or desert '(' and all of the other characters though - this might be for the reason that the caracter " (0x22) is skipped because it is needed inside the json files to begin and end the strings. that means: & (0x26) is the 6th valid character (not the 7th, because " is skipped), the 6th key is 0, and 0 is water. for the deserts: ( (0x28) is the character for deserts, ( is the 8th valid character, the 8th key is 5, and 5 equals deserts.

Also, the slopes data does not have a data field because it needs no text translation - in the keys there are all needed informations: if you scroll back up the the post about the slope data, you can see that the dark-blue part on the ice caps of the planet are space characters in the text grid. the space character is the first valid character, so the slope at that position is equal to the first key - it is 0° (which is correct if i believe the map overlay).

the same is true for the elevation data: the elevation file has tons of keys because there are so many different heights. around the ice caps, there also are space characters in the json files. 0 is the first valid character, and the first key is 30, so the height at the poles is 30m, which is exactly the height shown on the map overlay. the elevation data has no textual representation, so the "data" field is empty as it is in the slopes data.

the elevation data has another quirk though - it has far more than 96 keys which would exhaust the ascii charset - in fact it has about 2000 different keys (which comply to about 2000 different elevations). the grid looks weird, because i guess that the file is then encoded in utf-8.

 

so much for the file format - what next?

 

 

---edit: i am now pretty sure about the UTF8 - first, because it is named "UTFgrid" (duh!) and second because there are UTF-specific decoding functions in the parser.

Edited November 21, 2016 by damaltor

[Drew Kerman]

I just now poked @IvanSanchez on twitter so hopefully he will stop by at some point to let us know how he's getting on with his v1.x leaflet version of KMaps. Holidaze are upon us so let's all be patient. In the meantime safe Thanksgiving to all celebrating.

Edited November 24, 2016 by Drew Kerman

[Poodmund]

@Drew Kerman, do you happen to know how the original /sat/ images were generated? I presume all the others were done through ScanSAT due to their resolution at 16k.