[0.22] Kerbal Maps: Web maps of all bodies. Biomes and text info. (v0.11 Oct 8)

[nascarlaser1]

46 minutes ago, Drew Kerman said:

A standalone app won't be my focus. I need to get things back up and running for my KSA Flight Tracker. Anything beyond that is beyond me at this time. Once I get the map tiles back up (don't need the Mega data, which is outdated - I can make my own) I can use the Github source to recreate the KerbalMaps.com service so ppl don't have to use the KSA Flight Tracker, as that doesn't by default take you straight to a surface map. Might be able to port over a few bells and whistles, like orbital plotting. No promises tho.

Still no response to email or voicemail. *cracks knuckles* guess I'll get cracking on this tomorrow...

Again, anyone with Javascript and/or GIS mapping experience (mainly the latter) that I can call upon for assistance please make yourself known. Might make things a bit easier and thus faster... no word from @Cyriak either on here or twitter, which is a shame cause he was originally working to help me get the map tiles setup before the original author became heavily involved and began handling things himself.

I have no doubt at all I will get this all figured out. Just can't say how long that will take.

I'll be coming by here daily to respond to stuff but for updates I'll mainly be posting over at @DKerman_KSA on twitter

Is there anyway of finding an archived version, then just copying the code over to a new url/site, then upgrading from there? (I have no idea if this is humanly possible. My only coding experience is via roblox, which was self taught via the roblox wiki .)

[Drew Kerman]

2 hours ago, swjr-swis said:

Thanks for picking this up and for what you can do. There's no rush, do things as time and circumstance permit you.

Suggestion: set up Patreon and/or PayPal accounts and post them here and on the site. Let us that cannot help any other way at least contribute to the recovery effort.

Pointless. Money won't gift me the knowledge I need and the time I spend on this wouldn't be increased by extra funds. Appreciate the sentiment tho

2 hours ago, nascarlaser1 said:

Is there anyway of finding an archived version, then just copying the code over to a new url/site, then upgrading from there? (I have no idea if this is humanly possible. My only coding experience is via roblox, which was self taught via the roblox wiki .)

I don't need the code, the code is on Github. I need the map tiles, and if I'm going to have to figure out how to do this I might as well redo it so everything is up to date, especially the biome maps. I also have the plugin the author coded that will export all the terrain and slope data from the game, so I can add bodies from OPM as well (can't guarantee any others I don't use). I just have to figure out how to process the GEOTiff data. I might also need someone to update it for v1.2 - and I'd rather not call out the obvious go-to forum member for that if anyone else would care to step up and have a look at the code once 1.2 drops this week

Edited October 9, 2016 by Drew Kerman

[seanth]

Would it be easier to take the data on mega and import it into something like google earth as a layer? I know you can import just the image of Kerbin as a skin on the globe, but that doesn't give you the terrain detail. Earth is also much bigger than Kerbin so distances would be wrong.

I'm unsure what can and can't be done with Google Earth. Just a thought.

[JAFO]

On 10/8/2016 at 1:30 PM, Drew Kerman said:

Sure you weren't just seeing a cached version? Still down for me

Thankfully, there's always the Internet Archive's Wayback Machine. Here's their last snapshot of the KerbalMaps site, from April 30th of this year:

http://web.archive.org/web/20160430022558/http://www.kerbalmaps.com/

(Mind you, it's quite a bit slower than the original site)

Edited October 10, 2016 by JAFO

[Drew Kerman]

Status update:

Okay so I spent all morning on this and I have everything about the map tiles figured out. So creating the actual images you see when the map loads is not a problem. Well, okay it kinda is since I don't really know how an image like this, which shows the entire terrain of a planet, is made. But once I can make that image I can tile it up and load it no problem. I already tested it with locally-hosted tiles and it worked a charm (after I figured out I was still trying to link to a CSS file on the kerbalmaps.com server, derp).

Next step is to dive into the GEOTIFF stuff, which is being handled by a UTFGrid that allows the map to pull up data on a location that includes biome, slope and terrain elevation for like every pixel on the map. There's also some JSON action going on I need to understand as well to load this data.

For the short term though, thanks to @JAFO I have linked to the WayBackMachine to get all the data needed, so I have a recompiled JS file I can pass to anyone that needs it. I already told the Telemachus dev about it so he should be able to drop that in and get full maps functionality back for the app.

[Beetlecat]

On 10/6/2016 at 1:53 PM, nascarlaser1 said:

Dang it.. I was using this to hunt down the anomalies... now how am I going to find them without having to watch videos, which show me the exact location!?!? I just want the general area... :(.

You'll have Kerbnet for that!

[nascarlaser1]

19 hours ago, Drew Kerman said:

Pointless. Money won't gift me the knowledge I need and the time I spend on this wouldn't be increased by extra funds. Appreciate the sentiment tho

I don't need the code, the code is on Github. I need the map tiles, and if I'm going to have to figure out how to do this I might as well redo it so everything is up to date, especially the biome maps. I also have the plugin the author coded that will export all the terrain and slope data from the game, so I can add bodies from OPM as well (can't guarantee any others I don't use). I just have to figure out how to process the GEOTiff data. I might also need someone to update it for v1.2 - and I'd rather not call out the obvious go-to forum member for that if anyone else would care to step up and have a look at the code once 1.2 drops this week

This is a noob question, but what are tiles? are they like floor tiles ?.

[Drew Kerman]

1 hour ago, nascarlaser1 said:

This is a noob question, but what are tiles? are they like floor tiles ?.

each map is broken up into sets of images w/dimensions in a power of two. KerbalMaps uses 256x256 tiles to create its maps. If you zoom the map all the way out you're actually looking at 2 tiles for 512x256 pixel map. Zoom in one level and you have 8 tiles for a 1024x512 pixel map. The number of tiles increases exponentially as you zoom in. I plan to add 3 additional zoom levels to the 5 already available - which at max zoom would be a 131072x65536 pixel map, or 131,072 tiles (~12GB of data per body)

Edited October 10, 2016 by Drew Kerman

[Lunar Sea]

5 hours ago, Drew Kerman said:

Status update:

Okay so I spent all morning on this and I have everything about the map tiles figured out. So creating the actual images you see when the map loads is not a problem. Well, okay it kinda is since I don't really know how an image like this, which shows the entire terrain of a planet, is made. But once I can make that image I can tile it up and load it no problem. I already tested it with locally-hosted tiles and it worked a charm (after I figured out I was still trying to link to a CSS file on the kerbalmaps.com server, derp).

Next step is to dive into the GEOTIFF stuff, which is being handled by a UTFGrid that allows the map to pull up data on a location that includes biome, slope and terrain elevation for like every pixel on the map. There's also some JSON action going on I need to understand as well to load this data.

For the short term though, thanks to @JAFO I have linked to the WayBackMachine to get all the data needed, so I have a recompiled JS file I can pass to anyone that needs it. I already told the Telemachus dev about it so he should be able to drop that in and get full maps functionality back for the app.

I for one very much appreciate your hard work sir! Kerbalmaps is an invaluable resource for mission planning. 

[seanth]

On 10/10/2016 at 11:23 AM, Drew Kerman said:

Status update:

Okay so I spent all morning on this and I have everything about the map tiles figured out. So creating the actual images you see when the map loads is not a problem. Well, okay it kinda is since I don't really know how an image like this, which shows the entire terrain of a planet, is made. But once I can make that image I can tile it up and load it no problem. I already tested it with locally-hosted tiles and it worked a charm (after I figured out I was still trying to link to a CSS file on the kerbalmaps.com server, derp).

Next step is to dive into the GEOTIFF stuff, which is being handled by a UTFGrid that allows the map to pull up data on a location that includes biome, slope and terrain elevation for like every pixel on the map. There's also some JSON action going on I need to understand as well to load this data.

For the short term though, thanks to @JAFO I have linked to the WayBackMachine to get all the data needed, so I have a recompiled JS file I can pass to anyone that needs it. I already told the Telemachus dev about it so he should be able to drop that in and get full maps functionality back for the app.

I have a version of Kerbin Maps working locally on my computer. I grabbed all the stuff off web archive and edited the paths in the javascript files so it loads the images correctly.

https://goo.gl/photos/qxSizmstShgx4V6y9

I'm not going to waste my time downloading all the tiles for each body at each zoom, though.

What I am currently doing is downloading all the maps on the Kerbal Maps download page. Once I have all those hi-res maps, I'll use the GDAL python tool to chop the maps into tiles. I _think_ that will automatically populate the tiles into the folder structure the leaflet js files wants.

If I'm successful I'll put the the whole thing up on github for people to download and run locally, or for people to build on. 

[Jiraiyah]

3 hours ago, seanth said:

I have a version of Kerbin Maps working locally on my computer. I grabbed all the stuff off web archive and edited the paths in the javascript files so it loads the images correctly.

https://goo.gl/photos/qxSizmstShgx4V6y9

I'm not going to waste my time downloading all the tiles for each body at each zoom, though.

What I am currently doing is downloading all the maps on the Kerbal Maps download page. Once I have all those hi-res maps, I'll use the GDAL python tool to chop the maps into tiles. I _think_ that will automatically populate the tiles into the folder structure the leaflet js files wants.

If I'm successful I'll put the the whole thing up on github for people to download and run locally, or for people to build on. 

That would be nice, I always wanted to have something like this on local

[seanth]

51 minutes ago, Jiraiyah said:

That would be nice, I always wanted to have something like this on local

It'll be a pretty big download. With all the images for all the planets, we're probably looking at something around 7gig

[Jiraiyah]

I know, actually i was thinking a torrent link can be a better solution for this compared to git hub?

Edited October 14, 2016 by Jiraiyah

[JAFO]

2 minutes ago, Jiraiyah said:

I know, actually i was thinking a torrent link can be a better solution for this compared to git hub?

Google drive is also a viable option for multi-GB sized files.

[Jiraiyah]

Just now, JAFO said:

Google drive is also a viable option for multi-GB sized files.

would it allow 7 gig or something? i know free drop box only lets 2 GB

[JAFO]

So far as I recall, it should. There was some user-created content for another game I was involved in the development of, which consisted of several 3GB files, and we used google drive to distribute them.

[Drew Kerman]

4 hours ago, seanth said:

I grabbed all the stuff off web archive and edited the paths in the javascript files so it loads the images correctly.

Yes, that is the same as what I did, modify the JS paths to point to the archive. Do you understand the JSON fetching that is going on to get the UTFGrid data that is used to display the biome/slope/terrain height in the lower-left display? That stuff is still beyond me, tho I haven't taken a serious look at it yet

4 hours ago, seanth said:

I'll use the GDAL python tool to chop the maps into tiles. I _think_ that will automatically populate the tiles into the folder structure the leaflet js files wants.

Yes, the original author mentioned something similar as part of his workflow in getting the maps setup:

Quote

Here is the short version of the workflow, I explain some basic ideas but I'll leave the rest to additional research.

The DEM (digital elevation model) stores the elevation in a georeferenced format (so we can unproject a pixel to it's lat,lng). Geotiff is one such format. It's literally just an image with a bunch of metadata about the projection, extent, and other things. It wont open in most image viewers though, because instead of 32 bits of rgba color each pixel is an encoded float32 of the elevation for that x,y.

Once I have the DEM i use some tools (some bundled with gdal and some written in python by myself using the python gdal bindings) to render the raw elevation data. Color relief just maps the elevation to a color gradient stored in a txt file. The slope is then calculated by measuring the maximum angle of each pixel to its neighbors, then again rendered by mapping the value of the slope model to a color. I have some tools to render the DEM to color relief and slope (in the same format, geotiff so it's stil georeferenced, but now it DOES contain rgba color data).

You can download the DEM and the rendered geotiffs, i link to them on the kerbalmaps forum post.

Once I have all the renders I then need to chop it up into tiles, thankfully gdal had a bundled tool for this. It creates all the folders and png tile images. https://www.mapbox.com/help/how-web-maps-work/ <-- similar but different projection

Finally, i run some png optimizers to get the file size as low as possible on all the images (optipng, pngquant, etc. easy to automate via scripting)

Basically each step in the process is one command invocation that I more or less copy and paste but change the inputs for each body.

something like:

make_slope kerbin.elevation.tif kerbin.slope.tif
render_color kerbin.elevation.tif kerbin.colorrelief.txt kerbin.colorrelief.tif
render_color kerbin.slope.tif kerbin.sloperelief.txt kerbin.sloperelief.tif
slice_tiles kerbin.colorrelief.txt kerbin/color
slice_tiles kerbin.sloperelief.txt kerbin/slope

(This is an oversimplification, because it does not include the satellite view, the biome view, or all the utfgrid data (elevation, biome))

If you get this to work I'd love to know more about it so I can do the same when (hopefully) creating new maps

Updates from my end on the new maps

Sadly, I totally forgot that I only have the DLLs for the heightmap extractor that Saik0 handed to me, so it can't be recompiled for v1.2. I also have preliminary reports from other people looking at the data that it's not being exported properly. Still waiting to hear from more ppl who have offered to take a look. So new maps for right now are in a serious holding pattern :/

[Jiraiyah]

8 hours ago, Drew Kerman said:

Yes, that is the same as what I did, modify the JS paths to point to the archive. Do you understand the JSON fetching that is going on to get the UTFGrid data that is used to display the biome/slope/terrain height in the lower-left display? That stuff is still beyond me, tho I haven't taken a serious look at it yet

Yes, the original author mentioned something similar as part of his workflow in getting the maps setup:

If you get this to work I'd love to know more about it so I can do the same when (hopefully) creating new maps

Updates from my end on the new maps

Sadly, I totally forgot that I only have the DLLs for the heightmap extractor that Saik0 handed to me, so it can't be recompiled for v1.2. I also have preliminary reports from other people looking at the data that it's not being exported properly. Still waiting to hear from more ppl who have offered to take a look. So new maps for right now are in a serious holding pattern :/

you can convert those dll back to c# code, use something like dotpeek

[Drew Kerman]

3 hours ago, Jiraiyah said:

you can convert those dll back to c# code, use something like dotpeek

Interesting. Will keep this in mind if I can find a modder to update it or am forced to look into myself...

[seanth]

 

12 hours ago, Drew Kerman said:

Yes, that is the same as what I did, modify the JS paths to point to the archive. Do you understand the JSON fetching that is going on to get the UTFGrid data that is used to display the biome/slope/terrain height in the lower-left display? That stuff is still beyond me, tho I haven't taken a serious look at it yet

The info in the lower left display (biome, elevation, etc) isn't showing up for me either. I will look at it more later this morning. I'll read through the javascript and look at the leaflet examples online to try and get an idea of why it's not working (it seems like it should be working)

I did get some tiles made: sat image at 6 zoom levels. Total folder sizer of ~150meg. I'll make the colour relief, slope, and biome ones next. 

@Drew Kerman: when I have them, do you want me to PM you and give you an url so you can download them, or just start putting things on github. If you started a github project, I could fork and add the images for you to pull in. However you like. You stepped up to recover things, and I just want to help.

[Drew Kerman]

1 hour ago, seanth said:

when I have them, do you want me to PM you and give you an url so you can download them, or just start putting things on github. If you started a github project, I could fork and add the images for you to pull in. However you like. You stepped up to recover things, and I just want to help.

give me your Github ID and I will just add you as a member to my fork. I already started to add some tiles to the project that I made for proof of concept to ensure I understood properly how they were loaded.

Appreciate the help!

[IvanSanchez]

On 7.10.2016 at 4:49 AM, Drew Kerman said:

I'm also probably going to need help, so anyone with javascript experience and GIS geo-mapping experience please raise your hand... you still around @Cyriak?

Hey there.

Actually I'm one of the maintainers of Leaflet, and created Leaflet.Kerbal to try and replace the old Leaflet.KSP codebase (and bring it up to speed to Leaflet 1.0 features, such as nice support for wrapping equirectangular map projections). It's gonna be very increasingly hard to develop with Leaflet.KSP, as it uses Leaflet 0.7.x (which is now deprecated).

If I have some spare time, I'll gladly help out with this. I know my way around map projections, reprojections, tiles, UTFgrids and most GIS concepts.

[seanth]

A potential problem: I just realized that the maps up on mega don't have the biomes. I've made all the map tiles for kerbin (except for the biome map) and they are all working fine on my local. When I try and download the images as a complete zip the file is corrupted, so I am downloading them in batches now. I'll make the rest of the tiles for the bodies as the maps download.

We need to find high res maps for biomes though 

[Drew Kerman]

31 minutes ago, IvanSanchez said:

Actually I'm one of the maintainers of Leaflet, and created Leaflet.Kerbal to try and replace the old Leaflet.KSP codebase (and bring it up to speed to Leaflet 1.0 features, such as nice support for wrapping equirectangular map projections). It's gonna be very increasingly hard to develop with Leaflet.KSP, as it uses Leaflet 0.7.x (which is now deprecated).

Well, this is pretty much the awesomest news I've seen all week (and that should mean a lot to you considering what else happened this week :P)

Yes, we shall talk!

18 minutes ago, seanth said:

We need to find high res maps for biomes though 

We can pull the biome maps straight from the game's Unity resource bundle and work them from there. In the meantime, I'm going to get coordinated with IvanSanchez

[seanth]

3 minutes ago, Drew Kerman said:

We can pull the biome maps straight from the game's Unity resource bundle and work them from there. In the meantime, I'm going to get coordinated with IvanSanchez

Should I keep hacking on the elevation/biome/slope data display in the lower left corner?