stupid_chris

[Plugin/Tool] [1.1.x] HeightmapExtractor v1.1/HeightmapManager v1.1 | 03/05/16

Recommended Posts

While working on a side project of mine, I ended up needing the exact ground altitude of each coordinate on the planets in KSP. Didn't take me quite long to realize how ridiculously slow the PQS is for this kind of job. I ended up creating this, and I thought releasing this in the wild may be useful to anyone else needing to know the altitude at precise locations on specific bodies.

HeightmapExtractor v1.1

Download: Github

Changelog:

Spoiler

May 3rd 2016
v1.1
-KSP 1.1 upgrade
-Upgraded to C# 6.0 code
-Fixed a few generation bugs
-Fixed problems with AppLauncher button

February 19th 2015
v1.0
- Initial release.

 

 


HeightmapManager v1.1

Download: Github

Changelog:

Spoiler

May 3rd 2016
v1.1
-Upgraded to KSP 1.1
-Fixed ReadPixels2D returning the same array as the heightmap

February 19th 2015
v1.0
- Initial release.

 

 


License | Source


This tool is twofold, so I'll explain both parts separately:

HeightmapExtractor:

This first part is what does the big, slow work. Basically, what this does is that upon loading the SpaceCenter scene, it will access the PQS system and start to read the altitude of every CelestialBody in the game over all it's surface, store it, and output the result as a data dump as well as a grayscale image. You can control the resolution of the maps, the starting/ending points in both latitude and longitude, which CelestialBodies are extracted, the scanning speed (adjust to your hardware), if the maps are flipped horizontally/vertically, altitude limits for the maps (say you want to ignore oceans), if the grayscale colours are inverted, and how the heightmap is saved (image, binary dump, or both). It makes for a flexible tool that can map specific portions of each body in the game if needed. The data dumps are saved under the .bin format and can then be moved and used at other places.

The extraction process can be very long. For 1440x720px maps, it can take about one to two minutes per map, and doubling the resolution usually augments map creation time by a factor of four. If my memory serves me right, doing all the bodies in the stock game at 2880x1440px resolution took about 30 minutes on my machine (AMD A-10). This is long because the PQS is ridiculously slow. This is why this exists, to have a faster alternative to probing the PQS to get ground altitude.

The Space Center also has an Applauncher button to pop a menu allowing to reload the settings config in game, as well as restarting the scanning process if necessary.

 


HeightmapManager:

This second part is basically just a tool to treat and use the data created by the extractor. Simply reference the .dll in your project and include it in your Plugins folder. Multiple occurences of the plugin in the GameData folder are not a problem, if correctly referenced your mod should only use the version you packaged. HeightmapManager only contains two classes, Heightmap.cs, and MathUtils.cs. The first is the one you're interested in, the second is a bunch of Math methods used internally.

The Heightmap object will allow you to load the raw data dumps created by the HeightmapExtractor into a memory array by only from their path. From there, you can access the height value of each pixel using an indexer. Please note that the the (0, 0) point on the map is the top left corner, and for a given point (x, y), x is the position on the horizontal axis, and y on the vertical axis.

The Heightmap also has an included bilinear reading method which basically takes two numbers between 0 and 1, treats them as proportions on the map, and does a bilinear approximation of the four closest pixels if it does not exactly match a specific pixel.

To give you an idea of how fast using those Heightmap objects is compared to probing the PQS, for my personal use while using the data to create maps depending on the altitude, getting the altitude from 1440x720px maps took 1.5s with those objects, against around a minute and a half while probing the PQS. Here is an example:

Spoiler

using System;
using System.IO;
using System.Collections.Generic;
using System.Diagnostics;
using UnityEngine;
using HeightmapManager;

/* HeightmapExtractor was made by Christophe Savard (stupid_chris), and is licensed under
* CC-BY-NC-SA 4.0 INTL. If you have any questions about this, please contact me on the forums. */

namespace HeightmapExtractor
{
	[KSPAddon(KSPAddon.Startup.SpaceCentre, true)]
	public class SpeedTest : MonoBehaviour
	{
		private CelestialBody kerbin = null;
		private readonly Color red = new Color(1, , , 1);
		private readonly Color blank = new Color(1, 1, 1, );

		private void Start()
		{
			kerbin = FlightGlobals.Bodies.Find(b => b.bodyName == "Kerbin");
			StartCoroutine(CreateMaps());
		}

		private double TerrainAltitude(double longitude, double latitude)
		{
			Vector3d radial = QuaternionD.AngleAxis(longitude, Vector3d.down) * QuaternionD.AngleAxis(latitude, Vector3d.forward) * Vector3d.right;
			return kerbin.pqsController.GetSurfaceHeight(radial) - kerbin.pqsController.radius;
		}

		private IEnumerator<YieldInstruction> CreateMaps()
		{
			Texture2D map = new Texture2D(1440, 720, TextureFormat.ARGB32, false);
			Color[] pixels = new Color[1036800];

			//Generating from looking up the PQS
			Stopwatch timer = Stopwatch.StartNew();
			for (int y = ; y < 720; y++)
			{
				double latitude = ((y * 180d) / 720d) - 90d;
				for (int x = ; x < 1440; x++)
				{
					double longitude = ((x * 360d) / 1440d) - 180d;
					int index = (1440 * y) + x;
					double altitude = TerrainAltitude(longitude, latitude);
					pixels[index] = altitude >  ? red : blank;
				}
				if (y % 90 == ) { yield return new WaitForEndOfFrame(); }
			}
			timer.Stop();
			print(String.Format("Total time using PQS: {0}s", timer.Elapsed.TotalSeconds));
			map.SetPixels(pixels);
			map.Apply();
			File.WriteAllBytes(Path.Combine(HeightmapUtils.mapsURL, "PQS_Kerbin_test.png"), map.EncodeToPNG());

			map = new Texture2D(1440, 720, TextureFormat.ARGB32, false);
			pixels = new Color[1036800];

			//Generating from the precomputed heightmap
			timer.Reset();
			timer.Start();
			Heightmap heightmap = new Heightmap(Path.Combine(HeightmapUtils.mapsURL, "Kerbin_raw.bin"));
			for (int y = ; y < 720; y++)
			{
				for (int x = ; x < 1440; x++)
				{
					int index = (1440 * y) + x;
					double altitude = heightmap[x, y];
					pixels[index] = altitude >  ? red : blank;
				}
				if (y % 90 == ) { yield return new WaitForEndOfFrame(); }
			}
			timer.Stop();
			print(String.Format("Total time using heightmap: {0}s", timer.Elapsed.TotalSeconds));
			map.SetPixels(pixels);
			map.Apply();
			File.WriteAllBytes(Path.Combine(HeightmapUtils.mapsURL, "Heightmap_Kerbin_test.png"), map.EncodeToPNG());

			Texture2D.Destroy(map);
		}
	}
}

 

 

Basically what this does is try to create an image of the land above the sea level for Kerbin. It first does so by using the PQS to know the altitude of a specific coordinate, then the second time using a pregenerated Heightmap. Here are the results:

Spoiler

Using the PQS:

TUyFjF5.png

Total generation time: 97.3493251s

Using the heightmap:

x6ppsgk.png

Total generation time: 1.6762808s

That makes it only 1.71% the time of the PQS technique. That's quite the improvement.

All you need to do, is to run the extractor once, create the binary dumps, include the dumps as well as the HeightmapManager.dll with your mod, and you don't need to touch snail-trail PQS anymore. Simple enough?

And to show what I mean, here are 2880x1440px maps create by the extractor (open them in full scale to understand how large):

 

The images being .png are 8bit grayscale images, unfortunately, restrictions from KSP prevent saving 16bpp grayscale images directly, so the color resolution is only 256 shades. That's why the data dumps are required.

Edited by stupid_chris
Updated to 1.1

Share this post


Link to post
Share on other sites

What exactly is the utility of this plugin?

Share this post


Link to post
Share on other sites

Of course, now that you've done all the extracting, you could just put up the resulting files and save us from needing to actually use the plugin for the stock planets... but don't mind me... I'm just a silly duct tape owner, I don't really know anything.

Share this post


Link to post
Share on other sites
Man, it really brings out the Perlin in KSP...

Yeah, all the small bodies are pretty much just that. Probably why they all look the same, and why they look nothing like real life asteroids.

This confirms that Erin was nothing more than Laythe with a deformed height map.

Eh, Laythe is mostly seaground, and that is clearly Perlin too.

Of course, now that you've done all the extracting, you could just put up the resulting files and save us from needing to actually use the plugin for the stock planets... but don't mind me... I'm just a silly duct tape owner, I don't really know anything.

I could, but then again, the plugin is there so that people can create the mpas they need, not those I need. This can be use to extract the data from RSS, or Kopernicus, or any other planetary mod. This can be used to create area specific zoomed in maps. Some people may only need low resolution maps. Some may need very high resolution ones. If I posted the set I have, most people may need to create their own anyway. That's why this tool is out there, so that everyone can do whatever.

I'm going to post them anyway, but yeah, thought it was worth mentioning.

Also: I'll write a short example to show the difference between using this and directly accessing the PQS.

Share this post


Link to post
Share on other sites
What exactly is the utility of this plugin?

"Nap-of-the-earth" KSP cruise missile guidance mod..... :)

Share this post


Link to post
Share on other sites

And as promised, the speed test:

using System;
using System.IO;
using System.Collections.Generic;
using System.Diagnostics;
using UnityEngine;
using HeightmapManager;

/* HeightmapExtractor was made by Christophe Savard (stupid_chris), and is licensed under
* CC-BY-NC-SA 4.0 INTL. If you have any questions about this, please contact me on the forums. */

namespace HeightmapExtractor
{
[KSPAddon(KSPAddon.Startup.SpaceCentre, true)]
public class SpeedTest : MonoBehaviour
{
private CelestialBody kerbin = null;
private readonly Color red = new Color(1, 0, 0, 1);
private readonly Color blank = new Color(1, 1, 1, 0);

private void Start()
{
kerbin = FlightGlobals.Bodies.Find(b => b.bodyName == "Kerbin");
StartCoroutine(CreateMaps());
}

private double TerrainAltitude(double longitude, double latitude)
{
Vector3d radial = QuaternionD.AngleAxis(longitude, Vector3d.down) * QuaternionD.AngleAxis(latitude, Vector3d.forward) * Vector3d.right;
return kerbin.pqsController.GetSurfaceHeight(radial) - kerbin.pqsController.radius;
}

private IEnumerator<YieldInstruction> CreateMaps()
{
Texture2D map = new Texture2D(1440, 720, TextureFormat.ARGB32, false);
Color[] pixels = new Color[1036800];

//Generating from looking up the PQS
[COLOR="#0000FF"]Stopwatch timer = Stopwatch.StartNew();
for (int y = 0; y < 720; y++)
{
double latitude = ((y * 180d) / 720d) - 90d;
for (int x = 0; x < 1440; x++)
{
double longitude = ((x * 360d) / 1440d) - 180d;
int index = (1440 * y) + x;
double altitude = TerrainAltitude(longitude, latitude);
pixels[index] = altitude > 0 ? red : blank;
}
if (y % 90 == 0) { yield return new WaitForEndOfFrame(); }
}[/COLOR]
timer.Stop();
print(String.Format("Total time using PQS: {0}s", timer.Elapsed.TotalSeconds));
map.SetPixels(pixels);
map.Apply();
File.WriteAllBytes(Path.Combine(HeightmapUtils.mapsURL, "PQS_Kerbin_test.png"), map.EncodeToPNG());

map = new Texture2D(1440, 720, TextureFormat.ARGB32, false);
pixels = new Color[1036800];

//Generating from the precomputed heightmap
timer.Reset();
[COLOR="#0000FF"]timer.Start();
Heightmap heightmap = new Heightmap(Path.Combine(HeightmapUtils.mapsURL, "Kerbin_raw.bin"));
for (int y = 0; y < 720; y++)
{
for (int x = 0; x < 1440; x++)
{
int index = (1440 * y) + x;
double altitude = heightmap[x, y];
pixels[index] = altitude > 0 ? red : blank;
}
if (y % 90 == 0) { yield return new WaitForEndOfFrame(); }
}
timer.Stop();[/COLOR]
print(String.Format("Total time using heightmap: {0}s", timer.Elapsed.TotalSeconds));
map.SetPixels(pixels);
map.Apply();
File.WriteAllBytes(Path.Combine(HeightmapUtils.mapsURL, "Heightmap_Kerbin_test.png"), map.EncodeToPNG());

Texture2D.Destroy(map);
}
}
}

Basically what this does is try to create an image of the land above the sea level for Kerbin. It first does so by using the PQS to know the altitude of a specific coordinate, then the second time using a pregenerated Heightmap. Here are the results:

Using the PQS:

TUyFjF5.png

Total generation time: 97.3493251s

Using the heightmap:

x6ppsgk.png

Total generation time: 1.6762808s

That makes it only 1.71% the time of the PQS technique. That's quite the improvement.

EDIT:

Also uploaded the precomputed Heightmaps I have, downloads available in the OP.

Edited by stupid_chris

Share this post


Link to post
Share on other sites

That's great - thanks..

Now the ignorant question.. - a little OT maybe, but I hope you stumbled over an answer while doing this

Could this or a similiar technique be used to extract the planet textures?

To create a highres area foto of a sort?

Does the PQS also hold the texture info somewhere?

I'm fondling for a while with this idea, but didn't get usefull results..

Share this post


Link to post
Share on other sites

Most, but not all, bodies in KSP have heightmaps. However, the heightmap is just one PQSMod amongst many. What Chris's tool does is export the *final* heightmap, i.e. the topography of the planet rendered to map form, not the original heightmap used by the VertexHeightMap PQSMod.

Share this post


Link to post
Share on other sites

This is huge!

I mean... endless possibilities... :kiss: Thanks Chris!

Share this post


Link to post
Share on other sites
That's great - thanks..

Now the ignorant question.. - a little OT maybe, but I hope you stumbled over an answer while doing this

Could this or a similiar technique be used to extract the planet textures?

To create a highres area foto of a sort?

Does the PQS also hold the texture info somewhere?

I'm fondling for a while with this idea, but didn't get usefull results..

As Nathan said, not like I did it. It has been done and can be done, but Im pretty sure you dont need to itterate like I did to do it, I think the maps are stored somewhere. Probably try probing the ScaledSpaceTransform materials?

This is huge!

I mean... endless possibilities... :kiss: Thanks Chris!

Thanks! Looking forward to what Can come out of this :)

Share this post


Link to post
Share on other sites

Does anyone know of a free utility program that could turn this information into isometric height maps?

Something like a more simple version of this.

3D_TopoMap.gif

Share this post


Link to post
Share on other sites
Does anyone know of a free utility program that could turn this information into isometric height maps?

Something like a more simple version of this.

http://forums.autodesk.com/autodesk/attachments/autodesk/66/104084/1/3D_TopoMap.gif

I'm pretty sure tools like this must be custom built, it needs to access the specific program of whatever made it.

This has great potential for optimizing ScanSat, a big performance killer.

I'm not exactly sure how ScanSat works, but that could be a possibility.

Share this post


Link to post
Share on other sites
This has great potential for optimizing ScanSat, a big performance killer.

I've considered something like this before, but there are a few complicating factors for SCANsat. The map is re-sizable, so you would need one very large data file for each planet. Most of the time you could just interpolate a smaller map from the bigger data file. But if you wanted something bigger it wouldn't be possible, or you would have to revert to the slow method. It might be best just to impose some certain maximum size, and if someone wants to export a larger map maybe a separate interface could be provided.

There is also potentially a problem with on-disk size of all the data files. With mod planets such databases could easily run into the 100s of MB. 16 bits per position (~65000 possible values) would probably be enough to cover the height range of most planets, so that would help keep the size down to some extent, but it's still a lot of space.

Share this post


Link to post
Share on other sites

If those are the problem, I can point out solutions.

First off, this is already 16bit of data precision. The maps are stored as arrays of signed shorts, and the resolution of each pixel is to the meter. for the *very* large maps (2880px per 1440px) That makes 7.9Mb of space per map. For all stock bodies, this is about 118MB in total. Honestly, thats pretty small in disk space. And 2880x1440 is large for a map I dont think much larger should be needed. Since the format is .bin, theyre not loaded by KSP unless you are using them, and since you can only load the current body and that the GC throws out arrays fast, that means barely no effect on RAM. Once you have larger maps, you can easily downscale loslessly. As I said, bilinear interpolation is included to read the map.

Basically, all this means is longer download times. Its in game speed vs disk space.

Edited by stupid_chris

Share this post


Link to post
Share on other sites

To get the heightmap, you need to read out the data from the MapSO that's a member of the PQSMod_VertexHeightMap object. That's just an uncompressed array of the heightmap image.

Share this post


Link to post
Share on other sites
To get the heightmap, you need to read out the data from the MapSO that's a member of the PQSMod_VertexHeightMap object. That's just an uncompressed array of the heightmap image.

Wait, what are you replying to? :P

Share this post


Link to post
Share on other sites

To this:

As Nathan said, not like I did it. It has been done and can be done, but Im pretty sure you dont need to itterate like I did to do it, I think the maps are stored somewhere. Probably try probing the ScaledSpaceTransform materials?

Because the texture itself will not exist.* You can compile the MapSO to texture2D, then export that, though.

*I dearly hope it won't. That would be the utmost of stupidity / profligacy, to keep two uncompressed copies of the heightmap lying around...

Share this post


Link to post
Share on other sites
To this:

Because the texture itself will not exist.* You can compile the MapSO to texture2D, then export that, though.

*I dearly hope it won't. That would be the utmost of stupidity / profligacy, to keep two uncompressed copies of the heightmap lying around...

I'm rather suprised I'll admit that the full texture of the planet isn't stored in the ScaledSpaceTransform materials :/ I'd expect a reference to be found for it to display correctly.

Share this post


Link to post
Share on other sites

The diffuse and normal maps used for scaledspace are linked to in the material applied there, yes, but they're marked as unreadable so you can't extract them. But we were talking about the heightmap, which is neither of those.

Share this post


Link to post
Share on other sites
The diffuse and normal maps used for scaledspace are linked to in the material applied there, yes, but they're marked as unreadable so you can't extract them. But we were talking about the heightmap, which is neither of those.

Philotical was asking about surface map, not heightmap :P

Share this post


Link to post
Share on other sites

Wow, I misread that rather badly. Only way to get hi-res planet textures is by doing what Chris did but in color, or using the built-in CreateMaps() method that he despises. You can't get the existing scaled space textures (unless you dig in the assets) because they're loaded (and set to read-only) before plugins are, to the best of my knowledge, but besides they're only 2048x1024.

Share this post


Link to post
Share on other sites
This thread is quite old. Please consider starting a new thread rather than reviving this one.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.