ISA MapSat "Legacy" discussion

[somnambulist]

I did it the other way around, I replaced the map textures with very tiny image files, 8x8, instead of the maps to see what happened when Mapsat loaded them.

Could I trouble you for a full file list of all the map PNGs that ISA generates? I want to try throwing different image sizes and formats at it later and see if I can get it to break.

[AncientGammoner]

I too want to see a mapping program similar to MapSat (which I'm not running right now in my 'playing install' because of the aforementioned 700mb RAM craziness). I've looked over the code and I honestly can't figure out how he's using so much memory. I mean what it does clearly show is he's loading every map for every planet into memory on startup even if the maps blank (in which case it's loaded with an all black map, which would take up just as much space uncompressed). Thing is, if you copy one of his 2048x1024 maps to a clean install of ksp and paste it like 30 times into Gamedata or wherever, it loads all of them but only increases the RAM usage by like 150mb or so which is what you would expect, so it's gotta be a bug/"feature" of the code that's eating up 700mb.

Anyways, the code for the v4 dev build is like 2500 lines of gut cringing inefficiency. Personally I think it's a good thing that this is being revisited, since the whole thing could use a rewrite from the ground up. First of all, maybe this whole technique of making every single player spend a huge amount of memory and computational resources to pull the terrain data from the game to generate a 2mb png was useful back in the day (I can't see how), but we have ridiculously precise elevation maps of all the planets now. Even without those, you only need to generate the terrain png ONCE (as the developer) then have a way to slowly reveal it to the player as they fly over.

To show you what I mean, I wrote a really crude test version so you can see how it would work. I used the 16k elevation maps (which are way overkill) and rendered them in some GIS software using roughly the same color scheme as ISA and exported them at the same resolution:

Minmus map from ISA.... and one I generated

Then I added an all black (full transparent) alpha layer to the map I made (which also conveniently makes the map invisible to the user in Windows). Then I wrote a really short plugin (<100 lines) that's tied to one of the stock antennas that basically pulls the lat/long coordinates and transforms them using a map projection to get the x y pixel coordinates on the png. Then all it does is flip the alpha channel of the pixels to opaque as the user flies over which "reveals" the map, which is displayed in a GUI box atm:

^There you go, really rough version of ISA that uses virtually no processing power, doesn't lag the game at all, and makes maps that are even more accurate because you don't have the inherent inaccuracies of pulling game data. Check out the comparison (both at the same resolution):

As far as memory, even if you loaded all the textures beforehand it wouldn't take more than 150mb or so, but I see no reason to do that since you could just as easily load the texture of the planet only when you enter it's SOI. I'm also fairly certain you could use Resources.UnloadAsset() on the textures when you leave the planets SOI, which would mean you would only have the textures loaded for the planet you're at (so a maximum RAM usage of maybe 10-15mb during runtime, 0mb if you're not near a planet). What do you guys think?

Edited September 1, 2013 by AncientGammoner

[somnambulist]

As far as memory, even if you loaded all the textures beforehand it wouldn't take more than 150mb or so, but I see no reason to do that since you could just as easily load the texture of the planet only when you enter it's SOI. I'm also fairly certain you could use Resources.UnloadAsset() on the textures when you leave the planets SOI, which would mean you would only have the textures loaded for the planet you're at (so a maximum RAM usage of maybe 10-15mb during runtime, 0mb if you're not near a planet). What do you guys think?

Wow really really awesome work. Do you think it would be possible to load/unload the map textures on the fly? Tingle figured out a way to unload game textures with Universe Replacer; it might be possible to unload non-active maps to save a good bit of memory.

[AncientGammoner]

somnambulist: Sure, I just tested it in game. I generated list of PlanetMap's (class containing name and texture) and assigned a key to creating one, loading the texture into it, and adding it to the list. Then I pressed it over and over and watched as the memory usage grew in task manager as it loaded the texture for each one. Then I assigned another key to iterate through the list and call a class method where the object would simply do "Resources.UnloadAsset(texture);" and sure enough it deletes all the textures from memory instantly. So yeah it's definitely possible to load and unload stuff on the fly.

[Torminator]

Okay, that's incredible. Amazing work.

[Tiron]

Could I trouble you for a full file list of all the map PNGs that ISA generates? I want to try throwing different image sizes and formats at it later and see if I can get it to break.

There's two maps per planet, in the format <Planet>_<MapType>_Topo.png The two map types are 'ISA' and 'Polars'. So for example: 'Mun_ISA_Topo.png' and 'Mun_Polars_Topo.png'. The full name of the planet is used. Files are placed in \Innsewerants Space Agency\Plugins\PluginData

All the other textures (GUI textures, blankmap.png, etc) are in \Guitextures in Plugindata. Removing them didn't seem to do much to the memory usage, though I didn't try using placeholders.

The SOHO Images when you select 'Sun' are actually dynamically loading the very latest images from NASA's SOHO website, at the address given in the window, which is actually kinda neat.

I too want to see a mapping program similar to MapSat (which I'm not running right now in my 'playing install' because of the aforementioned 700mb RAM craziness). I've looked over the code and I honestly can't figure out how he's using so much memory. I mean what it does clearly show is he's loading every map for every planet into memory on startup even if the maps blank (in which case it's loaded with an all black map, which would take up just as much space uncompressed). Thing is, if you copy one of his 2048x1024 maps to a clean install of ksp and paste it like 30 times into Gamedata or wherever, it loads all of them but only increases the RAM usage by like 150mb or so which is what you would expect, so it's gotta be a bug/"feature" of the code that's eating up 700mb.

First, Unity does decompress the textures, it's been a complaint for a few versions now that PNG Textures get massively inflated. There was one that reducing the texture quality setting doesn't help as well.

My analysis indicates that if you replace the map textures, and ONLY the map textures, with 8x8 pixel solid color textures, it knocks about 600MB off the memory usage, though the full 700MB gets allocated at load and falls off after. So it's clearly something with how the map textures are being loaded that's such a memory hog.

As for pasting them into Gamedata, are you sure they're actually being loaded into the Memory? If you load a craft without Mapsat on it, it also knocks about 600MB off the memory usage (only with no initial allocation spike), so it's pretty clear that they're not loaded from plugindata until mapsat is started. If you're just dumping them into gamedata with nothing calling them it's entirely possible they're not being fully loaded into Memory.

Anyways, the code for the v4 dev build is like 2500 lines of gut cringing inefficiency. Personally I think it's a good thing that this is being revisited, since the whole thing could use a rewrite from the ground up. First of all, maybe this whole technique of making every single player spend a huge amount of memory and computational resources to pull the terrain data from the game to generate a 2mb png was useful back in the day (I can't see how), but we have ridiculously precise elevation maps of all the planets now.

IS isn't exactly a professional developer. He started the original ISA Mapsat project basically just because he wanted to see if he could. He's an amateur programmer with very little training or experience who frankly, did an incredible job considering. You've also missed one of the key features of Mapsat, probably because it's crippled in the developer version: Mapsat doesn't just make a '2mb png', that's just the ingame map. It's also capable of saving the raw scanned data into CSV files, which can be used to generate a map of an arbitrary size and resolution, limited only by the scanning resolution you managed. On Kerbin, this can be as high as 18 times the resolution of the default map.

Edit: That map you linked uses the same basic method to generate the map that ISA does, to wit: "The map data is a collection of point samples..." :End Edit

He's also been doing this in his spare time, and when I spoke to him on IRC a few months back, he told me that he hadn't actually played KSP in a very long time because he spent all his KSP time working on Mapsat.

Even without those, you only need to generate the terrain png ONCE (as the developer) then have a way to slowly reveal it to the player as they fly over.

To show you what I mean, I wrote a really crude test version so you can see how it would work. I used the 16k elevation maps (which are way overkill) and rendered them in some GIS software using roughly the same color scheme as ISA and exported them at the same resolution:

Minmus map from ISA.... and one I generated

[*Snip*]

Then I added an all black (full transparent) alpha layer to the map I made (which also conveniently makes the map invisible to the user in Windows). Then I wrote a really short plugin (<100 lines) that's tied to one of the stock antennas that basically pulls the lat/long coordinates and transforms them using a map projection to get the x y pixel coordinates on the png. Then all it does is flip the alpha channel of the pixels to opaque as the user flies over which "reveals" the map, which is displayed in a GUI box atm:

*Snip*

^There you go, really rough version of ISA that uses virtually no processing power, doesn't lag the game at all, and makes maps that are even more accurate because you don't have the inherent inaccuracies of pulling game data. Check out the comparison (both at the same resolution):

*Snip*

As far as memory, even if you loaded all the textures beforehand it wouldn't take more than 150mb or so, but I see no reason to do that since you could just as easily load the texture of the planet only when you enter it's SOI. I'm also fairly certain you could use Resources.UnloadAsset() on the textures when you leave the planets SOI, which would mean you would only have the textures loaded for the planet you're at (so a maximum RAM usage of maybe 10-15mb during runtime, 0mb if you're not near a planet). What do you guys think?

That method has a serious, serious problem: If you're using a pregenerated map in the first place, why bother with 'scanning' at all? Especially as it'd be virtually impossible to prevent someone from editing the save data to indicate it was fully mapped when it wasn't, or just to strip the alpha so it didn't do that in the first place. You might as well just make it a 'map for each planet' mod and dispense with the pretense.

I also don't see any kind of indicator of what your present location is, which hinders navigation. It's also lacking the Polar views, which would make navigation near the poles difficult due to the impossibility of creating a rectangular map that accurately conforms to the entire surface of a sphere. And there's no anomaly detection either.

So, it's not only crude, it's vulnerable to cheating AND lacking almost all of Mapsat's core features. Actually matching Mapsat's level of functionality is going to take a LOT More work than you let on.

I'd also like to see what happens to your memory usage when you try loading 30 maps into memory. I'm betting it ends up quite a bit higher than you'd expect.

Edit: And I forgot to

Wow really really awesome work. Do you think it would be possible to load/unload the map textures on the fly? Tingle figured out a way to unload game textures with Universe Replacer; it might be possible to unload non-active maps to save a good bit of memory.

It would be possible, but it would present a problem: any time you wanted to change what map you were looking at, you'd have to load the new map in. That's not going to happen instantly: There's another thread on these very forums that indicates that the number one thing slowing down the load times is texture loading. So you're looking at either limiting it to viewing the map of the body you're currently orbiting and getting a hang every time you change SOIs, or getting a hang every time you change what map you're viewing if you don't limit it.

Edited September 1, 2013 by Tiron

[Mihara]

There you go, really rough version of ISA that uses virtually no processing power, doesn't lag the game at all, and makes maps that are even more accurate because you don't have the inherent inaccuracies of pulling game data.

Dammit, forums, stop being broken and let me +rep this man.

That method has a serious, serious problem: If you're using a pregenerated map in the first place, why bother with 'scanning' at all? Especially as it'd be virtually impossible to prevent someone from editing the save data to indicate it was fully mapped when it wasn't, or just to strip the alpha so it didn't do that in the first place. You might as well just make it a 'map for each planet' mod and dispense with the pretense.

Pardon me, but it's all really about the pretense and nothing else. Exploratory gaming in a sandbox is inherently a pretend activity. You are not really discovering anything new, someone else has definitely been there before. If I didn't want to bother with scanning, there's always an option for me to use Telemachus -- which ties into www.kerbalmaps.com these days, which offers more than enough resolution for all practical purposes -- and get to it with a phone or tablet, so that map doesn't even take any of my valuable main screen space. That is, there already is a "map for each planet" mod. If I want to dispense with the pretense, I can do it immediately, and this whole discussion would be completely pointless.

There is one function that MapSat offers which this combo doesn't -- it wouldn't list new statics, such as those created with KerbTown. But I expect that if it's about statics and them alone, something can be cooked up relatively quickly.

[Tiron]

Dammit, forums, stop being broken and let me +rep this man.

Pardon me, but it's all really about the pretense and nothing else. Exploratory gaming in a sandbox is inherently a pretend activity. You are not really discovering anything new, someone else has definitely been there before. If I didn't want to bother with scanning, there's always an option for me to use Telemachus -- which ties into www.kerbalmaps.com these days, which offers more than enough resolution for all practical purposes -- and get to it with a phone or tablet, so that map doesn't even take any of my valuable main screen space. That is, there already is a "map for each planet" mod. If I want to dispense with the pretense, I can do it immediately, and this whole discussion would be completely pointless.

There is one function that MapSat offers which this combo doesn't -- it wouldn't list new statics, such as those created with KerbTown. But I expect that if it's about statics and them alone, something can be cooked up relatively quickly.

Actually, there's two more things Mapsat offers which this combo couldn't.

It isn't just a change in the statics it can cope with, it can handle a change in the Terrain Itself without needing an update. Which it actually already HAS, because the current version was released right after 0.20 came out. It's still giving me accurate maps of everything, even though the terrain's been redone on more than a few of the planets.

It's also my understanding that it's also able to cope with having entirely new *planets* added without an update, though for obvious reasons I've yet to see this in action.

Regardless, adding *all* the features mapsat offers wouldn't be the trivial exercise it's been made out to be, or someone else would've done it by now. No alternative offers both the flexibility and completeness that Mapsat does, and a product like Gammoner suggests simply couldn't. Every time the terrain changed, or if a new body gets added, or new objects get added to planetary surfaces, a 'revealer' type 'scanner' is going to have to wait for updated maps and issue a new release containing them.

Not saying it's perfect by any means, but coding and memory issues aside, it's really QUITE an impressive piece of work. Especially for being an experimental dev version he pushed out due to the old version being broken by 0.20.

And I've still yet to hear a confirmed solution OR cause for the memory issues, more specific than 'using way too much memory for the Maps'. If dude above can get all 30 maps loaded and not use 600MB doing it, that would be interesting. So far what I've seen is nothing special or useful.

Edit: Also, if you really wanna +rep him, you can do it by right clicking on the star and hitting 'open link in new tab' or whatever. The link itself and the rep system itself both still work, it's just not doing it when you just click on it for whatever reason.

Edit2: Oh crap, I think I just figured it out. The Map textures are 2048x1024. Uncompressed, with mipmaps, that's only 10.6 MB.

But I just now had an epiphany. And checked it.

2048x2048 uncompressed texture with mipmaps. 21.3 MB.

I'm going to try scaling the map textures up to 2048x2048 and see if that increases the memory usage any.

Edit3: Okay, well, that increased the memory usage like, another 700mb or some stupid thing. Sheesh. There goes that theory.

Edited September 1, 2013 by Tiron

[AncientGammoner]

As for pasting them into Gamedata, are you sure they're actually being loaded into the Memory? If you load a craft without Mapsat on it, it also knocks about 600MB off the memory usage (only with no initial allocation spike), so it's pretty clear that they're not loaded from plugindata until mapsat is started. If you're just dumping them into gamedata with nothing calling them it's entirely possible they're not being fully loaded into Memory.

Yes they are being loaded. As mentioned in my last post I also made it so pressing a key will load a texture and then clear it, but like I said you really don't need to load them all.

You've also missed one of the key features of Mapsat, probably because it's crippled in the developer version: Mapsat doesn't just make a '2mb png', that's just the ingame map. It's also capable of saving the raw scanned data into CSV files, which can be used to generate a map of an arbitrary size and resolution, limited only by the scanning resolution you managed. On Kerbin, this can be as high as 18 times the resolution of the default map.

The elevation map I pulled it from is 64 times the resolution of the map. And perhaps Mapsat can generate maps outside KSP from the raw data, but I seriously doubt anyone actually used that capability since it's not related to playing the game at all. Not to mention the maps from the csv would be worthless at higher resolution because of the inaccuracies of pulling data from the game that I mention in the comparison.

That method has a serious, serious problem: If you're using a pregenerated map in the first place, why bother with 'scanning' at all? Especially as it'd be virtually impossible to prevent someone from editing the save data to indicate it was fully mapped when it wasn't, or just to strip the alpha so it didn't do that in the first place. You might as well just make it a 'map for each planet' mod and dispense with the pretense.

Hmm? This is a computer game, people like having stuff to do. It's like asking why bother making a new engine mod when you can just use hyperedit or infinite fuel. If people want to strip the alpha layer and just have the map, what does it matter. There's nothing preventing people from copying their ISA maps and using them for new installs or sending them to other people, or editing any other part of the game for that matter.

I also don't see any kind of indicator of what your present location is, which hinders navigation. It's also lacking the Polar views, which would make navigation near the poles difficult due to the impossibility of creating a rectangular map that accurately conforms to the entire surface of a sphere. And there's no anomaly detection either.

So, it's not only crude, it's vulnerable to cheating AND lacking almost all of Mapsat's core features. Actually matching Mapsat's level of functionality is going to take a LOT More work than you let on.

Did you actually read my post, I'm beginning to think not, because I clearly stated "I wrote a really crude test version so you can see how it would work". I was demonstrating the core concept, nothing more.

I'd also like to see what happens to your memory usage when you try loading 30 maps into memory. I'm betting it ends up quite a bit higher than you'd expect.

I already stated how I would deal the memory issue.

It would be possible, but it would present a problem: any time you wanted to change what map you were looking at, you'd have to load the new map in. That's not going to happen instantly: There's another thread on these very forums that indicates that the number one thing slowing down the load times is texture loading. So you're looking at either limiting it to viewing the map of the body you're currently orbiting and getting a hang every time you change SOIs, or getting a hang every time you change what map you're viewing if you don't limit it.

Loading a map takes a fraction of a second for me (very brief pause), and the game already "hangs" every time you change SOIs anyways so you wouldn't even notice it.

Pardon me, but it's all really about the pretense and nothing else. Exploratory gaming in a sandbox is inherently a pretend activity. You are not really discovering anything new, someone else has definitely been there before. If I didn't want to bother with scanning, there's always an option for me to use Telemachus -- which ties into www.kerbalmaps.com these days, which offers more than enough resolution for all practical purposes -- and get to it with a phone or tablet, so that map doesn't even take any of my valuable main screen space. That is, there already is a "map for each planet" mod. If I want to dispense with the pretense, I can do it immediately, and this whole discussion would be completely pointless.

Exactly, everything in a computer game is contrived anyways so this whole conversation is pointless.

There is one function that MapSat offers which this combo doesn't -- it wouldn't list new statics, such as those created with KerbTown. But I expect that if it's about statics and them alone, something can be cooked up relatively quickly.

I believe MapSat uses the lat/long coordinates to find stuff just like this method, so I don't see why this couldn't do the same thing.

It isn't just a change in the statics it can cope with, it can handle a change in the Terrain Itself without needing an update. Which it actually already HAS, because the current version was released right after 0.20 came out. It's still giving me accurate maps of everything, even though the terrain's been redone on more than a few of the planets.

It's also my understanding that it's also able to cope with having entirely new *planets* added without an update, though for obvious reasons I've yet to see this in action.

Regardless, adding *all* the features mapsat offers wouldn't be the trivial exercise it's been made out to be, or someone else would've done it by now. No alternative offers both the flexibility and completeness that Mapsat does, and a product like Gammoner suggests simply couldn't. Every time the terrain changed, or if a new body gets added, or new objects get added to planetary surfaces, a 'revealer' type 'scanner' is going to have to wait for updated maps and issue a new release containing them.

Yes if an entirely new planet comes out you'd have to update the mod... and?

Not saying it's perfect by any means, but coding and memory issues aside, it's really QUITE an impressive piece of work. Especially for being an experimental dev version he pushed out due to the old version being broken by 0.20.

And I've still yet to hear a confirmed solution OR cause for the memory issues, more specific than 'using way too much memory for the Maps'. If dude above can get all 30 maps loaded and not use 600MB doing it, that would be interesting. So far what I've seen is nothing special or useful.

You have a better suggestion on how to do things, then suggest it, otherwise you're adding "nothing special or useful" to this thread, seriously. This is a thread about improving upon ISA/maybe developing a new mapping program and all I'm hearing is how great MapSat is... when it's currently a total mess. Do you know anything about plugin writing, have you read the source code at all? You're posting in a development thread but speaking like a player.

[Reddot99]

Anyways, the code for the v4 dev build is like 2500 lines of gut cringing inefficiency. Personally I think it's a good thing that this is being revisited, since the whole thing could use a rewrite from the ground up. First of all, maybe this whole technique of making every single player spend a huge amount of memory and computational resources to pull the terrain data from the game to generate a 2mb png was useful back in the day (I can't see how), but we have ridiculously precise elevation maps of all the planets now. Even without those, you only need to generate the terrain png ONCE (as the developer) then have a way to slowly reveal it to the player as they fly over.

^There you go, really rough version of ISA that uses virtually no processing power, doesn't lag the game at all, and makes maps that are even more accurate because you don't have the inherent inaccuracies of pulling game data. Check out the comparison (both at the same resolution):

As far as memory, even if you loaded all the textures beforehand it wouldn't take more than 150mb or so, but I see no reason to do that since you could just as easily load the texture of the planet only when you enter it's SOI. I'm also fairly certain you could use Resources.UnloadAsset() on the textures when you leave the planets SOI, which would mean you would only have the textures loaded for the planet you're at (so a maximum RAM usage of maybe 10-15mb during runtime, 0mb if you're not near a planet). What do you guys think?

I believe MapSat uses the lat/long coordinates to find stuff just like this method, so I don't see why this couldn't do the same thing.

Yes if an entirely new planet comes out you'd have to update the mod... and?

You have a better suggestion on how to do things, then suggest it, otherwise you're adding "nothing special or useful" to this thread, seriously.

Actually, there's two more things Mapsat offers which this combo couldn't.

It isn't just a change in the statics it can cope with, it can handle a change in the Terrain Itself without needing an update. Which it actually already HAS, because the current version was released right after 0.20 came out. It's still giving me accurate maps of everything, even though the terrain's been redone on more than a few of the planets.

Suggestion here, why not have Gammoner's code able to 'update' areas as they're flown over. In other words redraw that little segment of the map image being 'scanned' with a fresh read of that terrain. Figure out what the scanning arc would be, and make an array of raycasts coming straight down over the coordinates of the pixels, that reports the first position they encounter something solid. That position is used to repaint that little segment of map, possibly catching anything large enough (or close enough, if another craft happened to pass below you.) to affect that ray. (This would be a runtime toggle, ideally.)

Seems sorta like a fair compromise between the procedural methodology/mapping of MapSat and the memory reductions of Gammoner's version.

Also, I assume the terrain map generation would be best done on the first entry of a mapping-enabled craft into each world's sphere of influence (or perhaps with a gui button trigger while in a stable orbit), possibly putting a KSP version flag onto the map, so it'll know to rescan only if the image is either out of date, or missing from it's 'library'.

Edited September 1, 2013 by Reddot99

[John FX]

I would be very happy with a new mapper mod that had premade maps that were revealed when flying over them if it used hardly any CPU or RAM, my two main problems with Mapsat. As an end user there would be no difference to my experience. The only issue I can think of is if new anomalies were placed, then the quick and dirty mapper would not catch them until the next mod update. I am guessing somewhere there will still be a slow mapper that would catch them all. Maybe the quick mapper could have an offline verifier that checks the validity of the quick maps and flags any changes? Just leave your game running overnight with the `check maps` flag and it will just crunch map points and verify the current maps? might be the best of both worlds, speed whilst playing and full map checks when not...

[Tiron]

I would be very happy with a new mapper mod that had premade maps that were revealed when flying over them if it used hardly any CPU or RAM, my two main problems with Mapsat. As an end user there would be no difference to my experience. The only issue I can think of is if new anomalies were placed, then the quick and dirty mapper would not catch them until the next mod update. I am guessing somewhere there will still be a slow mapper that would catch them all. Maybe the quick mapper could have an offline verifier that checks the validity of the quick maps and flags any changes? Just leave your game running overnight with the `check maps` flag and it will just crunch map points and verify the current maps? might be the best of both worlds, speed whilst playing and full map checks when not...

I've yet to see anyone demonstrate that it's POSSIBLE to use less RAM while still loading all the maps. I've been chatting with people that actually DO make mods about it, including Innsewerants himself, and it was a pretty interesting discussion. He said he's going to look at the memory usage when he gets a minute, because he's as mystified as I am. Someone else speculated that it might be loading it into memory twice, once in Unity, which then throws it to OpenGL, but doesn't bother to wipe the original when it does.

If someone can set up something that loads 30 2048x1024 textures at once with less memory usage, then we'd have found something useful. Equally if someone actually does set up something that dynamically loads and unloads them and can demonstrate it not causing unacceptable slowdown when it does, that would be useful.

Right now mostly what I'm seeing is a lot of uninformed whining. There's a general assumption that it could be done better without any actual data to base that on, because nobody has so far done anything even close.

For a start, try this one on: Unity definitely loads the textures as uncompressed. 30 maps at 2048x1024, uncompressed, without mipmaps would take up 240mb. That's the bare minimum without lowering the detail level. So how does he come up with 150mb if they are being loaded correctly?

And For the record, I said that you could do about 18 times the resolution on Kerbin with Mapsat. That maximum is not because that's as high as it can go, it's because that's as high as you can go while still remaining outside the atmosphere. You can go to an arbitrarily high level, but only by using aircraft. The resolution is determined by your altitude. The maps are a fixed size, however, and thus have a higher resolution for smaller bodies than for larger ones. With the mun, for example, you can again get a little over 18 times the ingame map's maximum resolution with a 23km orbit. In that case you can go down to as low as 7.5km without hitting the surface, so you can go MUCH higher resolution than that, at a greatly increased time and difficulty. But the Mun's ingame map is already 3 times the resolution of Kerbin's to start with: so at 23km you're getting a map with 54 times the resolution of Kerbin's ingame ISA map.

But it boils down to this: Uniformed whining isn't going to help anything. People are whining about Mapsat causing problems because, frankly, they're overloaded on other mods. They see Mapsat as being the cause of the problem because they 'only have problems when I use a ship with mapsat on it'. I've already established that Mapsat only loads the maps into memory if you have a Mapsat module on your ship. If there's not enough room left for it to do that because you're using 27,000 different add-on parts and didn't bother to account for that, well frankly that's your problem. He's going to look at the Memory usage issue. I'm personally pretty sure that it's a problem with the way the game loads textures and not something that can be reasonably solved without degrading the functionality.

And seriously, if you think you can do better? Quit whining about it and DO it. Or at least supply hard data demonstrating that you're correct. Complaints and speculation don't help anyone, we need hard FACT, and so far I'm pretty much the only one trying to get any.

[Torminator]

But it boils down to this: Uniformed whining isn't going to help anything. People are whining about Mapsat causing problems because, frankly, they're overloaded on other mods. They see Mapsat as being the cause of the problem because they 'only have problems when I use a ship with mapsat on it'. I've already established that Mapsat only loads the maps into memory if you have a Mapsat module on your ship. If there's not enough room left for it to do that because you're using 27,000 different add-on parts and didn't bother to account for that, well frankly that's your problem. He's going to look at the Memory usage issue. I'm personally pretty sure that it's a problem with the way the game loads textures and not something that can be reasonably solved without degrading the functionality.

And seriously, if you think you can do better? Quit whining about it and DO it. Or at least supply hard data demonstrating that you're correct. Complaints and speculation don't help anyone, we need hard FACT, and so far I'm pretty much the only one trying to get any.

What people are trying to do is reduce the memory footprint of Mapsat so that it can actually be used alongside other mods. I'd love to use Mapsat, but I'm not, because I'm already at the precipice of the memory limit, and can't really get rid of the mods that I use. Any optimization of a mod's memory usage is a good thing.

Also, Ancient Gammoner IS doing it, and as far as I can tell, IS demonstrating that the memory usage can be reduced. He even mentioned that dynamically loading maps could cause a slowdown, but it could easily be cloaked within the SOI change loading. I'm not sure why you're being so defensive about this...

[Tiron]

What people are trying to do is reduce the memory footprint of Mapsat so that it can actually be used alongside other mods. I'd love to use Mapsat, but I'm not, because I'm already at the precipice of the memory limit, and can't really get rid of the mods that I use. Any optimization of a mod's memory usage is a good thing.

Also, Ancient Gammoner IS doing it, and as far as I can tell, IS demonstrating that the memory usage can be reduced. He even mentioned that dynamically loading maps could cause a slowdown, but it could easily be cloaked within the SOI change loading. I'm not sure why you're being so defensive about this...

His 'lower memory use' is below what I understand to be the minimum possible for loading all 30 maps, so it's plain that something's not right there.

Dynamically loading and unloading them is fine, the question is what kind of a hang does it have. Part of ISA is being able to look at any map at any time, and if it's going to hang it would every time you switched maps. If you can do that without causing a hang, that'd be very useful in reducing the memory footprint, but I've yet to see someone test that.

I'm not a developer so I've no clue how to do that myself, or I would. I'd also set up something just to load all 30 maps and see how much memory that used.

I'm also being told that Unity itself can load compressed DDS textures, but KSP doesn't allow it for some reason. That would help tremendously as well.

And I'm defensive because, well, I don't like it when people tear something down based on incorrect ideas stemming from Ignorance. I very much follow the scientific ideal that you should be able to hold your ideas up to experiment. People tearing down something fantastic based purely on ignorance pisses me off something fierce.

Do I say it's perfect and couldn't be improved? No. But I've yet to see one verifiable way to improve it come out of this so far(Other than drastically improving the CPU use at the cost of defeating the entire point.)

[AncientGammoner]

I've been chatting with people that actually DO make mods about it, including Innsewerants himself, and it was a pretty interesting discussion. He said he's going to look at the memory usage when he gets a minute, because he's as mystified as I am.

The guy hasn't logged into his forum account in 3 months and you managed to talk with him... hmmm. Forgive my skepticism, but please tell us all how you accomplished this magical feat.

Equally if someone actually does set up something that dynamically loads and unloads them and can demonstrate it not causing unacceptable slowdown when it does, that would be useful.

I already did this and wrote about how to do it, read the rest of the thread if your going to post.

So how does he come up with 150mb if they are being loaded correctly?

Maybe they weren't, maybe that numbers off, but the whole test quickly became completely irrelevant once I figured out how to delete the unused maps from memory, there's no need to load them all.

And For the record, I said that you could do about 18 times the resolution on Kerbin with Mapsat....

Again, you didn't read what I posted. Any extra resolution with MapSat is worthless because of the inherent stuttering you get from pulling data from the game while flying. From my previous post:

But it boils down to this: Uniformed whining isn't going to help anything.

How ironic, the only person whining here is you.

Or at least supply hard data demonstrating that you're correct. Complaints and speculation don't help anyone, we need hard FACT, and so far I'm pretty much the only one trying to get any.

I confess this made me LOL. First off, I already posted how to do it better, with a demonstration and screenshots. What programming suggestions have you posted? Trying to get facts, laff, the only thing you're doing in this thread is cutting down everyone who tries to suggest improvements on the 'super incredible amazing' MapSat. You honestly come off as sounding like you wrote the program and won't accept criticism of it.

[Tiron]

The guy hasn't logged into his forum account in 3 months and you managed to talk with him... hmmm. Forgive my skepticism, but please tell us all how you accomplished this magical feat.

I already did this and wrote about how to do it, read the rest of the thread if your going to post.

Maybe they weren't, maybe that numbers off, but the whole test quickly became completely irrelevant once I figured out how to delete the unused maps from memory, there's no need to load them all.

Again, you didn't read what I posted. Any extra resolution with MapSat is worthless because of the inherent stuttering you get from pulling data from the game while flying. From my previous post:

How ironic, the only person whining here is you.

I confess this made me LOL. First off, I already posted how to do it better, with a demonstration and screenshots. What programming suggestions have you posted? Trying to get facts, laff, the only thing you're doing in this thread is cutting down everyone who tries to suggest improvements on the 'super incredible amazing' MapSat. You honestly come off as sounding like you wrote the program and won't accept criticism of it.

He logs into the IRC chat just about every day. Spoke with him yesterday morning. He's somewhat mystified about the memory usage himself and says he'll look at it. Apparently he's been spending most of his time working on his own game idea he's had, and hasn't touched Mapsat or the forums because everyone indicated to him that the dev version still worked. Which it does.

On the other hand I just discovered something absolutely incredible memory usage wise that I'm going to be making my own thread about, which is...very mysterious.

Edit: Okay, they say this is 'a known bug', but it might still be useful.

If you suspend windows with KSP open and a flight in progress, it uses A FEW HUNDRED KB MB [Why did I type KB?] until you load something else. Even after several loads it was still depressed 500-600 KB MB below my normal levels.

And that's on a ship WITH Mapsat with all the Maps loaded.

Edit2: Okay, if I interpret what I'm seeing now correctly, the memory problems in general appear to be the following:

Almost everything KSP loads into memory at any point is cached, remaining in memory even after it's not actually being used out of main memory anymore. Textures and Part Models that are already loaded into Vram. This could explain Mapsat's apparently high memory usage rather handily: The textures are loaded, decompressed, and passed to OpenGL, which passes it to the vidcard(At some point re-compressing them as DDS textures). The subsequently unused results of each of those steps remains cached in main memory afterwards.

This also accounts for the 'memory leak': Every time something that wasn't being used previously is loaded into Memory, it stays there afterwards. If you're switching between craft that have parts on them that weren't used on a ship you'd focused previously, those assets go through the entire process and then THAT sits there in main memory as well. This would result in climbing memory usage if you were playing in such a way that more and more previously-unloaded parts were getting loaded as time went on. That would account for why I've never seen it: I tend to use a pretty limited subset of parts, in part because I don't really use parts packs and in part because there are significant numbers of parts I tend to avoid using. If this is the case, however, in theory loading into the VAB should load...well, most things if not everything. It uses the part models and textures to draw the icons on the GUI, so in theory even things you don't actually use would get loaded. They might not get FULLY loaded unless you actually went to whatever page they're on. I suppose it's possible the VAB has better garbage collection on it for this reason too though, but that's just speculation.

The key reason that it's a problem looks like, based on what I can see anyway, that the things that are merely being 'cached' are being flagged like they're in active use. In theory, memory that's not in active use but merely storing cached things is supposed to be able to be reallocated if needed. It seems fairly likely to me that KSP isn't doing that, for whatever reason, because if it were it seems like there ought to be a lot fewer instances of 'out of memory' errors, if any at all (it really appears to not use very much at all the majority of the time!)

Edit2:

Also, I did read your post, I just didn't tear it apart completely enough for your taste evidently.

You only get stuttering if you've set your 'Resolution' setting too high for your CPU to handle(It's not really the Resolution, though, it's the number of scanlines it uses,) and even then only while you're actively scanning. See, there's this little button on the UI that turns the scanning feature off and cuts the CPU use to basically zero. There's also a 'GPS' part that's incapable of scanning at all.

Also, I can cut map segments out and blow them up by a factor of 2 as well:

Strangely it looks better than your 'ISA' image despite the fact that it's also 127x101 image blown up to 254x202 (a process that would naturally exaggerate any quality problems, scanline artifacts, or compression artifacts present.)

Which also raises the question of what the heck resolution are you using for your maps, and did you actually load that into the game? Because to get resolution THAT much better it'd have to be a MUCH larger map, and therefore using even more memory... especially since recent evidence suggests that unloading it might not actually even unload it.

Your...thing isn't even an approximation. It's a crude demonstration of an exceptionally basic method to accomplish a superficially similar task. It's missing almost everything that makes Mapsat useful, and it additionally contains very little to suggest it's actually better. Your 'Scanned' image isn't complete enough to gauge the resolution you've actually used ingame, you didn't do a complete enough job to verify the memory usage actually IS any better, and there's quite obviously serious problems with the way you're tracking the current location and applying it to the map (You're going off the top and bottom of the map, and your 'scanlines' are perfectly straight and symmetrical at all latitudes. Neither of those are possible on a rectangular map.)

Edit3:

Oh, I see. 16384x8192 resolution maps. No wonder you didn't mention the memory requirements, because by my calculations a 32bit uncompressed bitmap at that resolution is 536,870,912 bytes, or pretty much dead on 512 megabytes. Given that Mapsat's memory usage was over double the size of the uncompressed version of the texture and there's evidence that suggests that may be intrinsic...well, that'd put you somewhere in the neighborhood of 1.2gb of RAM to load your one map. Gee.

Also, comparing an image that's been upsized by a factor of 2 to one that's been downsized by a factor of 4 (which is at least equivalent to 4x supersampling anti-aliasing) is kinda dirty pool, don't you think?

Edited September 3, 2013 by Tiron

[Majiir]

In the interest of keeping this thread open, let's just cool down a bit.

Using textures to store any kind of data that isn't actually an image is a bad idea if you're worried about memory consumption. (I'll skip over all the ways it reduces code quality, et cetera.) At runtime, you need to have a texture that renders a map onto the screen (if you're making a 2D display), but this is completely different data than what you generate when you actually perform a scan. When you scan an area, fundamentally you're saying "I've been here, so I want access to that elevation data later" regardless of whether that's what you actually do in memory. If you need to store that elevation data (e.g. for performance reasons) you can do that, but don't store it as a texture, because a texture stores colors, not elevations. CPUs are really, really fast, and converting elevations to colors won't impact performance. Even if you don't use any kind of compression, you're making a savings of at least a factor of 24 in memory.

[Tiron]

In the interest of keeping this thread open, let's just cool down a bit.

Using textures to store any kind of data that isn't actually an image is a bad idea if you're worried about memory consumption. (I'll skip over all the ways it reduces code quality, et cetera.) At runtime, you need to have a texture that renders a map onto the screen (if you're making a 2D display), but this is completely different data than what you generate when you actually perform a scan. When you scan an area, fundamentally you're saying "I've been here, so I want access to that elevation data later" regardless of whether that's what you actually do in memory. If you need to store that elevation data (e.g. for performance reasons) you can do that, but don't store it as a texture, because a texture stores colors, not elevations. CPUs are really, really fast, and converting elevations to colors won't impact performance. Even if you don't use any kind of compression, you're making a savings of at least a factor of 24 in memory.

Okay, that's kind of a point, but I'm not sure how exactly you'd apply that to an actual topographic map. I mean, sure, it works for Kethane because you're only having to store a few, smallish textures and use them over and over. Mapsat IS perfectly capable of storing an elevation dataset as CSVs, but I can't personally figure out how you'd get that into a usable ingame form without generating a texture out of it that's the same size as the maps we're already using, which is going to hit the memory just as hard. There aren't really repeating elements you can reuse like there are for your Kethane Hex-Map.

Also, did you miss the part where I said the high memory usage appears to be entirely because it's leaving things loaded in cache that it's not actually using at the moment? I had a mapsat equipped plane sitting on the runway earlier using less than 400,000K of memory for the entire game, even with all the maps loaded.

It looks like the game's caching all the intermediate steps in getting it off the hard drive and into the vram. Worse, the sheer number of complaints about out-of-memory crashes suggests it's not reallocating that memory when it runs low. Even worse, that could also lead to the 'memory leak' some people say they're encountering, and technically it would actually qualify.

Which would mean almost the entire memory usage of the game is one massive, non-iterative memory leak.

Edit:

Working set: Total memory currently being used by the program.

Peak Working Set: Maximum value of Working Set achieved during the current session.

Private Working Set: Memory being used by the program that cannot be reallocated.

Commit Size: Total Memory Allocation reserved for use of the program.

PF Delta: The change in the number of Page Faults since the last update: basically the current rate page faults are occuring at. A page fault is the term for when a requested item is not found in memory, and thus has to be retrieved.

Edited September 3, 2013 by Tiron

[Majiir]

Okay, that's kind of a point, but I'm not sure how exactly you'd apply that to an actual topographic map. I mean, sure, it works for Kethane because you're only having to store a few, smallish textures and use them over and over. Mapsat IS perfectly capable of storing an elevation dataset as CSVs, but I can't personally figure out how you'd get that into a usable ingame form without generating a texture out of it that's the same size as the maps we're already using, which is going to hit the memory just as hard. There aren't really repeating elements you can reuse like there are for your Kethane Hex-Map.

Haven't you heard of arrays (or other collections)? I don't like rectangular map formats, but if that's what you want, a rectangular array is all you need. If you're storing booleans, be sure to use something like BitArray (you'll have to add the second dimension manually) and not bool[,].

Also, did you miss the part where I said the high memory usage appears to be entirely because it's leaving things loaded in cache that it's not actually using at the moment? I had a mapsat equipped plane sitting on the runway earlier using less than 400,000K of memory for the entire game, even with all the maps loaded.

I didn't miss anything. It's a good find, and reinforces that textures should be used sparingly.

[jrandom]

Couldn't the program just use memory-mapped files and let the OS handle caching? Break each set of sampled points into segments, read/write the appropriate segment when necessary, and disk activity will only happen when the OS decides it's necessary. Put the reader/writer in it's own thread and you've got something that won't pause KSP -- the map window could simply display a little red light meaning "reading data" and display when ready. Writes would simply be handed off to the read/write thread. No stuttering, tiny memory footprint. Textures can be generated on-the-fly from the sampled points. There's no reason to have every high-res map loaded in memory all the time, right?

[Tiron]

Haven't you heard of arrays (or other collections)? I don't like rectangular map formats, but if that's what you want, a rectangular array is all you need. If you're storing booleans, be sure to use something like BitArray (you'll have to add the second dimension manually) and not bool[,].

True, but to get any kind of resolution you'd need a LOT of them, and that's going to be a major pain in the butt to code.

And there's a reason that the map comes in two modes: Rectangular and Polar. The Polar uses a pair of polar azimuthal projections laid out side-by-side. You can swap back and forth between the two on the fly: This is the reason there's 30 maps instead of 15.

Couldn't the program just use memory-mapped files and let the OS handle caching? Break each set of sampled points into segments, read/write the appropriate segment when necessary, and disk activity will only happen when the OS decides it's necessary. Put the reader/writer in it's own thread and you've got something that won't pause KSP -- the map window could simply display a little red light meaning "reading data" and display when ready. Writes would simply be handed off to the read/write thread. No stuttering, tiny memory footprint. Textures can be generated on-the-fly from the sampled points. There's no reason to have every high-res map loaded in memory all the time, right?

I can't see why not, except that KSP seems to be botching something up here: When I check the memory usage without doing my little trick, the Private Working set is still only a tiny bit smaller than the full Working set: Somehow, all the 'cache' bits aren't getting flagged as such (or at least that's what it looks like). Which is probably the sole reason textures, including Mapsat's maps, are using so much memory in the first place.

Which isn't a problem in and of itself, but I get the impression one of the side effects is that it's not dumping parts of the cache to make room for new stuff when it needs to, which is probably a lot of what's resulting in the out-of-memory errors people with a lot of mods get.

Edited September 3, 2013 by Tiron

[Majiir]

True, but to get any kind of resolution you'd need a LOT of them, and that's going to be a major pain in the butt to code.

What? No, you need exactly one. C# has true multidimensional arrays (not jagged like Java) and even if it didn't (e.g. BitArray is one-dimensional) you can fake it by multiplying one coordinate by the length of the prior dimensions.

[AncientGammoner]

Using textures to store any kind of data that isn't actually an image is a bad idea if you're worried about memory consumption. (I'll skip over all the ways it reduces code quality, et cetera.) At runtime, you need to have a texture that renders a map onto the screen (if you're making a 2D display), but this is completely different data than what you generate when you actually perform a scan. When you scan an area, fundamentally you're saying "I've been here, so I want access to that elevation data later" regardless of whether that's what you actually do in memory. If you need to store that elevation data (e.g. for performance reasons) you can do that, but don't store it as a texture, because a texture stores colors, not elevation. CPUs are really, really fast, and converting elevations to colors won't impact performance. Even if you don't use any kind of compression, you're making a savings of at least a factor of 24 in memory.

If I understand the approach your thinking of, it's basically to generate the texture displayed in the GUI (which is much smaller resolution than 2k) and change each pixels color in real time based on stored data and the current zoom/scroll positions? Interesting... I mean far more work to build a plugin that would use that method but I suppose that would use the least memory. What sort of storage method did you have in mind that you could easily read specific pieces of but that the game didn't insist on loading the whole thing into memory? I'm not sure if the extra effort is even necessary though... personally I just want to see a mapping program that doesn't use a crazy amount of memory like 700mb, whether it uses 20mb or 1mb makes less of a difference.

[Tiron]

What? No, you need exactly one. C# has true multidimensional arrays (not jagged like Java) and even if it didn't (e.g. BitArray is one-dimensional) you can fake it by multiplying one coordinate by the length of the prior dimensions.

I meant a lot of CELLS. The current ISA maps are 2048x1024 (although the polars map obviously has substantial areas that aren't part of the actual map because it has two round bits in a rectangular texture.) Easy enough to set up logically, but I can just imagine that setting it up graphically could be a real pain.

You'd also have to either recode recode the way the plugin records the raw scan data, or set it up to translate it into the matrix format on the fly. Right now it's set up to save it as CSVs that pretty much just defines the coordinates of the scan location and the altitude of the result(I think, anyway.) It's kinda crippled in the dev version, I'm not sure it's working quite right, and he didn't include the map generator program with it either (something about redoing the way it works, or something), though the 'RAW' option is present and does generate a file. It was designed to save whatever data you managed to collect very precisely so that you could make maps of an arbitrary size with it, even ones far larger and higher resolution than what you're using ingame.

If I understand the approach your thinking of, it's basically to generate the texture displayed in the GUI (which is much smaller resolution than 2k) and change each pixels color in real time based on stored data and the current zoom/scroll positions? Interesting... I mean far more work to build a plugin that would use that method but I suppose that would use the least memory. What sort of storage method did you have in mind that you could easily read specific pieces of but that the game didn't insist on loading the whole thing into memory? I'm not sure if the extra effort is even necessary though... personally I just want to see a mapping program that doesn't use a crazy amount of memory like 700mb, whether it uses 20mb or 1mb makes less of a difference.

It's the method he uses for his Kethane Hex Map. It stores the scan data basically as a mask in the savefile, with a single character indicating if each segment's been scanned or not. There's some other data in there which I'm not sure exactly what it's for, but it's a really simple solution that only requires 1 byte per cell. He's also got it set up to have the cell colors be based on the amount of Kethane in the cell, and there's no actual on-disk textures for any of it. So far as I can tell, he generates them on the fly too. It's very slick.

Unfortunately if I'm right about this memory thing, his method might be just about the only way to keep it from using tons of memory. Unless the overall 'cache' thing gets resolved, in which case it won't really matter that much.

Edited September 3, 2013 by Tiron

[AncientGammoner]

Also, comparing an image that's been upsized by a factor of 2 to one that's been downsized by a factor of 4 (which is at least equivalent to 4x supersampling anti-aliasing) is kinda dirty pool, don't you think?

Please stop editing your old posts to add new stuff to them, its annoying. And to answer your question, both maps were 2048x1024 when I cropped them. They were both scaled up x2 so you could actually see the differences better.