Any way to track how mch memory each mod consumes?

[MisterFister]

I'm a non-purist when it comes to KSP, and I constantly run into the 4GB ceiling for installing and running mods. It's difficult to know how much of that problem is generated by any given mod install. When troubleshooting, I have no problem simply uninstalling mods one by one in reverse-order of how important I think they are to what I want to accomplish in my savefiles. However, I have to do this blindly, since I have no way of prioritizing them based on actual memory footprint. Something tells me that at least sometimes it might be possible to uninstall one mod to solve my problem rather than pulling three smaller mods to accomplish the same goal.

Is there a way to monitor or log memory footprint size on a per-mod basis? Or even a per-install basis (I'd take a baseline sterile install, then install one mod at a time, running the utility after each one, to generate a holistic view)? I run Windows 8.1 and Windows 7 on two different machines, and this seems like good info to know. I can use the log files to generally pinpoint "where" in the startup the memory crash occurs, but I cannot identify how culpable any one mod might be.

Yes, I know I can run texture reduction packs. I'm not asking how to REDUCE memory consumption, I'm asking how to TRACK it. Any thoughts?

[diomedea]

There is a tool I once used that could come useful to allow to see how a process (KSP.exe being a single process) uses memory: Cheat Engine. Though, trying to get good readings without knowing how the data to be tracked is called, is probably quite difficult. Never tried on KSP myself.

[Faark]

No, you cannot, unless you find a way to attach a unity profiler to KSP. Those info is scattered all around Unity. You best bet is to look for obviously heavy resource files like images (usually take up sth between (width*height/2) and (width*height*4 for TGA)) or sounds and guess...

[MisterFister]

To clarify, I am already aware that smaller plugin-type mods generally tend to have less footprint impact than a part-heavy mod such as B9 / KW / NovaPunch, and that part-centered mods tend to scale in footprint in loose proportion to the amount of parts contained therein. Still, I was kind of hoping that there's a way to look either on a per-mod basis, or even a per-part basis, as to how much memory is being allotted. (Per-part allotment would allow me to experiment with cutting individual parts or entire sections / categories of parts from specific modpaks. For example, procedural wings tend to render most pre-fab wing parts (generally over-represented in any given modpak to begin with) almost or completely obsolete.

I already feel comfortable reading the user-parsable [part].cfg files, such that I have a general feel for parts that are longer or shorter... but those file contents, to my admittedly vague understanding, constitute a paltry sum in comparison to the "rest" of the part asset, particularly the model and texture meshes. This is why, for example, texture reduction modpaks (mod-of-a-mod) bear so much fruit when implemented.

What I want to know, however, is if the size of the mesh files is directly correlated to memory footprint. Do memory-hungry part assets always derive from modfiles that take up more file space? Is this a linear relationship, exponential, logarithmic, or an uncorrelated random relationship? While I would be very surprised, based on what I know of the broad outlines of how computer programs work in general, if I could track these on a per-file or per-asset basis, I would at least like to know if I can macro-snapshot the footprint usage. To explain by analogy, sometimes at the vet's office a dog won't sit still on a scale long enough to be weighed. Therefore, you use a human handler to stand on the scale, holding the pet, and then you weigh that handler with no pet in his or her hands, so you can deduct the weight of the human from the combined total to derive the weight of the dog; I can run sterile-KSP, get a snapshot of footprint size, install my first mod, get a snapshot, and the difference is then an approximation of the footprint size of that mod. As long as I uninstall that first mod, sterilize my KSP install, and then install any additional modpak individually, I should be able to always deduct the size of my sterile footprint to calculate the approximate footprint size of any given mod.

I should like to emphasize here that I'm convinced that something along these lines has to exist already. With the texture reduction packs (Squad, B9, KW, and NovaPunch being the ones I browse past repeatedly and often on SpacePort) advertise average memory reductions in the real of XXXMB, often several hundred megabytes each. So someone already did what I'm asking about -- they ran a non-reduced install, took something like my snapshot idea, and then reduced it, to derive what benefit the reduction yielded.

I guess I'm asking how that information was obtained, so I can turn around and experiment on my own. (For example, I might find interconnective patterns. Perhaps Mod1 might have a footprint difference of 100MB, Mod2 of 55MB, but running Mod1 and Mod2 together would mean a footprint size of 135MB above the size of sterile KSP -- or even 160MB. I'd care to identify and troubleshoot those differences, as I'm convinced that such mod conflicts (or benefits) have to occur somewhere at least occasionally.

[NathanKell]

For textures at least (which take up most of the memory for mods), it shouldn't be that hard to write a tool to trawl through GameDatabase and output how much memory is taken up by textures of the same root folder (it's in the texture URL after all).

EDIT: That said, it's easy to calculate by hand.

Assuming all non-normal-map textures get compressed (and you should be running ATM at least in basic so textures *do* get compressed), then they take up the following amounts of space:

No alpha channel: 512kb per 1024x1024 pixels (1024x1024x24/8 /6 for DXT1)

Alpha channel: 1024kb per 1024x1024 pixels (1024x1024x32/8 /4 for DXT5)

If RGB/RGBA (uncompressed, for normal maps), remove those last divisors (so 3mb and 4mb, the same size as the uncompressed MBM or TGA on disk. If png or compressed TGA, calculate).

Edited April 14, 2014 by NathanKell

[Majiir]

It's important to note that while part memory consumption is fairly easy to predict, plugins are much harder to predict. In particular, plugins can have bursty memory consumption. So, even if everything seems fine when idle, a plugin might need to allocate a large portion of memory even for just a millisecond, and that can contribute to instability. Some plugins might maintain large data sets in memory at all times, too.

The best way to establish a plugin's memory consumption is to inspect the code. (For example, I did a thorough analysis of Kethane and found that it would only use a few megabytes of memory even under peak conditions. This is the result of aggressive memory optimizations.) A good rule of thumb is that anything that maintains a texture (like a mapping mod) probably uses more memory than most plugins.

[MisterFister]

EDIT: That said, it's easy to calculate by hand.

Assuming all non-normal-map textures get compressed (and you should be running ATM at least in basic so textures *do* get compressed), then they take up the following amounts of space:

No alpha channel: 512kb per 1024x1024 pixels (1024x1024x24/8 /6 for DXT1)

Alpha channel: 1024kb per 1024x1024 pixels (1024x1024x32/8 /4 for DXT5)

If RGB/RGBA (uncompressed, for normal maps), remove those last divisors (so 3mb and 4mb, the same size as the uncompressed MBM or TGA on disk. If png or compressed TGA, calculate).

I respectfully hope that someone can make more constructive use of that info than I can. I intuit, however, that it was helpful information and I gather that it is indeed possible to develop a user-friendlier method of deriving the answer to my question, possibly with an external widget of some sort. (My programming familiarity ended with BASIC twenty years ago, and my only modding familiarity with KSP is reading the [part].cfg files.

It's important to note that while part memory consumption is fairly easy to predict, plugins are much harder to predict. In particular, plugins can have bursty memory consumption. So, even if everything seems fine when idle, a plugin might need to allocate a large portion of memory even for just a millisecond, and that can contribute to instability. Some plugins might maintain large data sets in memory at all times, too.

The best way to establish a plugin's memory consumption is to inspect the code. (For example, I did a thorough analysis of Kethane and found that it would only use a few megabytes of memory even under peak conditions. This is the result of aggressive memory optimizations.) A good rule of thumb is that anything that maintains a texture (like a mapping mod) probably uses more memory than most plugins.

This is something I hadn't realized, though upon your explanation it makes sense. Generally, however, my motivation for asking me original question was load-failure-type crashes, where KSP simply refuses to launch to title screen at all. Your statement imparts to me a back-of-the-mind need to always factor a healthy margin when using plugins. Inspecting the code, however, is well beyond my ken, and beyond that of most end-users, I'd wager.

To the unifying theme of both of your replies above that speak of textures; I'm aware that graphics and modeling textures tend to be highly compressible, but not uniformly so -- this means that the size of the asset files within the GameData folders may or may not correlate to the memory consumptive footprint of a mod. This is especially the case with regarding WIP mods that have yet to implement "aggressive memory optimizations" as you describe.

[Trann]

Never mind.

Edited April 15, 2014 by Trann

[Majiir]

This is especially the case with regarding WIP mods that have yet to implement "aggressive memory optimizations" as you describe.

It's more a matter of timing. It doesn't optimize anything but disk space to use compressible image formats because KSP stores everything uncompressed in memory. Older mods typically use uncompressed textures on disk because other formats weren't well-supported, and newer mods use compressed textures because they're well-known to work now.

[NathanKell]

Well, it does compress them in memory, usually--to DXT1/5.

Nor (where plugins are concerned) can you tell just from reading the code, alas--when I increased the frequency of the Voronoi PQSMod (that which makes the small procedural craters on the moon) usage shot up by 500MB (there's caching involved, evidently).

MisterFister: If a texture has the extension .mbm, one of two things will be true. Either it is a normal map and will retain its entire file size in memory, or it is a regular texture. If it is a regular texture, see how large it is: if its divisible by 3 (768kb, 3mb, etc), that means it has 3 channels and will take 1/6 its space in memory (3mb->512kb). If it is not, it is 4 channel and will take 1/4 its space in memory.

The a texture is a png or tga, you can open it in an image viewer and determine (a) whether it's a normal map (it will look either blue-purple or solid gray with darker/lighter highlights) and ( its resolution and number of channels (3 or 4).

Then you can figure out its size uncompressed, and then its size compressed by KSP.

[pxi]

A naive observation/question:

I do a fair amount of PHP programming, it's quite easy to track the memory footprint of a script, by calling functions such as memory_get_usage() and memory_get_peak_usage().

Is there similar functionality available for Unity? And would you be able to return memory used by each dll?

[Faark]

System.Diagnostics would allow you to get the amount of memory allocated by KSP, but you are not allowed to use Diagnostics.Process. I'm not aware of any other in-framework way to get this information, so you would have to use one of the many OS-specific methods. GC.GetTotalMemory might be interesting for mod devs, but only counts "script memory".

Beside that you could use developer tools / debuggers, but that requires a lot of experience with .net/c#. As for a basic KSP user the answer stays the same: You cannot.

Btw, did someone already mention that KSP by default does not compress TGA files? (Or was that fixed in 23.5?)

[MisterFister]

What about going the noob route? Can something silly like Windows Task Manager tell me this info reliably? I see some Processes and whatnots that seem to be KSP-related, but are the numbers shown relative to the question I'm asking?