mr_trousers

How the hell does KSP/its mods take up so much memory?

Recommended Posts

I guess I don't really know how all of this works in the first place, but I'm just confused. It seems that parts especially, and their textures, are the main memory hogs for KSP. But it seems to me that many games, especially modern ones, have just as many assets, and just as many textures, and furthermore these textures are generally in higher resolution than in KSP, and all of these run just fine without taking up crazy amounts of memory. What is it about KSP/a modded KSP game that is different that requires such greater of a load?

It's not really important and I'm not mad about it or anything, but I am confused and I'm just legitimately curious about this, haha.

Share this post


Link to post
Share on other sites

It's a combination of two things:

  • KSP loads all of its textures at startup. A lot of other modern games load textures on demand - they'll load the textures for the elements that are on screen, and unload textures that aren't needed. I recall reading once that this was done to work around problems with the on-demand loading system, possibly related to the other major problem.
  • The DirectX renderer in Unity doesn't release textures from main memory after they've been blatted to the GPU. This is why you can frequently get decent memory savings by running the game with the OpenGL renderer.

I don't know if either of these issues have been fixed in the 1.1 prerelease.

Share this post


Link to post
Share on other sites

Well imagine that 1mb of texture takes up around 1 megabyte of ram, tho  i also feel like the x64 is more  wasteful with overhead than x32 was in 1.05 and sadly neither opengl nor dx11 work anymore which handled textures better overall.

(opengl easily releases all textures, dx11 is able to release textures probably uses heuristics to 'prophesize' what to keep, at least it seemed like that in 1.05 x32)

 

there are a few tools with which you can resize your textures

Share this post


Link to post
Share on other sites

I've always maintained that once the program barrier of 2 GB would be removed (as will be with the upcoming release, it seems) it will simply be replaced by complaints about running out of real memory. It's not like it's a different order of magnitude after all, and when people claim that having five dozen mods active is "not that many mods" one can only expect the best for the future. :)

Share this post


Link to post
Share on other sites
23 minutes ago, lude said:

Well imagine that 1mb of texture takes up around 1 megabyte of ram

This is correct if you're using DXT compressed textures (DDS files.) PNG, TGA, MBM which KSP can still read and use get compressed by the engine into DXT during load, which can sometimes make them larger than the original file (DXT has fixed compression.)

26 minutes ago, lude said:

sadly neither opengl nor dx11 work anymore which handled textures better overall.

If I had to guess, this is probably something that is only due to the pre-release nature of 1.1. I'm guessing this is fixed before full release (I think someone had mentioned it on the subreddit.)

 

7 hours ago, mr_trousers said:

I guess I don't really know how all of this works in the first place, but I'm just confused. It seems that parts especially, and their textures, are the main memory hogs for KSP. But it seems to me that many games, especially modern ones, have just as many assets, and just as many textures, and furthermore these textures are generally in higher resolution than in KSP, and all of these run just fine without taking up crazy amounts of memory. What is it about KSP/a modded KSP game that is different that requires such greater of a load?

Exactly as @stibbons said, Unity preloads textures into memory. However, mods exist that allow you to save a little bit of ram by unloading dynamically: 

 

Share this post


Link to post
Share on other sites
1 hour ago, MainSailor said:

This is correct if you're using DXT compressed textures (DDS files.) PNG, TGA, MBM which KSP can still read and use get compressed by the engine into DXT during load, which can sometimes make them larger than the original file (DXT has fixed compression.)

If I had to guess, this is probably something that is only due to the pre-release nature of 1.1. I'm guessing this is fixed before full release (I think someone had mentioned it on the subreddit.)

 

 

 

I just dug out and downloaded DDS4KSP just because well over 1200 files were PNG and a few others, also resizing them in the progress, I hope that will make a noticable difference from using 13gb before.

 

I really really do hope so because I would love to use dx11 or 12 even if it caused more bugs, it also performs faster

but on the bug tracker and other places squad people said that it's a win10 issue and that neither ogl nor dx11 will be supported in the 1.1 (and that they don't work anymore because they're unity related arguments and not ksp engine arguments or something)

-- edit --

perhaps the tool isn't that useful to me, i just saw how a 1.1mb png got turned into a 25mb one and that despite being resized, ok mbm and quite a few other files are now smaller and the whole game loads in half the time now let's see what happens when i resume my game

Edited by lude

Share this post


Link to post
Share on other sites
4 hours ago, Kerbart said:

I've always maintained that once the program barrier of 2 GB would be removed (as will be with the upcoming release, it seems) it will simply be replaced by complaints about running out of real memory. It's not like it's a different order of magnitude after all, and when people claim that having five dozen mods active is "not that many mods" one can only expect the best for the future. :)

8 GB is pretty standard today, this gives the game around 6 Gb. Note as PS3 / One has 8 Gb I expect pc system demands to increase to 12-16 Gb in the future.

Part of the reason most games has used little memory is that PS3 / 360 only had 512 Mb so developers had to treat memory as weight om Eve rockets upper stage :)

Share this post


Link to post
Share on other sites
4 hours ago, Kerbart said:

I've always maintained that once the program barrier of 2 GB would be removed (as will be with the upcoming release, it seems) it will simply be replaced by complaints about running out of real memory. It's not like it's a different order of magnitude after all, and when people claim that having five dozen mods active is "not that many mods" one can only expect the best for the future. :)

When you run out of physical memory, doesn't Windows just create a swap file on the disk? Seems like it would be laggy, but I don't think it would crash. That said, sometime in the future I may indeed drop $40 and pop in two more 4GB DIMMs. Just because. Also because it feels weird for my Macbook Pro to have more RAM than my desktop.

29 minutes ago, magnemoe said:

Note as PS3 / One has 8 Gb I expect pc system demands to increase to 12-16 Gb in the future.

PS3? I believe you mean PS4.

Edited by Hobbes Novakoff

Share this post


Link to post
Share on other sites
Just now, Hobbes Novakoff said:

When you run out of physical memory, doesn't Windows just create a swap file on the disk? Seems like it would be laggy, but I don't think it would crash. That said, sometime in the future I may indeed drop $40 and pop in two more 4GB DIMMs. Just because. Also because it feels weird for my Macbook Pro to have more RAM than my desktop.

While it will not crash, performance would likely be pure agony. I suspect that 95% of the users running into memory problems now will be fine. But it's the remaining 5% that will be complaining and posting about it. My experience in software development is that removing one barrier simply means people are going to run into another barrier. Unrelated but related; 1.1 seems to give a huge performance boost in relation to the number of parts we can use. Is that used to enjoy a smoothly moving game with the current vessels we build? When I see Das Valdez putting a 700-part monster together on Twitch, I doubt it. Partcount will increase until regular choppiness is achieved!

Share this post


Link to post
Share on other sites
3 hours ago, lude said:

but on the bug tracker and other places squad people said that it's a win10 issue and that neither ogl nor dx11 will be supported in the 1.1 (and that they don't work anymore because they're unity related arguments and not ksp engine arguments or something)

That's disappointing if that's the case. DX11 performs flawlessly for me under Win 7 (I've never had severe rendering issues or graphical bugs, at least.)

3 hours ago, lude said:

perhaps the tool isn't that useful to me, i just saw how a 1.1mb png got turned into a 25mb one and that despite being resized, ok mbm and quite a few other files are now smaller and the whole game loads in half the time now let's see what happens when i resume my game

That's the downside of DXT and fixed compression. PNG's can be compressed rather well (in my experience, not as far down as a JPEG unless you use hardcore tools like PNGGauntlet), but any of that compression is shot when converting to DDS. If I recall correctly, DDS4KSP doesn't use DXT1, which some textures can take advantage of (those that have no alpha or specular layers). I've used the tool previously to convert old mods that still had MBM files, but most others I will convert manually using the Nvidia DDS plugin for Photoshop. DDS textures do speed up loading, as you noticed, because Unity doesn't have to convert the file, it writes the DXT format directly into memory.

Share this post


Link to post
Share on other sites

DX11 worked very well for me too, better and more performant than ogl.

So far I can say that from 12.5-13gb private and an 6-9 gb working set ram usage it got reduced to 10715/5835 despite me having added KSP interstellar extended and the before is without

the loading time is about a third or fourth of what it was before, I read something earlier about ksp or unity doing something with non dxt on each load

and there were also at least 40 very sizy MBM files the current version does DXT1 and 5 but I've also downloaded two more tools to my existing to batch change these files to reduce ram load, tho with 10gb private I can live well.

Share this post


Link to post
Share on other sites

Thanks for the explanation. Honestly it seems pretty crazy that they would load all textures at start up instead of on demand. Are the problems that fixes so considerable? It seems hugely inefficient.

 

1 hour ago, Kerbart said:

While it will not crash, performance would likely be pure agony. I suspect that 95% of the users running into memory problems now will be fine. But it's the remaining 5% that will be complaining and posting about it. My experience in software development is that removing one barrier simply means people are going to run into another barrier. Unrelated but related; 1.1 seems to give a huge performance boost in relation to the number of parts we can use. Is that used to enjoy a smoothly moving game with the current vessels we build? When I see Das Valdez putting a 700-part monster together on Twitch, I doubt it. Partcount will increase until regular choppiness is achieved!

You're surely right that many people will now just be butting up against the new part count and ram barriers, but even so, the fact that much larger ships and much more heavily modded games *can* be supported are still great improvements to the game which greatly improve its quality. The game is still better, even if people will find new limits!

Edited by mr_trousers

Share this post


Link to post
Share on other sites
15 minutes ago, mr_trousers said:

Thanks for the explanation. Honestly it seems pretty crazy that they would load all textures at start up instead of on demand. Are the problems that fixes so considerable? It seems hugely inefficient.

Inefficient, yes. Much easier to work with as an developer, yes.

How much fun would it be if the game froze for a bit to load all the space center textures on your approach to the space center with your bad gliding space plane?

Share this post


Link to post
Share on other sites
12 minutes ago, Daid said:

How much fun would it be if the game froze for a bit to load all the space center textures on your approach to the space center with your bad gliding space plane?

Nah just do it like GTA and have you crash into an object that hasn't actually rendered yet :D

Share this post


Link to post
Share on other sites
13 hours ago, mr_trousers said:

But it seems to me that many games, especially modern ones, have just as many assets, and just as many textures…

No, not really. Many game engines, even modern ones will just crash if you try to actually use too many assets at once, they are just designed not go there.  If you ever played good 'ol halflife you surely noticed how there is U shaped hallway  before every large scene to cut down scenegraph. More modern games may not be so obvious about it, but they still have to use many tricks to prevent you seeing "too much". Most of those are not easily applicable for KSP. You are completely free to design rocket using every available part and game should handle it. (and seeing some things here on forums this is not hypotetic scenario :-)

Its not like KSP can't be better optimized here or there, but that will raise the bar only so much. 

6 hours ago, lude said:

… i also feel like the x64 is more  wasteful with overhead than x32…

Umm, its called x86. I don't want go all grammar bad person on you, but x32 is a different fish. It is, quite ironicaly, ABI designed to avoid some of x64 overhead (code uses 32bit memory model, but can use  64bit registers, linking is PIC etc…). Even more ironicaly, it's a linux thing. It never caught on, which should tell you something.

1 hour ago, Hobbes Novakoff said:

When you run out of physical memory, doesn't Windows just create a swap file on the disk? Seems like it would be laggy, but I don't think it would crash.

No crash, but would still ruin your game and make GPU scream in agony. If you, say, scroll through large document, waiting a second to load some pages from disk is doable. If you want to render twenty frames per second, you don't have that kind of time. (Well, one probably could use swap as crude  ondemand loader, but only between scene changes. If your  scene (read: craft on screen) uses more textures then can be fit into working set its game over)

Share this post


Link to post
Share on other sites

1

5 hours ago, Hobbes Novakoff said:

When you run out of physical memory, doesn't Windows just create a swap file on the disk? Seems like it would be laggy, but I don't think it would crash. That said, sometime in the future I may indeed drop $40 and pop in two more 4GB DIMMs. Just because. Also because it feels weird for my Macbook Pro to have more RAM than my desktop.

PS3? I believe you mean PS4.

Windows create an swap file, this will mostly be of the other junk not the game itself. 
Many games will crash, this is an general issue then playing on minimum specification hardware.

And yes its obviously PS3 who has 512 memory, worse it divided in 256 MB video memory and 256 system memory. No virtual memory, anybody who has played Skyrim on PS3 know that being 5KB low on system memory works as well as an Tylo gravity turn with Pe  at -5 km. 
 

Share this post


Link to post
Share on other sites
6 hours ago, mr_trousers said:

Honestly it seems pretty crazy that they would load all textures at start up instead of on demand. Are the problems that fixes so considerable? It seems hugely inefficient.

Well it was the easiest way to program it. And as others mentioned, in a game like KSP the developers have very little control over what textures need loading it. There's some room for improvement - no need to have Eve surface loaded when the player is landing on Ike, for example. But when it comes to parts, consider that a player could have for example one ship with loads of different rocket parts, and then bring in another ship with loads of different spaceplane parts, and they could be closing at considerable speed, dynamic loading isn't so easy. Does the game freeze as it loads stuff in? Does it show potato-level textures until it loads the better ones and hope the player doesn't notice?

Share this post


Link to post
Share on other sites
19 hours ago, lude said:

i also feel like the x64 is more  wasteful with overhead than x32 was in 1.05

That's to be expected, as all pointers (variables pointing to memory addresses) will take up - drum roll - 64 bits instead of 32.

14 hours ago, Hobbes Novakoff said:

When you run out of physical memory, doesn't Windows just create a swap file on the disk? Seems like it would be laggy, but I don't think it would crash. That said, sometime in the future I may indeed drop $40 and pop in two more 4GB DIMMs. Just because. Also because it feels weird for my Macbook Pro to have more RAM than my desktop.

PS3? I believe you mean PS4.

I've managed more than once to drive KSP to my physical memory limit on Linux, and although I have a whole partition dedicated to swap space, KSP didn't even dare to touch it. Of course a lot of other stuff that was running on my PC as well had been swapped out, but KSP itself remained completely in physical memory at all times.

Share this post


Link to post
Share on other sites
17 hours ago, radonek said:

If you ever played good 'ol halflife you surely noticed how there is U shaped hallway  before every large scene to cut down scenegraph. More modern games may not be so obvious about it, but they still have to use many tricks to prevent you seeing "too much". Most of those are not easily applicable for KSP. You are completely free to design rocket using every available part and game should handle it.

It takes an extraordinarily large rocket to have a similar amount of polygons and textures in a KSP scene as you have in a typical modern game.  Hard to imagine such extremes are a serious considerations in the design of the game.

Having all those parts constantly doing physics interactions even when no external forces are acting on the vessel - that is the major performance hog in KSP (though not a memory hog), not the polycount and amount of textures. 

I see enormous differences in performance between various Unity games with similar content (The Forest vs Savage Lands), so it looks like it's not just a matter of fundamental technical limitations but also a matter of coding skill.

Share this post


Link to post
Share on other sites

It's also halfway impossible to split physics over several cores since none has convened technics and methodology to do it, let's wait a few decades.

 

DDS4KSP reduced my load from 13gb private to ~10gb private despite adding KSPIe before checking the reduction it also loads a lot faster with the 124 mods I have.

Next thing I'll try is reduce some of the big DDS file which weren't affected by it.

I had at least 60 MBM files and over 700 PNGs (excluding all icons and stuff)

Share this post


Link to post
Share on other sites
11 hours ago, Hobbes Novakoff said:

@cantab: The thing is, what if I have a super-fast rocket that can go from Ike to Eve? Or I use hyperedit? The game has to be able to handle that.

I see no reason hyper edit should even be considered by the developers.  It's a hack that completely bypasses the flow of the game.  

Share this post


Link to post
Share on other sites

I suspect KSP suffers from the same problem as the vast majority of current programming: laziness. 

(Boring old fart nostalgia alert)

When you had just a few K of memory to use you had to write elegant code. You had to fight for every byte. This meant you had to really think about what was essential for your program. Now though, with GBs of RAM and TBs of very fast disk space, you can just grab a load of DLLs, stick all your textures in memory, link to every library you might never need, write code that is really way suboptimal, don't bother optimising your data queries and still find that on a powerful development machine it will run OK.       

The plus side though is that it makes coding a lot quicker and simpler if you don't have to care about resource constraints. 

And that is where we were up to KSP 1.0.5. It was developed with a game engine that made things relatively easy to develop but did result in something that could just barely squeeze into a 32b OS. 

Now along comes a 64 bit version and the ceiling gets raised again.  A warning for Squad though along the lines of "Work expands to fill the time available" - if you start building KSP for 64b and throw caution to the wind then you will really ruin the experience for those folks out there who will still be on Mickey Mouse hardware for the next 10 years. 

Share this post


Link to post
Share on other sites
15 hours ago, Foxster said:

I suspect KSP suffers from the same problem as the vast majority of current programming: laziness.

As a current generation software engineer to an old generation software engineer: Laziness is good.

It means you can focus on functionality instead of limitations of your system. You could program KSP so it uses half the resources. But it would take a quadruple amount of hours. Meaning you have 1/4 of the features.

You want to use the stock libraries, they are better tested then any code you will ever make. They also have better APIs and logging/debugging then that you will make.

RAM is cheap and plentiful these days. It's rare these days to encounter anything with less then 4GB ram (unless it's only used for browsing and text editing)

Share this post


Link to post
Share on other sites
2 hours ago, Daid said:

As a current generation software engineer to an old generation software engineer: Laziness is good.

It means you can focus on functionality instead of limitations of your system. You could program KSP so it uses half the resources. But it would take a quadruple amount of hours. Meaning you have 1/4 of the features.

You want to use the stock libraries, they are better tested then any code you will ever make. They also have better APIs and logging/debugging then that you will make.

RAM is cheap and plentiful these days. It's rare these days to encounter anything with less then 4GB ram (unless it's only used for browsing and text editing)

I agree with all you say. Was just saying how we got to where we are. 

As for "RAM is cheap...4GB..." Maybe on Windoze but that's certainly not the case for a phone, some consoles, tablets, in-car systems, etc. So there is a sizeable market out there for lean code. 

Edited by Foxster

Share this post


Link to post
Share on other sites
This thread is quite old. Please consider starting a new thread rather than reviving this one.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.