Decentralize Save-Game files

[AdmiralTigerclaw]

This suggestion involves the KSP save game file structure.

 

As the common phrasing goes, 'back up your save games before you try anything'...  Tends to go along nicely when using mods.

 

However, I feel there is a weakness in the current save-game structure, namely, everything about a game instance is saved to one large file.  Not only does this result in sometimes extended load times (as the game searches through the ENTIRE file), it leaves the file easy to corrupt, and save-games easily lost.  Even a minor syntax error compromising the file because the game itself encounters an error during loading/saving can result in a complete loss of the file.  A writing pass on a save game may result in an incomplete file because of an error, again, total game loss.

 

I would propose to break the save game structure into independent elements less susceptible to corruption-related loss.  The save games would be kept in the folder for that save.

 

1: Master Game Status file. (File)

- Contains information regarding player's funds, science, rep, achievements, tech unlocks, and building status.

 

2: Career Contracts File. (File)

- Contains information on available, active, and successful contracts.  If the contracts file is corrupted, it is simply regenerated based on the rep and achievement status from the master status file.  Active contracts and contract history are lost, but at no real major loss to player activity.

 

3: Flights master file. (File)

- Contains basic information on status and position of any active flights (in-flight or landed).  Used as a pointer to reference individual flights without the software having to read the ENTIRE content of every flight on every load pass, improving speed of loads especially with larger games.

** If the Flights master file is damaged or corrupted, the information from the Individual Flights folder (Below) can be used to regenerate the file.

 

4: Individual Flights (File Folder)

- Contains one file for every active in-game flight instead of placing all active flights into one large mass.  If any one flight save file becomes corrupted, the game can ignore or wipe it without corrupting the entire save structure.  Conversely, a player can 'inject' a flight into a save game by copying a (assumed compatible) flight save into the folder, and it will be added to the flights master file.

 

 

In this manner, if a player's save game file structure becomes partially corrupted at any point, the isolation of the various sections of a game into individual files prevent the damage from wiping the game out.  If the master file is corrupted, the player's career data is lost, but easily located and hand-edited in while the active flights remain undamaged.  If a flight causes a problem and data is corrupted or lost, only the flight is lost.  The idea overall is to improve the robustness and failure-safe behavior of the save game structure.

[Wyzard]

This is going off on a bit of a tangent, but I keep my KSP saves in Dropbox (via a symlink from the KSP folder to the Dropbox folder).  In addition to synchronizing files between computers, Dropbox keeps a history of previous versions of files.  If something gets corrupted, or accidentally deleted, you can restore it to an earlier version.

[AdmiralTigerclaw]

50 minutes ago, Wyzard said:

This is going off on a bit of a tangent, but I keep my KSP saves in Dropbox (via a symlink from the KSP folder to the Dropbox folder).  In addition to synchronizing files between computers, Dropbox keeps a history of previous versions of files.  If something gets corrupted, or accidentally deleted, you can restore it to an earlier version.

This may be true, but one should not have to go to such lengths for a save game.  And I'm certain there are plenty of people who would play KSP, but not know a clue about how to keep their program synced up with a custom cloud setup like dropbox.  There are of course, other benefits I listed aside from protecting saves.

[mattinoz]

So instead of embedding the craft file into the save file it would basically stay as a separate file that gets copied in to the save container and then gets modified/split/merged to show current state.

I wonder if such a move would make multi-player easier. Maybe go further and break master flight file into SOI files. Active physics then only needs deal with writing one the SOI file and the craft files in physics range. Different processes or machines could be handling different SOI co-ordination without dealing with two processes trying to write to the same file.

[Renegrade]

On 2016-04-21 at 7:07 PM, AdmiralTigerclaw said:

However, I feel there is a weakness in the current save-game structure, namely, everything about a game instance is saved to one large file.  Not only does this result in sometimes extended load times (as the game searches through the ENTIRE file), it leaves the file easy to corrupt, and save-games easily lost.  Even a minor syntax error compromising the file because the game itself encounters an error during loading/saving can result in a complete loss of the file.  A writing pass on a save game may result in an incomplete file because of an error, again, total game loss.

KSP loads the entire save into memory anyhow (with no searching at all), splitting into multiple files would actually slow things down with more unnecessary file I/O.  (Unity used to, and may very well still have, a hideous bug where any file open operation checks network connections as well as local files for some idiotic reason that could cause massive slowdown on any file open if the system had certain types of network interfaces, which would make this a thousand times worse if it's still a thing).  NB: the actual LOADING of a save takes about a second.  Most of the delay at start is loading assets.

The current system actually ALREADY backs up your save to a directory within the save folder called "Backup" (ex, KSP_install_dir/saves/MySaveName/Backup), so you should be less boned when and if the file corrupts (I've never had an actual save file corruption, except one time when I was testing and editing orbits and made a data entry mistake).   Also, quicksaves are just regular saves with a different name, so you can restore a main save from a quicksave by copying the quicksave.sfs over persistent.sfs.  NB: the hard modes allow quicksave, but not quickload, so "F5 for safety" (as Das says) is always a good idea, even in hard modes.  That being said, I wouldn't mind better control over that backup dir (more files, more frequently, etc).

One thing that DOES need to be addressed is that the thumbnails aren't stored with a save, but instead clumped together in a folder that ALL saves use, so just copying an entire save folder to a new machine/install doesn't copy the little ship icons.   it's in the base KSP dir, under "thumbs" (KSP_install_dir/thumbs as per my previous example).  It should really be "thumbs" under the SAVE directory.  I'd like to know why it ISN'T...

TL;DR: I say NO to idea, it would be slower.  Splitting files is no replacement for backing up.

On 2016-04-22 at 3:17 AM, mattinoz said:

I wonder if such a move would make multi-player easier. Maybe go further and break master flight file into SOI files. Active physics then only needs deal with writing one the SOI file and the craft files in physics range. Different processes or machines could be handling different SOI co-ordination without dealing with two processes trying to write to the same file.

Craft aren't actually SOI specific, but you could write a quick perl script or similar to actually break the craft in a file down by SOI (it's the ref number in the ORBIT block).  Also you could parse out all ships individually into subdirectories via SOI similarly too.

[swjr-swis]

KSP 1.1 actually keeps a changeable default of 5 backup copies of your (persistent) savefile now. That goes a long way to resolve possible corruptions.

[Snark]

I strongly prefer having it in one file-- makes it much more portable, much less confusing to work with, there's only one file to schlep around.

[mattinoz]

6 minutes ago, Snark said:

I strongly prefer having it in one file-- makes it much more portable, much less confusing to work with, there's only one file to schlep around.

Folders are pretty portable too, better still use a document package and you have a folder that looks just like a file. 

 

[Andem]

Honestly, these designations already exists as subsets within the file. making them each there own file would make editing marginally easier, for a preformance hit. Not a bad idea, but one not suited for ksp at this time.

[Guest]

3 hours ago, Snark said:

I strongly prefer having it in one file-- makes it much more portable, much less confusing to work with, there's only one file to schlep around.

Especially considering that KSP now backs up save files.

[SpaceplaneAddict]

I just want to be able to shove different missions into different saves. Like if I have a very successful flight to Jool in a test save, I could take the mission file and push it into my main save. Paritally supported.

[FancyMouse]

Well, the good thing about splitting is when saving to the same save, you don't need to pay for the I/O each time for the flights you didn't touch, which is the majority of the flights, actually.

But that's about the only thing. Mods may poke parts on inactive crafts so splitting doesn't save anything even on reading (and in fact slows down a lot in this case, as mentioned before).

And what about the case when you load from quicksave and then need to save to persistent? If you wipe out persistent folder first and then save, then splitting doesn't save anything. If you want to do incrementally, then that's a hell of potentially buggy code to make the folder clean and consistent. Neither case is going to give you benefit more than headache.

 

And just to add - subassembly grouping isn't stored per-save or even per-game as well - it's also globally somewhere, which IMO is really bad.

[Renegrade]

37 minutes ago, FancyMouse said:

Well, the good thing about splitting is when saving to the same save, you don't need to pay for the I/O each time for the flights you didn't touch, which is the majority of the flights, actually.

Ah, but in my current save, about 20 of 45 flights have EPHs* from their ORBIT blocks that were updated within 45 seconds of the quicksave.  I haven't analyzed in detail which ship is what, but a quick scan of some of the oldest entries revealed landed debris... so even that isn't a very good saving here

(they might actually be able to fudge it in some cases; in THEORY you could ignore the orbits when saving and such, although you'd have to have an "epoch on load" count or something)

I agree with your other points though.  Especially the one about subassembly grouping.

 

* EPH is the 'epoch' of the orbit, which is the time since last periapsis.  It and the MNA determine where you are in an orbit.  I learned a bit about the ORBIT block when testing the PR

[FancyMouse]

2 hours ago, Renegrade said:

Ah, but in my current save, about 20 of 45 flights have EPHs* from their ORBIT blocks that were updated within 45 seconds of the quicksave.  I haven't analyzed in detail which ship is what, but a quick scan of some of the oldest entries revealed landed debris... so even that isn't a very good saving here

(they might actually be able to fudge it in some cases; in THEORY you could ignore the orbits when saving and such, although you'd have to have an "epoch on load" count or something)

Well, I was focusing on more of "fundamental differences to make a design choice". Those orbit parameters are probably not among them - I can easily squeeze these parameters into the "master file" and still get that particular benefit of splitting, and store the structure of parts in separate files that can be left untouched. This is thus just implementation detail which is not going to change design decisions.

[something]

Yeah something needs to be done about those loading times. They're getting ridiculous:

KSP 1.1 likes to parse all the save files in your folder in order to display information like budget, date, flights reputation etc. Now, if you play career for a certain time your save file gets really big. Like 8+MB. If you have a lot of backups, like I do, then you run into 190+ files having that size. In that case KSP just needs ages to load the loading screen in which you can choose the file to load. That's a no-go in my eyes. KSP 1.0.5 simply listed the file names and everything was fast.

To put that on a scale: With my quadcore 3.5 GHz machine it takes the program approximately 10 seconds to parse 10 savefiles of 8MB each (from hitting "Load Save" to getting the response in terms of a list). Seriously, that's not too much in my eyes and my computing power shouldn't be that outdated...
Multiply that by 19 and you get the time I should have been waiting to see a list. 2 minutes? Really? Instead I assumed, the game froze for some reason and killed it. So, some sort of work needs to be done here. I would be happy with a switch somewhere in the options to get back to the 1.0.5 list.

[nanobot]

A zip file of a bunch of files, individually upated, presumably combines the advantages.