The format of sfs save files

[SalehRam]

Hello,

 

I have asked this in the addons development forum but got no answer...

So I woul like to know the format of the sfs save files, are they following a standard or is it just a format made up by squad?

I need to know this as I am planning to do a small app using .NET that can work with sfs to do specific tasks that I could not find somewhere else...

Thanks a lot!

[Xyphos]

Have you tried opening them in a hex editor and search for patterns?
look on youtube for data hacking tutorials, the stuff's not hard to figure out once you understand that everything has a structure, but the shape of those structures is unknown.

[Snark]

14 minutes ago, Xyphos said:

Have you tried opening them in a hex editor and search for patterns?

Or, indeed, in a text editor, since it's not a binary file. 

2 hours ago, SalehRam said:

are they following a standard or is it just a format made up by squad?

...as far as I know, it's just a format made up by Squad.  Would happily be proven wrong if someone knows better.

That said... it's a pretty straightforward format to reverse-engineer, just look at the text.  It's your basic key = value, named-structure-with-curly-brace-contents format.

[Xyphos]

Just now, Snark said:

Or, indeed, in a text editor, since it's not a binary file. 

oh, I know. I was just trying to learn him a thing or two.

[jwenting]

10 hours ago, Snark said:

Or, indeed, in a text editor, since it's not a binary file. 

...as far as I know, it's just a format made up by Squad.  Would happily be proven wrong if someone knows better.

That said... it's a pretty straightforward format to reverse-engineer, just look at the text.  It's your basic key = value, named-structure-with-curly-brace-contents format.

Which is why I wonder why Squad didn't use XML as that's a pretty well standardized format with parsers available for most programming languages (and indeed provided as standard with a lot of them, including C#).

[Padishar]

9 minutes ago, jwenting said:

Which is why I wonder why Squad didn't use XML as that's a pretty well standardized format with parsers available for most programming languages (and indeed provided as standard with a lot of them, including C#).

This is a very common practice when you don't need all the features of XML.  A well written, simple text parser should be substantially more efficient than any full-blown XML parser.  Also, the extra cruft in XML files tends to make them harder to read...

[cicatrix]

It's JSON... Well, almost. I couldn't make standard JSON libraries work with this, but after certain amount of effort I could successfully parse those files.

Edited May 30, 2016 by cicatrix

[peteletroll]

The APIs to parse the Squad file format are accessible to modders. Probably you can use them from your own program if you link it with the KSP DLL files. Not sure if legal, but I think it's possible.

[SalehRam]

6 hours ago, cicatrix said:

It's JSON... Well, almost. I couldn't make standard JSON libraries work with this, but after certain amount of effort I could successfully parse those files.

Looking at JSON structure (which I am familiar with, but somehow I missed the very similarity with sfs), I think with some few manipulations to text after loading it into my application I can utilize standard JSON libraries to read the text

[SalehRam]

Just for the sake of sharing ideas and knowledge, these are my observations and idea about reading the sfs file as a JSON:

Part of sfs file:

GAME
{
    version = 1.1.2
    Title = DrMath (CAREER)
    Description = No description available.
    linkURL =
    linkCaption =
    Mode = CAREER
    Status = 1
    scene = 5
    editor = None
    flag = Squad/Flags/hexagon
    launchID = 47
    modded = True
    envInfo =  - Environment Info - Win32NT 7FFFFFFFFFFFFFFF  Args: KSP_x64.exe  -
    PARAMETERS
    {
        preset = Custom
        FLIGHT
        {
            CanQuickSave = True
            CanQuickLoad = True
            CanAutoSave = True
            CanUseMap = True
            CanSwitchVesselsNear = True
            CanSwitchVesselsFar = True
            CanTimeWarpHigh = True
            CanTimeWarpLow = True
            CanEVA = True
            CanIVA = True
            CanBoard = True
            CanRestart = True
            CanLeaveToEditor = True
            CanLeaveToTrackingStation = True
            CanLeaveToSpaceCenter = True
            CanLeaveToMainMenu = False
        }
        EDITOR
        {
            CanSave = True
            CanLoad = True
            CanStartNew = True
            CanLaunch = True
            CanLeaveToSpaceCenter = True
            CanLeaveToMainMenu = False
            startUpMode = 0
            craftFileToLoad =
        }
    }
}

And an example of a JSON structure:

{"widget": {
    "debug": "on",
    "window": {
        "title": "Sample Konfabulator Widget",
        "name": "main_window",
        "width": 500,
        "height": 500
    },
    "image": { 
        "src": "Images/Sun.png",
        "name": "sun1",
        "hOffset": 250,
        "vOffset": 250,
        "alignment": "center"
    },
    "text": {
        "data": "Click Here",
        "size": 36,
        "style": "bold",
        "name": "text1",
        "hOffset": 250,
        "vOffset": 100,
        "alignment": "center",
        "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
    }
}}

So, I can load the sfs into memory (it is going to eat the hell out of the memory), then implement a function that will modify the text to match the JSON structure exactly, such as replacing the equal signs, adding quotation marks and value commas, etc... and then the remaining is easy for a JSON library...

This is one approach, which I both like and dislike, as it will add some complexity and make some double work...

Other approach, which I kinda prefer, is to write a function that will read the whole sfs line by line and determine the key and the value, then outputs it in an XML format...

However, that will require more work but the result will be much more rewarding from the previous approach for me...

Edited May 30, 2016 by SalehRam

[Xyphos]

there are similarities to JSON, but I think YAML would have been a better choice.

 

[cicatrix]

Instead of loading the whole file in memory you can make SFSReader class which would read it element by element. I wish I had more time to make something presentable out of my C# code that messes with these files.

[steuben]

You guys are all over thinking this. I've got a little Excel file that will strip struts out of a craft file without using scripts.

As much as I think that learning how to translate stuff into the various standard formats is a good thing... I'm going to ask "why". It feels like much of the leverage that you would get from translation will be lost. Mostly because in doing the translation you have the data in hand. So why process it to <insert format here>, do whatever you want to it, process it out of <insert format here>?

 

[Padishar]

2 hours ago, steuben said:

You guys are all over thinking this. I've got a little Excel file that will strip struts out of a craft file without using scripts.

As much as I think that learning how to translate stuff into the various standard formats is a good thing... I'm going to ask "why". It feels like much of the leverage that you would get from translation will be lost. Mostly because in doing the translation you have the data in hand. So why process it to <insert format here>, do whatever you want to it, process it out of <insert format here>?

While I basically agree with your second paragraph that there is no point converting the format to XML so you can use an XML library to read it, (the conversion process, by definition, must be decoding the data out of the format in order to generate useful XML so just use the data directly) the first paragraph isn't particularly related.  It is significantly harder to write a program that can remove parts from vessels in flight as the linkages between the parts are stored differently in the .sfs file than in a .craft file.  In a craft file you can basically just strip out whole parts as you feel like but, if you do the same in a save file then it is highly unlikely to load because references to other parts are specified as indices into the part array of the vessel and removing parts will change the index of every following part.  For this reason, when messing with vessels in save files, you really need to load the data into an in-memory structure that resembles the actual layout of the file, so that, when you write it out again, you can fix up the part references...

[cicatrix]

3 hours ago, steuben said:

You guys are all over thinking this. I've got a little Excel file that will strip struts out of a craft file without using scripts.

As much as I think that learning how to translate stuff into the various standard formats is a good thing... I'm going to ask "why". It feels like much of the leverage that you would get from translation will be lost. Mostly because in doing the translation you have the data in hand. So why process it to <insert format here>, do whatever you want to it, process it out of <insert format here>?

 

Because if you do and make it open source other people might do something really useful with your library.

[SalehRam]

First, I would go through the steps to do this from scratch is because 1. I enjoy it, 2. I tend to go to the "do it yourself" approach when I really and badly need a thing and not find it, at one point, I really needed to work with SFS files quickly and easily, and while I appreciate the tools that are available out there, my own criteria was still not satisfied...

 

So, the idea I will work on is:

Read the SFS file line by line, map every line in the file, then manipulate the whole file and convert it into something compatible with standard markdown formats (be it JSON, XML or whatever, though JSON for me seems easier). and link every line of the new file to the matching line of the original file

When I finish working with the file, I simply merge the values of file with the original one based on the map and links I created earlier in the process...

Because of the nature of this, I will need to use a tool (programming language) that I feel very comfortable with, and that is VB.NET as I know others, but I have been with VB/VB.NET since 1996. (as a hobby)

I'll put the project on github later for contributions

Edited June 2, 2016 by SalehRam

[Curveball Anders]

On 2016-05-30 at 10:39 AM, jwenting said:

Which is why I wonder why Squad didn't use XML as that's a pretty well standardized format with parsers available for most programming languages (and indeed provided as standard with a lot of them, including C#).

"Everything looks like a nail for someone used to a hammer" -- Unknown Ancient Sage.

Any coder's (or other person involved in creating software) choice of tool or tool chain often comes down to known and/or previously used stuff and not always (or even ever) the tool or tool chain that would be most suitable based on perfect knowledge and 20/20 hindsight

It's also a known fact that when a team member returns from a demo/training/briefing on a tool, she/he/it will insist on converting all possible (and some impossible) code to this new and revolutionary concept ...

Squad's (most likely HarvesteR's) choice of format was simply the one that was most obvious right there and then

 

[SalehRam]

Got something going... Will get it on Github once I get home, it is C# though (I thought why not get out of the box anyways and do something new all together).

Basically, I take the file, read it line by line, index the nodes, and their sub-nodes.

Using 5-d array, a treenode object to build the structure.

I thought about using dataset, xml file, dictionaries, but the thing is, I could think about it easily done with a 5-d array like:

[original line number in sfs file, tab spaces (/t), property name, property value, the index of the property in treenode]

The brackets "{,}" are going to be ignored, and the node headers are not going to be included in the array as well.

I'm now at the point of building the treenode structure, and once that is done, everything is done.

 

Once that is completed, searching, through the array will be possible and I believe I can do anything with the content that I will get at the end.

 

As for the memory usage, so far it goes up to 30mb, implementing the array will bump it another 60 or 90 mb, so I am talking about a total of 150mb

Edited June 9, 2016 by SalehRam

[steuben]

I'm curious about it... how big is the sfs you're feeding into it?

[SalehRam]

2 hours ago, steuben said:

I'm curious about it... how big is the sfs you're feeding into it?

About 6.5mb, it has about 313,000 lines as well

I will include it in the github repository once I upload it

[cantab]

Yeah, it's a fairly straightforward file format. I don't know if it's any standard or something Squad cooked up. There is already a program capable of reading and parsing the .sfs format - KSP itself!

It's nothing like XML. It's not JSON, though at first glance it looks close. Practically, probably the biggest deal is that .sfs is a line-by-line format, whereas JSON uses punctuation. It's not INI, YAML, or libconfig either.

It is I think the same basic format as the .cfg files for parts, and the settings.cfg file. And the format is inherited and extended by Module Manager.

I believe the KSP Trajectory Optimization Tool also reads .sfs files , though it only cares about a small portion of them.

[Ippo]

In case anyone needs it, a long time ago I had written a small parser for ConfigNode objects. It has never been 100% complete or tested, and I haven't checked if it works in ages, but it should be a good start I hope.

https://github.com/Ippo343/OpenNodeParser

License is public domain (aka "do what you want with it").

P.S: oh, and it's neither JSON nor YAML. It's just a Squad proprietary standard.

Edited June 9, 2016 by Ippo

[SalehRam]

https://github.com/salehram/SFSReaderClass

That's my mess... Might look awkward the way I did this, but I was just going through with my thoughts...

The idea is you will get 2 references, one is visualized in the node tree, the other one is a sort of dictionary for the whole file. When an item from the tree is selected, there is an index for it, that index then will be matched from the other reference which also contain the line number of each line of the SFS file, that way, direct search and modifications are possible. a complete file rewrite can be done...

There are few things that I need to complete yet, but this is about 90% of the work, and that is my whole POC project about this... I did it because it was fun - and to break the wall with C#

Edit:

I took the approach of looping through the file and looking for nodes start and ends rather than regex because I built all of this on the assumption that I know nothing about the format of the file and what standard if follows. I know that node titles are capitalized, and I know the main node titles (GAME, PARAMETERS, CAREER, VESSEL, etc...), but that was not enough for me... it is not like having a full documented standard like JSON, YAML or others... so I approached this as a completely unknown format and dealt with it based on that

PS: Can I ask for this to be moved to the development forum now that it has a content relevant to there?

Edited June 10, 2016 by SalehRam