[answered] Documentation on what can and can't go in a string value of ConfigNode?

[Dunbaratu]

I'm having a heck of a hard time diagnosing a problem with Base64 encoded content stored inside a ConfigNode of the persistent.sfs file getting mangled. It would help if I could find actual hard facts about the syntax of the persistent.sfs file and what is and isn't allowed in it. For example, when the file has a line like so:

name = someValue

When what is the regular expression that defines what is legal and illegal for someValue?

I'm getting the characters '='. '_', and ',' being stripped off of someValue if they happen to occur at the end of the string, but not if they occur in the middle of the string.

Edited December 15, 2014 by Steven Mading
added '[answered]' prefix to subject

[jamespicone]

As far as I can tell, ConfigNode parsing tokenizes on newline, followed by removing any token that starts with "//" and removing parts of tokens following "//", then it tokenises on "{" and "}", then it splits tokens into two around the first "=" character, and finally it trims whitespace on either side of the token. I don't think it strips "_" or ",". How are you actually getting the ConfigNode?

[Dunbaratu]

As far as I can tell, ConfigNode parsing tokenizes on newline, followed by removing any token that starts with "//" and removing parts of tokens following "//", then it tokenises on "{" and "}", then it splits tokens into two around the first "=" character, and finally it trims whitespace on either side of the token. I don't think it strips "_" or ",". How are you actually getting the ConfigNode?

This is the sequence of events I'm getting:

1: I leave the scene of the ship, and go back to the space center. Knowing this will write persistent.sfs, I pause the game and view the persitence file in a text editor, and confirm that my value is in it, and ends with "__".

2: I quit the editor (not saving anything) and then in the game, exit the game nicely.

3: I look at the persistence file again, now the value is missing the trailing "__".

The *FIRST* time it writes it out, it writes it correctly. Then after a complete quit of the game it's gone again.

[Diazo]

This intrigued me and I did some reading. I did not run any tests or anything but I found the Base64 wiki article that was an interesting read. This page specifically.

Notably is the fact that base64 uses = symbols as padding at the end of the string if it needs to add characters to line up. I can easily see this messing KSP up with having a second = sign on the value line.

valueName = valueStrint==

That is a valid base64 string and I could see the fact that it is a second equals sign messing things up. I could also see the fact that it is an equals sign with nothing after it and I could also see the fact it is two equals signs in a row being the issue.

One workaround that I did come across was rather then letting the Base64 do the padding itself, add the padding yourself before calling Base64 and then stripping the padding out yourself after converting back from base64.

Having said that, it seems not all Base64 implementations use the = for padding. If you stick a print() into your save code and dump the base64 string to the output_log.txt, does it match what you see in the persistent.sfs? And is there padding being added?

D.

edit: Seeing philotical's post below, I did something similar for ship names in AGext to ensure all ship names would be acceptable to save, even if the player used funky symbols.

foreach (Char ch in name)
            {
                hashedName = hashedName + (int)ch;
            }

creates a string of numbers that represent the ship name. I never had an issue saving hashedName as it was guaranteed to only contain the numbers 0 through 9.

Note that I simply used hashedName as the ship identifer, I never converted it back so error check for things such as character 1 that might save as "1" instead of "01" is not present.

D.

Edited December 14, 2014 by Diazo

[philotical]

I've had similiar problems and skipped base64 for KSP..

I use binary now..

        // found here http://www.fluxbytes.com/csharp/convert-string-to-binary-and-binary-to-string-in-c/
        private string StringToBinary(string data)
        {
            UnityEngine.Debug.Log("### LCARS_Message_Type StringToBinary data=" + data);
            //return data;
            StringBuilder sb = new StringBuilder();
            foreach (char c in data.ToCharArray())
            {
                sb.Append(Convert.ToString(c, 2).PadLeft(8, '0'));
            }
            return sb.ToString();
        }
        private string BinaryToString(string data)
        {
            UnityEngine.Debug.Log("### LCARS_Message_Type BinaryToString data=" + data);
            //return data;
            List<Byte> byteList = new List<Byte>();
            for (int i = 0; i < data.Length; i += 8)
            {
                byteList.Add(Convert.ToByte(data.Substring(i, 8), 2));
            }
            return Encoding.ASCII.GetString(byteList.ToArray());
        }

so far it works well..

hth

[xEvilReeperx]

It looks like ConfigNode is capable of handling both cases:

[KSPAddon(KSPAddon.Startup.Instantly, true)]
class Base64Test : MonoBehaviour
{
    class TestSave
    {
        [Persistent]
        public string Base64Underscore = "someValue1__";
        [Persistent]
        public string Base64EqualsEquals = "someValue2==";

        [Persistent] public string RandomGarbage = "__==__";

    }

    void Start()
    {
        var testSave = new TestSave();

        var test = ConfigNode.CreateConfigFromObject(testSave);
        print("To configNode: " + test);

        var loadTest = new TestSave {Base64EqualsEquals = "", Base64Underscore = ""};
        ConfigNode.LoadObjectFromConfig(loadTest, test);

        // check to see maybe if it's getting stripped on file load instead
        test.Save(KSPUtil.ApplicationRootPath + "/testBase64.cfg");
        var fromDisk = ConfigNode.Load(KSPUtil.ApplicationRootPath + "/testBase64.cfg");
        print("from disk: " + fromDisk);

        loadTest.GetType().GetFields().ToList().ForEach(fi => print(string.Format("{0} = {1}", fi.Name, fi.GetValue(loadTest))));
    }
}

[LOG 11:50:52.424] To configNode: DebugTools.Base64Test+TestSave
{
	Base64Underscore = someValue1__
	Base64EqualsEquals = someValue2==
	RandomGarbage = __==__
}

[LOG 11:50:52.430] from disk: root
{
	Base64Underscore = someValue1__
	Base64EqualsEquals = someValue2==
	RandomGarbage = __==__
}

[LOG 11:50:52.432] Base64Underscore = someValue1__
[LOG 11:50:52.432] Base64EqualsEquals = someValue2==
[LOG 11:50:52.433] RandomGarbage = __==__

Since you mentioned persistent file, I tried a ScenarioModule as well:

[KSPScenario(ScenarioCreationOptions.AddToAllGames, GameScenes.SPACECENTER, GameScenes.FLIGHT)]
class Base64Scenario : ScenarioModule
{
    [KSPField (isPersistant = true)] public string ControlValue = "TestValue";
    [KSPField(isPersistant = true)]
    public string EqualsEquals = "TestValue==";
    [KSPField(isPersistant = true)]
    public string UnderscoreUnderscore = "TestValue__";
    [KSPField(isPersistant = true)]
    public string TestComma = "test,";
    [KSPField(isPersistant = true)]
    public string TestPeriod = "test...";


    public override void OnLoad(ConfigNode node)
    {
        print("Loaded from: " + node);
    }

    public override void OnSave(ConfigNode node)
    {
        // and manual values too, just in case
        node.AddValue("eqeq", "manual__");
        node.AddValue("com", "check for comma,,,");
        node.AddValue("period", "This is a sentence.");

        print("Saved into: " + node);
    }
}

	SCENARIO
	{
		name = Base64Scenario
		scene = 5
		ControlValue = TestValue
		EqualsEquals = TestValue==
		UnderscoreUnderscore = TestValue__
		TestComma = test,
		TestPeriod = test...
		eqeq = manual__
		com = check for comma,,,
		period = This is a sentence.
	}

I couldn't reproduce your problem. What does your save/load code look like?

[Dunbaratu]

I'm aware of the '==' at the end of base64 already. I already tried replacing the '==' at the end with '__' or ',,' and it *still* strips them off at some point during the save.

Now I'm suspecting that I'll have to dig through the code to find somewhere *else* that the saving is also occurring. That's the problem with picking up mods someone else wrote and continuing with them. Things often occur in places in the code that make no sense and so you don't catch them. If this problem isn't universal then that has to mean the code is trying to manipulate the same data in the save file from *more* than one location in the code, because I verified that it works right in the one place I edited it. There must be some other location where it's repeating the same work, in a very not-object-oriented way.

Edited December 14, 2014 by Steven Mading

[Dunbaratu]

yeah definitely it's NOT a problem with what the low level persistence file is doing.

I was able to make a dumb test to do this:

node.AddValue("testaa","aaaaa==");

node.AddValue("testbb","bbbbb__");

node.AddValue("testcc","ccccc,,");

And then later, this:

UnityEngine.Debug.Log("reading back in: testaa="+configNode.GetValue("testaa"));

UnityEngine.Debug.Log("reading back in: testbb="+configNode.GetValue("testbb"));

UnityEngine.Debug.Log("reading back in: testcc="+configNode.GetValue("testcc"));

And that proved that the file is perfectly capable of storing values with "__", "==", or ",," at the end of them without stripping them off.

So whatever is happening in the code that's causing them to be stripped off, it's happening elsewhere, which is utterly confusing to me. Oh well, I'll have to keep looking.

[Dunbaratu]

Actual Problem Found. Oh god this whole thing was one massive red herring.

The actual Problem: The slash ('/') is a valid Base64 character. It represents the bit pattern 111111 (dec 63). It's therefore possible if you have a lot of bits of '1' in a row, to get two contiguous slashes in the base64 string. Two contiguous slashes forms a comment ('//') in the persistence file and thus causes the value to be truncated at that point.

It always got truncated when the content had '=' in it and was not truncated when it did, purely because of the sheer coincidence that in my test cases when the content had '=' in it, it also happened to have double-slashes ('//') in it. There's no causal relationship between those two conditions - it was just a random fluke of the data I was testing with.

[ratchet freak]

you should be able to choose the 2 non alpha numberic chars used in base 64, - and _ are typically used in URI compatible outputs instead of + and /