[RESOLVED] Serialization of data between a prefab part and live parts (see OP for solution)

[Shadowmage]

Hi all,

Running into a bit of a problem regarding basic serialization of part data.

I have partmodule with a field that I need to be serialized from the prefab part into any live/actual parts. The field is a custom class with the [serializable] attribute. It is populated int he prefab during the prefabs OnLoad(ConfigNode ...) method, and definitely not null on the prefab part (can print the reference/hashcode/etc).

However, the custom field is not serialized into the live parts -- the reference/field is always null. I have tried using the [serializeField] attribute, but it does not seem to make any difference. I have tried a (shortened, all KSP references removed) test-case of the same custom class in a basic Unity-editor test; the custom class/field serialized properly and was displayed in the editor as editable component fields. So... the basic class appears to be serializable and serialize just fine outside of KSP.

No errors are thrown or logged.

Trimmed source code for the part class

public class SSTUConverter : PartModule
{
        //converter recipe field -- SHOULD be serialized from the prefab into any sub-instances; but is not....
        //does not work with or without the serializeField attribute
        [SerializeField]
        public ConverterRecipe recipe;  

         public override void OnLoad(ConfigNode node)
        {
            if(node.HasNode("CONVERTERRECIPE"))//load the recipe config for the prefab part instance;
            {
                loadRecipeFromNode(node);
            }
        }

        private void loadRecipeFromNode(ConfigNode node)
        {
            if (node.HasNode ("CONVERTERRECIPE"))
            {
                recipe = new ConverterRecipe ().loadFromNode(node.GetNode ("CONVERTERRECIPE"));
            }
            else
            {
                print ("ERROR - no recipe node to load converter recipe from!");
            }
            print ("Constructed recipe: "+recipe);//recipe prints successfully when loaded on prefab, is definitely not null at this point in the prefab
        }
}

Trimmed source code for the custom class:

using System;
using System.Collections.Generic;
using System.Linq;
using UnityEngine;

namespace SSTUTools
{
    [Serializable]
    public class ConverterRecipe
    {
        [SerializeField]
        private List<ConverterResourceEntry> inputs = new List<ConverterResourceEntry>();
        //ConverterResourceEntry is another serializable class ;
        [SerializeField]
        private List<ConverterResourceEntry> outputs = new List<ConverterResourceEntry>();      
    }    
}

What I'm trying to accomplish is persist some complex configuration data between the prefab and the live parts that are defined in sub-nodes within the modules config node in the part.cfg file.

E.G

MODULE

{

...stuff...

CUSTOMNODE

{

...custom complex data...

}

}

However, this config data is only available to the prefab part as it is first compiled - So I have been trying to find a way to persist the data into the cloned parts.

For some of my modules I have just given them a 'moduleID' (in case multiple of the same type are present in a single part) and then reload the data from the part.partInfo.configNode data by manually finding the proper MODULE reference in that data. However, it seems like a bit of a hack and workaround to do that for something as simple as a list of resources (the other places I've used it were much more complex data with object references that would not serialize properly even if I wanted it to).

So, I guess the question is -- does standard Unity Serialization work for fields in PartModules? (I would think it should... some stock modules appear to use it)

If yes, are there any known issues? Anything obviously wrong with the above posted code? I will be attempting some very trimmed down versions in KSP later to see if it is something deeper in the class preventing it from serializing (as the basic one works in Unity... it _should_ then work in KSP, unless the entire serialization system is broken).

Alternatively - what other ways are there to get complex data to be cloned properly into live parts (rather the partModules therein)?

Thanks in advance; completely baffled by this one at the moment, lost an entire nights worth of work-time trying to figure this one out...with absolutely no progress.

Edit:

The answer:

Standard Unity serialization using the [serializable] attribute does not work for mod/plugin supplied standard classes (though it does work for any stock supplied classes using said attribute). All primitives (int/float/double/string/etc) and KSP-supplied classes will still serialize properly.

Workarounds include:

1.) Custom serialization routines that serialize to/from strings or other primitives; store your data as primitives for serialization and deserialize manually on the other side.

2.) Unity's ScriptableObject appears to still work properly for serialization of custom data classes. Instead of using the [serializable] attribute, make the class extend from ScriptableObject. Beware, there are some reference differences here - Scriptable objects are serialized as a reference and not a data instance; so duplication from a prefab part will result in the cloned part having a -reference- to the prefabs serialized object (need to verify this behavior; it is the documented behaviour.... but the documentation also states that standard serialization should work... so you can see how reliable the docs are when it comes to KSP modding).

Edited August 14, 2015 by Shadowmage

[sarbian]

Did you have a look at http://docs.unity3d.com/Manual/script-Serialization.html ? Serialization in unity is tricky and in SmokeScreen I had to do a lot of thing by hand to bypass an old Unity limitation (I should rewrite that now...)

[Shadowmage]

I would have to agree that serialization is tricky; some odd stuff going on for some of it.

Aye, I've read through the docs. Everything appears to be proper for it to be serialized according to their documentation. And indeed it does serialize when I use it in the Unity editor.

Field is marked public in the PartModule, check.

Custom class needs Serializable attribute, check.

Custom class is not abstract, is a regular base class. check.

Custom class has a public no-param constructor (not sure if this is technically needed, but generally is for classes instantiated through generic API-driven reflection).

I'll take a look at your SmokeScreen source to see if I can draw any inspiration as to alternative methods. I'm also going to try some more contrived/trimmed down test cases to see if I can get a simpler object to serialize properly. It is entirely possible something in the class is not allowing it to serialize, something that was trimmed out in my unity-editor test cases.

I already have some work-arounds available to implement, but was hoping I could get the default/built-in serialization working; results in slightly cleaner code. Although, thinking about it, its not really cleaner code if I have to put a bunch of attribute tags on everything ([serializable][serializeField]). Might be better in the long run for me to implement a robust 'load from stored prefab config' system rather than trying to get the serialization working. As long as a PartModules index in the part.Modules list is stable I can probably work up something reliable and easier to use than worrying about the serialization stuff.

Thanks for the response. I'll keep poking around and testing things.

[Diazo]

As long as a PartModules index in the part.Modules list is stable I can probably work up something reliable and easier to use than worrying about the serialization stuff.

It is not.

Anytime mods, and their associated partModules, are added or removed from a KSP installation, the partModule index changes.

Now, if you mean within the same game scene, yes, the partModules index is fixed. Just don't expect it to stay the same across game save/loads.

I had a big fight with this in getting AGX to work with custom parts, especially those with multiple copies of the same part module. (Science parts with multiple experiments, I'm looking at you.)

99% of modules on have a single copy on a part though, so partModule.moduleName works as a unique identifier almost all of the time.

I can give you hand with suggestions on working with partModules when I get home after work to my KSP computer, but on the specific SerilizeField question, I can't help as I've never worked with it.

Because of that, for loading complex data I would either:

1) Load my own ConfigNode containing configurations, then give each part a KSPField with a ConfigID that loads from the ConfigNode at runtime.

2) Do a workaround of loading a KSPFIeld string as a CSV list, but if the data is complex this method is probably too clunky.

D.

Edited August 13, 2015 by Diazo

[Shadowmage]

It is not.

Anytime mods, and their associated partModules, are added or removed from a KSP installation, the partModule index changes.

Now, if you mean within the same game scene, yes, the partModules index is fixed. Just don't expect it to stay the same across game save/loads.

I had a big fight with this in getting AGX to work with custom parts, especially those with multiple copies of the same part module. (Science parts with multiple experiments, I'm looking at you.)

99% of modules on have a single copy on a part though, so partModule.moduleName works as a unique identifier almost all of the time.

I can give you hand with suggestions on working with partModules when I get home after work to my KSP computer, but on the specific SerilizeField question, I can't help as I've never worked with it.

Because of that, for loading complex data I would either:

1) Load my own ConfigNode containing configurations, then give each part a KSPField with a ConfigID that loads from the ConfigNode at runtime.

2) Do a workaround of loading a KSPFIeld string as a CSV list, but if the data is complex this method is probably too clunky.

D.

Thanks Diazo.

Currently I'm doing something similar to #1 for a few other modules. It seems to work well. A [KSPField] int that I can use to identify which particular module it was from the raw config data modules list (mostly (only?) needed for multiple modules of the same type in one part). I then go into the prefab config (part.partInfo.partConfig) and find the specific MODULE node by both module name and moduleID field. I was just hoping to clean things up a bit (and not need the extra moduleID/configID field).

I've also done #2 a bit, but have found the CSV lists for more complex data to be a bit cumbersome to write/manage properly (too easy to make lists of different sizes in the config file, or get values out of order and not even realize it). I currently use this method for simple one-off lists of data (like a list of transform names) that do not need ordering,

Hmm.. it seems the #1 option you are proposing is a bit different than I'm currently using. If I'm reading this correctly, you are proposing storing the complex config data for the modules in a completely separate and external .cfg file, and loading the data for each module from that external file, referencing the specific config by the moduleID/configID? Seems like it would work about like I'm doing; though I would really like to keep the config for the module in the parts .cfg file. Will have to give this method a bit of thought.

Good to know about the module index. I had assumed that they would stay the same at runtime (after MM patches are applied at least) between the prefab and any live parts, but have not yet done any significant testing. How would this effect things like ModuleSurfaceFX that need an engine module index supplied in the .cfg file? Mainly I guess the question on this would be - does the order (and indexes) of modules in a part stay the same between the prefab and live versions of that part - at runtime? I suppose I'll have to do some experimentation to find out what/when/if things get changed.

I would think they should be fairly consistent due to how parts are deserialized/reloaded from the persistence file - if modules are out of place between the prefab and persistence it complains about modules not being at the proper index (and you generally only see this if mods or MM patches are added/removed from an already in-progress game). I suppose that alone is enough to discount using the raw partModule index method, as I know that someone using my mod (like me!) would change their mod set after a game was started; and it could/would all fall apart at that point -- I would have to implement the same 'look around more if you don't find it in the proper index' stuff that stock does, which is more unreliable mess that I don't want to have to maintain .

I'll probably just go with my implementation of #1 again for this module if I cannot figure out the serialization (or other cleaner alternative) in a reasonable time. Sometimes the amount of -work- I do in order to be lazy is... totally counterproductive to the laziness

Thanks again for the info and giving me stuff to think on

[Diazo]

While the module index will generally stay the same at run time after module manager has processed, it is not guaranteed and I know there are mods out there that use the .AddModule() method, and there exists the .RemoveModule() method, for adding and removing partModules while the game is running.

Are you probably okay with assuming the partModule index stays the same during the same game session? Yes, just be aware it is not guaranteed.

On the configNode issue, I do want to clarify that there are two methods of doing this.

The first is

ConfigNode myConfigs = ConfigNode.Load("FileOnDisk");

I think this is the one you talk about, you directly load a file into a config node.

However, you can also use the GameDatabase to indirectly load a config node:

ConfigNode myConfigs = GameDatabase.Instance.GetConfigNode("Diazo/MyMod/MyConfigs");

Then make a .txt file in KSP\GameData\Diazo\MyMod\MyConfigs.cfg as follows:

name = MyConfigs
value1 = HelloWorld
NODENAME
{
value2 = ThisIsASubNode
}

The advantage to doing it this way is that Module Manager can patch this node, so in your part.cfg file, add this at the end:

@[MyConfigs] //no filter needed, there is only one MyConfigs configNode in the database
//standard Module Manager syntax now applies

Note I'm not 100% sure you can put module manager patches in the part.cfg file like that, but it is something to test.

And even with either of these methods, you still need to load a reference ID in the part.cfg so your code can pull the correct config from the MyConfigs node at runtime.

If you wanted to avoid having to KSPField a ConfigID, you could in theory load the MyConfigs ConfigNode and using the partName unique identifier for the sub nods, in your partModule you could simply:

ConfigNode myConfigs = GameDatabase.Instance.GetConfigNode("Diazo/MyMod/MyConfigs").GetNode(partName);

and then that instance of the myConfigs configNode would have the correct default configuration for that part.

This assumes that each part has a unique config though, if multiple parts share the same config that doesn't work and you have to use the ConfigID anyway. If you adding a unique config to each part.cfg file though, it sounds like you could do this.

D.

Edited August 13, 2015 by Diazo

[Shadowmage]

Updated info regarding serialization:

Strange stuff. Very strange.

A public string serializes fine. A public List of strings serialize fine. Likely all primitives will (testing...). -- this is without the [KSPField] or [serializeField] attributes. I can set the value in the OnLoad() method of the prefab, and it will properly carry across to new parts pulled out of the editor.

But I cannot get even the simplest of test-cases of custom classes to serialize, no matter what attributes I flag them with (the fields or the custom classes). Neither single instances or lists seem to work.

Here is the strange part: stock Propellant class gets serialized fine as a public field with no attributes(I just chose a stock class that had the [serializable] attribute). I can populate a field in the OnLoad method when the prefab is created, and it will be properly populated on new cloned parts in the editor (as OnLoad() is not called on -new- parts in the editor, I can be reasonably certain that it is in fact being serialized properly from the prefab).

My first suspicion is that the loading path for mod-dll files is different than the stock Unity loading used by the KSP .dll files, and somehow custom (non-KSP supplied classes) are not getting registered with Unity as being serializable as they are loaded through the alternate process. I'm really not familliar with the Unity Engine internals or any of the C# class-loading mechanisms though, so it is all just speculation at this point.

Will keep playing with it a bit... at least I've gotten -something- to serialize and survive the prefab->cloned part transition. It is a starting point at least. Would like to get a handle on this and find out what is going on (and hopefully make use of it!).

Edit: More strangeness, and semi-confirmation for my theory above. A custom class that directly subclasses from Propellant does -not- serialize properly (with or without the [serializable] attribute). It seems likely that mod/plugin code is limited to serialization of only primitives and stock classes, as mod/plugin classes are somehow not being recognized as serializable. This would also explain why my first test cases would serialize properly in the Unity editor (as a field in a basic MonoBehaviour attached to an empty gameObject), but would not work when attempted within KSP.

Bit more testing to do, I need to confirm if the [serializeField] attribute works properly for private fields in mod/plugin classes.

Further edit: [serializeField] -does- work for private fields for supported classes (primitives and KSP supplied classes).

Conclusion: ...umm... at least I can serialize Strings (and primitives)? Can actually do nearly anything with that capability with regards to manual (de)serialization/synchronization.

I'll probably whip up a quick string->ConfigNode parser (as stock methods only parse from files?) and store my needed ConfigNodes in string format for the (manual) serialization, re-parsing them into ConfigNodes in the cloned parts OnStart() method (manual deserialization). I'm chosing ConfigNode as the interface/storage class, as the rest of my code already deals with loading/reading from ConfigNodes, and it is easier in the long run than writing custom serialization/deserialization code for every class -in addition- to the already needed ConfigNode parsing for stock data-handling.

Further edit: - just found the stock ConfigNode.Parse method... if all works out, looks like I should at least have a workaround if I cannot otherwise find solutions to serialization of custom classes.

Edited August 14, 2015 by Shadowmage

[xEvilReeperx]

My first suspicion is that the loading path for mod-dll files is different than the stock Unity loading used by the KSP .dll files, and somehow custom (non-KSP supplied classes) are not getting registered with Unity as being serializable as they are loaded through the alternate process. I'm really not familliar with the Unity Engine internals or any of the C# class-loading mechanisms though, so it is all just speculation at this point.

I thought something similar. Stuff that works in the editor suddenly doesn't work in KSP. You can use Unity's shared reference serialization object, though:

public class SSTUConverter : PartModule
{
    [Persistent] public ConverterRecipe recipe = ScriptableObject.CreateInstance<ConverterRecipe>();

    public override void OnStart(StartState state)
    {
        base.OnStart(state);
        print("SSTUConverter.OnStart");

        if (recipe != null)
            print("Recipe: " + recipe);
        else Debug.LogError("(No recipe)");

    }

    public override void OnLoad(ConfigNode node)
    {
        base.OnLoad(node);

        if (node.HasNode("CONVERTERRECIPE"))
            ConfigNode.LoadObjectFromConfig(recipe, node.GetNode("CONVERTERRECIPE"));
    }
}


public class ConverterRecipe : ScriptableObject
{
    [Persistent]
    private List<ConverterResourceEntry> inputs = new List<ConverterResourceEntry>();

    [Persistent]
    private List<ConverterResourceEntry> outputs = new List<ConverterResourceEntry>();


    public override string ToString()
    {
        return string.Format("Recipe: using {0}, you get {1}", string.Join(",", inputs.Select(i => i.Resource).ToArray()),
            string.Join(",", outputs.Select(i => i.Resource).ToArray()));
    }
}


public class ConverterResourceEntry
{
    [Persistent]
    public string Resource = "default value";
}

It looks a bit ugly because I took advantage of KSP's existing ConfigNode serialization to construct the recipe. The PartModule ConfigNode the above uses looks like this:

MODULE
{
	name = SSTUConverter

	CONVERTERRECIPE
	{
		inputs
		{
			item
			{
				Resource = Hydrogen
			}
			item
			{
				Resource = Oxygen
			}
		}
		outputs
		{
			item
			{
				Resource = Water
			}
		}
	}
}

Edited August 14, 2015 by xEvilReeperx
removed unnecessary SerializeField attributes

[Shadowmage]

Very interesting. I would suppose the serialization works for subclasses of ScritableObject because Unity checks the class/type at runtime (serialization time) rather than looking for an attribute at load-time (although one would think they would just check for the attribute at run-time as well).

Looks like that might be an acceptable alternate solution. Not too much different than using the [serializable] attribute on a class for my particular use cases. Thanks for the additional info xEvilReeperx (and examples). Now I've got options!

I went into this particular project thinking "Ahh, a simple resource converter/generator/alternator module... that should only take like an hour to write up!". Took a bit longer than an hour to write up the core code, but not much. And then spent 2 1/2 nights trying to figure out the serialization for the recipe. Finally got to give it the first real test last night though. Everything seems to work well so far (using the config node->string->config node serialization path). Ever get the feeling that you wanted ((poorly/un)documented/inconsistent) APIs to GTFO of your way and let you get down to programming? (if only it worked that way....)

Going to mark this thread as resolved, as multiple acceptable alternate solutions have been found (and also much general info regarding general Unity serialization along the way).

Thanks for your time and help guys,