Cannot figure out how to save, ScenarioModule keeps resetting!

[TheBossk]

Hi!  This is my first post to this forum, so I hope everything is alright.  I've been searching here and on Google for a week and hit an enormous amount of dead ends.

As the title says, I've hit a wall with the KSPScenario system, trying to save/load data from my mod.  I have my save/load methods properly overridden in a ScenarioModule tagged with the KSPScenario attribute.  I have a static Dictionary in another class with the data I want to save.  Beyond there is where the problems begin.

My custom PartModule has to do some math upon startup to deduct resources.  Initially I figured, great, I'll do it in OnStart.  However, the Scenario Module always runs OnLoad after the PartModule's OnStart, loading data into memory from BEFORE the update.  Here is the current flow:

-- Flight Scene is loaded --

PartModule.OnStart(..)
  Check tracking info, which is already in memory, in State 1
  Update tracking to State 2

ScenarioModule.OnLoad(..)
  Loads tracking info still in State 1 - correct data is overridden!

ScenarioModule.OnSave(..)

PartModule.FixedUpdate(..)
  Sometimes starts as early as before ScenarioModule.OnLoad(..)

PartModule.OnStart takes the initial values (State 1), updates them (State 2), and then returns.  When ScenarioModule.OnLoad is called, it overwrites the data, putting it back in State 1.  Ideally OnLoad would only trigger when loading from a persistence file, but I haven't found a way to control for that.

I have tried an enormous amount of workarounds, including:

1. Manually invoke ScenarioModule.OnSave(..) at the end of OnStart with a modified, correct ConfigNode.  OnLoad still loaded the stale data.
2. Move OnStart logic to OnInitialize.  This worked when loading a vessel already in flight, although OnInitialize started an indeterminate time after FixedUpdate began iterating.  The problem arose when creating a new vessel (i.e. vessel rollout).  There, OnInitialize was triggered so EARLY that the PartModule's reference to vessel was null.
3. Try manually writing/loading to persistent.sfs at the end of OnStart, similar to item #1.  Made me uneasy, but no matter - it didn't work anyway.
4. Try waiting for FlightGlobals.ready before doing the initialization logic.  This flag sometimes triggered before OnLoad, and sometimes after.
5. Block the loader and/or the game from proceeding until the other had reached an acceptable point.  These worked in a vacuum, but are impossible to get working when multiple vessels are active, for the simple reason that I cannot tell whether OnLoad is simply loading from a scene change, or from a user-instigated persistence file load.
6. Manually modify the ScenarioModule's ConfigNode (via snapshot.GetData()).
7. Tweak the KSPScenario attributes.  I've tried just SpaceCenter, just PSystem (OnLoad/Save never fired), and the full collection of in-game scenes (SC, Tracking, Flight).  The problem is that if I remove some of these scenes, then it will not load, but it also will not save.  This was easily tested by trying to quicksave, then checking the log file.
8. Throw an exception in OnLoad to manually check the stack track for "GamePersistence.LoadGame."  This would work for OnSave (which I don't need to touch), but for OnLoad the source is ScenarioRunner.AddModule every time.
9. ...and more

How can I save data to the persistence file (using a ScenarioModule, I assume) without it being overwritten?  I've tried so many things at this point, I'm starting to go crazy.  It doesn't seem like anything involving the load sequence is deterministic.

I also suspect (from the aforementioned stack traces) that the game is actually adding a new copy of the scenario to every scene via ScenarioRunner.AddModule.  Even if I just specify one scene (e.g. GameScenes.SPACECENTER), it does this every time I load the SpaceCenter.

Every post I've found says ScenarioModules are the way to go if saving data to the persistence file, but it seems like such a convoluted way to operate.  On the other hand, I used to be load/saving via GameEvents.OnGameStateSaved.Add(..), and I suspect I'd have the same problems there.

...Bottom line is, I'm just looking to save some data.  Any feedback on how I can do that?

Thanks!

Edited March 31, 2016 by TheBossk
Clarified issue in early paragraph

[Crzyrndm]

Some more ideas:

• (probably) easy but slightly hacky way out, use a coroutine to delay the functionality currently executing in OnStart for 1-2 frames to put it after the scenarioModule is loaded
• Assign to a static variable in the scenario module from the partModule to tell it that data is already loaded and the data from OnLoad can be discarded
• Read through the source of scenario modules from other mods to see if any similar situations have already been dealt with(eg. FAR, TACLS, ...)

It sounds like you're trying to do something odd though as I'm not quite grasping why the bad data is ever being used if it already has good data

Edited March 31, 2016 by Crzyrndm

[Diazo]

I would need to see your code, especially the OnLoad method to help I think.

But can you change your design? By design I would expect data in the OnLoad method to override data in the OnStart method as presumably data being loaded from save is more correct. Where is the state 1 data in OnLoad coming from if you are not loading from a save file?

The other thing is to stick a Debug.Log line in each method. On issue I keep running into is that all these methods (Awake/Start/Load/Save) never seem to run in the order when I expect them to and I've lost significant chunks of time trying to fix "bugs" that were in fact just the methods not running at the time I expected them to.

D.

[TheBossk]

9 hours ago, Crzyrndm said:

Some more ideas:

• (probably) easy but slightly hacky way out, use a coroutine to delay the functionality currently executing in OnStart for 1-2 frames to put it after the scenarioModule is loaded
• Assign to a static variable in the scenario module from the partModule to tell it that data is already loaded and the data from OnLoad can be discarded
• Read through the source of scenario modules from other mods to see if any similar situations have already been dealt with(eg. FAR, TACLS, ...)

It sounds like you're trying to do something odd though as I'm not quite grasping why the bad data is ever being used if it already has good data

Thanks for the suggestions!  I've already tried ideas #1 and #2, and they don't work because if the flight scene is already loaded, and another ship comes into focus (e.g. during a rendezvous), it will run OnStart, but not the scenario's load.  No matter how you slice it, there are cases when either the part's OnStart or the scenario's OnLoad end up with a stale flag, and the code never reaches its proper state.

I may have to read some existing code in detail.  I've so far tried to avoid that, since I'm making a life support mod and I didn't want to end up implementing someone else's solutions/methods.

I'm not sure if this further clarifies what I'm trying to do, but I'm trying to store global info about how much life support is left in each Kerbal's suit.  Maintaining a global dictionary makes it very easy to deduct, whether they are on EVA, or in a part with no life support remaining.

The fundamental problem is that the game loads vessels+parts, runs the parts' OnStart methods, THEN loads up the scenario.  Ideally it would load everything (scenarios, vessels, etc.) first, then run the initialization (OnStart), and finally start running FixedUpdate.  Unfortunately FixedUpdate's first frame is very non-deterministic.

5 hours ago, Diazo said:

I would need to see your code, especially the OnLoad method to help I think.

But can you change your design? By design I would expect data in the OnLoad method to override data in the OnStart method as presumably data being loaded from save is more correct. Where is the state 1 data in OnLoad coming from if you are not loading from a save file?

The other thing is to stick a Debug.Log line in each method. On issue I keep running into is that all these methods (Awake/Start/Load/Save) never seem to run in the order when I expect them to and I've lost significant chunks of time trying to fix "bugs" that were in fact just the methods not running at the time I expected them to.

D.

Thanks for the reply.

For what it's worth, here's a link to my loader.  If you really want to poke around (which I'd appreciate but wouldn't ask of you), SimpleSurvivalLoader calls EVALifeSupportTracker.Load(..), which maintains global tracking data in the form of a dictionary: [Kerbal name] -> [current EVA life support, max life support].

In my PartModule's OnStart, I'm checking lastUT (when the vessel was last loaded) and the current time, in order to deduct LifeSupport that was lost when the vessel was not loaded.  The problem is that following this operation, the ScenarioModule is loading the tracking data that nullifies the resource drain.  I have been trying to find the ConfigNode in memory that OnLoad access, but so far nothing has worked.

I added the logging lines to all the startup methods (OnAwake, OnStart, OnLoad, OnInitialize, etc.) a while back and generated the output below.  I didn't include FixedUpdate, because its start point was not deterministic.

FYI, LSM == the PartModule that monitors life support.

  OnAwake (LSM 1)
  OnAwake (vessel, unloaded) ???
  OnLoad (LSM 1)

  OnAwake (LSM 2)
  OnLoad (LSM 2)

  OnStart (LSM 1)   <-----  This was modifying tracking data
  OnStart (LSM 2)

  SimpleSurvivalLoader.OnLoad()
  	EVALifeSupportTracker.Load()   <----   This was reloading stale data, undoing OnStart's changes

  SimpleSurvivalLoader.OnSave()
  	EVALifeSupportTracker.Save()

  OnInitialize (LSM 1)   <---- FOR FLIGHT ONLY. Loads before OnStart for VesselRollout.
  OnInitialize (LSM 2)

At this point, I'm thinking that the only way this will work is if I move the global tracking data into the PartModules, and shuffle the tracking around as Kerbals move with GameEvents.onCrewTransfer.  It seems like it may be a restriction that ScenarioModules are only useful for storing global data if it is considered read-only on load.  But I'll have to check other mods to see if that's true.

[Diazo]

Okay, looking at your program flow I think you need to make a design change.

My concern is that how can you have OnStart (LSM1) modifying data when the load has not finished yet? With how things are setup, the LSM partModule is deducting resources from an arbitrary amount as the ScenarioModule has not loaded yet so you the LSM OnStart is working with bad data.

I'd look at adding a OperationsQueue object to your EVALifeSupportTracker class that the LSM partModules queue commands into, but only once the LS tracker is ready (so the load is finished) does it start processing the commands in the queue.

So the LSM's would go Queue.Add(Kerbal,Amount); and once it was finished loading, so probably in FixedUpdate, the LS tracker would check the Queue and process any commands present in it.

D.

[Crzyrndm]

9 hours ago, TheBossk said:

Thanks for the suggestions!  I've already tried ideas #1 and #2, and they don't work because if the flight scene is already loaded, and another ship comes into focus (e.g. during a rendezvous), it will run OnStart, but not the scenario's load.  No matter how you slice it, there are cases when either the part's OnStart or the scenario's OnLoad end up with a stale flag, and the code never reaches its proper state.

This is setting off all sorts of alarm bells for me (I haven't actually looked at the code linked at this stage)

Originally, my impression was that you wanted to load saved data before any operations were performed on it. Easy enough, delay Onstart() functionality using a coroutine. Once the data is loaded, we then let OnStart do its thing which then unlocks execution any other functions (eg. FixedUpdate).
Now you're saying that a craft loading into physics range that isn't the active vessel is a problem because scenario.OnLoad doesn't run at this time. That indicates to me that either scenario.OnLoad method is doing more than just unpackaging saved data, or the "live" data is not always in the same location.

I think Diazo is correct in recommending a design change, because something smells very fishy

Somewhat obvious, but ensuring that the sequence is load->start->update is not difficult

Save : ScenarioModule
{
 public static bool hasLoaded = false; // set true at end of OnLoad, set false by OnDestroy
  //...
}

Module : PartModule
{
  public bool hasStarted;
 public void OnStart(StartState)
  {
   if (!Save.hasLoaded)
    StartCoroutine(delayed);
   else
     delayed();
  }
  
  public IEnumerator delayed()
  {
    while(!Save.hasLoaded)
      yield return 0; // I forget the exact syntax...
    // stuff you wanted after load
    hasStarted = true;
  }
  
  public void FixedUpdate()
  {
   if (!hasStarted) // lock out fixed update until data has been loaded and processed
     return;
    // do stuff
  }
}

 

[TheBossk]

Not sure if I'm responding to both of you directly.  This is my first time using a message board like this.  At any rate, I think I've nailed it down, but first, my responses to both of you:

Stalling OnStart with a coroutine was originally what I tried (and I tried it a bunch of different ways), but the problem is twofold: PartModule.FixedUpdate may begin either before or after OnLoad, and OnLoad may not run at all.  I don't remember whether I tried OnDestroy, although that is a good idea.  Perhaps I'll perform that experiment if I have time and update this post.

The solution, for anyone who stumbles across this

The solution seems to be using a KSPAddon instead of a ScenarioModule.  This is what I was originally doing, but was having several problems with ConfigNodes at the time (my own doing), and decided a ScenarioModule would be easier.  It turns out that I barely had to change anything moving back to an addon.

By plugging a KSPAddon method into GameEvents.onGameStateCreated, it will always load before PartModule.OnStart, which resolves any order-of-operations questions.  I've tested this switching from the tracking station to flight, quickloading from flight to flight, launching a new vessel, and loading a persistence file from the space center, but I'll update this post if I encounter any problems.

Here's the loader mod-agnostic form:

    [KSPAddon(KSPAddon.Startup.MainMenu, true)]
    public class MyModLoader : MonoBehaviour
    {
        // The name of your mod's config node
        // Use something descriptive, NOT "MY_MOD"
        private const string TOPNAME = "MY_MOD";

        // Fire ONCE, hook routines into GameEvents
        public void Awake()
        {
            KSPLog.Log("Loader Awake(..)");

            GameEvents.onGameStateCreated.Add(OnLoad);
            GameEvents.onGameStateSave.Add(OnSave);

        }

        public void OnSave(ConfigNode topnode)
        {
            KSPLog.Log("Loader OnSave(..)");

            // ConfigNode should be in the process of being constructed
            // from scratch, and not yet have the TOPNAME node
            if (topnode.HasNode(TOPNAME))
                KSPLog.Log("CheckThis -> Node " + TOPNAME + " already exists!");

            ConfigNode mod_node = topnode.AddNode(TOPNAME);

            EVALifeSupportTracker.Save(mod_node);
            ContractChecker.Save(mod_node);
        }

        public void OnLoad(Game game)
        {
            KSPLog.Log("Loader OnLoad(..)");

            // onGameStateCreated triggers OFTEN (including in the MainMenu scene)
            // Have to make sure we've actually loaded a game
            if (!HighLogic.LoadedSceneIsGame)
                return;
            
            ConfigNode mod_node;

            // If this save is being loaded for the first time
            // (b/c a new game was created OR the save
            // previously did not have this mod installed),
            // there is nothing to load, so create an empty node.
            // It's up to each component loader to properly process
            // an empty node and initialize its data.

            if (game.config.HasNode(TOPNAME))
                mod_node = game.config.GetNode(TOPNAME);
            else
                mod_node = new ConfigNode(TOPNAME);

            EVALifeSupportTracker.Load(mod_node);
            ContractChecker.Load(mod_node);
        }
    }

I'm not sure I understand the utility of ScenarioModules if their data isn't guaranteed to be loaded by the time FixedUpdate begins.  Then again, I'm looking for a very specific application, and ScenarioModule didn't fit the bill.

Thanks again @Diazo and @Crzyrndm for your input!

Edited April 2, 2016 by TheBossk
Cleaned up code sample formatting