Resource Flow Mechanics Overhaul

[Majiir]

There have been several discussions on this topic in #kspmodders, so I thought I'd bring this to the forum so we can better organize our thoughts.

Several mods are limited by the current resource flow mechanics. There's been discussion about overriding the resource request routines to better facilitate the needs of mods. Some relevant mods:

• Kethane would like to be able to simulate resource flow on unloaded vessels.
  
• KSP Interstellar has an internal resource manager for handling vast quantities of electricity.
  
• TAC Life Support bypasses the stock resource draw mechanic and substitutes its own.

There are implementation details which make it difficult for mods to individually implement desired features without conflicting with others. The goal of this post is to gather requirements for a resource flow API that mods can optionally use.

A few items to get us started:

• Batch resource requests. Producing or consuming multiple resources in proportion is non-trivial, and the rest of the library may necessitate this feature.
  
• Storage-independent request resolution. In other words, how much (or little) resource storage a vessel has should not influence how resources flow in that vessel. This applies to high-timewarp situations but also comes up when a vessel has zero storage for some resource but produces and consumes it simultaneously.
  
• Offline resource flow. This might be implemented by allowing modules to declare a constant resource flow rate and resolving storage deltas when the vessel is loaded again.

My intent is to foster discussion, so please, post suggestions and critiques.

[damny]

allowing modules to declare a constant resource flow rate

This may not be enough, e.g. the output of a solar panel depends on where it is in space. Ideally, it would be possible to (optionally) specify a class that can handle the calculations.

[Fractal_UK]

So, the Interstellar resource manager covers some of the objectives named here but let me go through and briefly discuss what it does well and what it doesn't do so well.

Capabilities of the Interstellar Resource Manager:

• Managed Supply of Resources - The idea here is that the resource manager doesn't want to produce more of a resource than can be used, so, if we are using the resource manager to manage a supply of Thermal Power created by a nuclear reactor, we want to supply enough to provide for the ship's power needs and no more - otherwise we are wasting nuclear fuel.
  
• Storage Independence - We don't want huge buffers to make everything work at high time acceleration, the Interstellar resource manager should still function properly with only 1 unit of storage of a given resource - it is capable of using ongoing supply to meet demand.
  
• Power Organisation and Prioritisation - Not everything needs power to the same degree, if you're running lots of science equipment on your ship that's nice but I'm guessing you want power to the engines first. The Interstellar resource manager works by organising resources by demand and by priority, it supplies the smallest demand of the highest priority and then moves onto the next smallest of the highest priority, etc. until either all the needed power has been supplied or the power has been exhausted.
  
• Minimal Conversion from Stock Resource API - No one wants to rewrite lots of code so that idea was to encapsulate all of this functionality into something very much like the part.RequestResource() method. The resource manager typically uses just two methods, supplyFNResource() and consumeFNResource(), though PartModules that wish to supply a resource must also declare which resources they intent to supply OnStart.
  

The Interstellar resource manager is very powerful but it isn't perfect and has some drawbacks

• Offline Resources - This is totally unsupported at present and Interstellar's handling of offline resources is, for the most part, a very simple delta fudge.
  
• One-Off Supply and Demand - The resource manager is heavily optimised toward the handling of power supply and demand that goes on continuously for at least several frames, supplies and demands that last for only an instant are not well supported.
  

Here is an example of the resource manager in action with many demands. Here all the draws are of the same priority so the resource manager has organised them solely by demand and you can see that the science lab with it's huge 5GW power draw is the one that missed out in order to keep the other components supplied.

[NathanKell]

Other desiderata:

*Max flow rate: necessary to properly model batteries (and provide a use for capacitors) but also useful to model a tank's max fuel outflow. It's probably too much work to separate that into volts and amps, though, and instead we can just consider a battery's max flow in its own units per second.

*More control over flow: fuel should flow from the lowest tank first...except if there are decoupleable tanks (aka "STACK_PRIORITY_SEARCH has issues").

[DerekL1963]

As a user, my main interest is backwards compatibility... should mod 'x' require version 2.3.2, it should still support the features of version 2.3.0 for mod 'y'. I give and grant there will be cases (such as changes to the game itself) where this is not possible, but backing the user into the corner of choosing one mod over another should be avoided if at all possible.

[rbray89]

This may not be enough, e.g. the output of a solar panel depends on where it is in space. Ideally, it would be possible to (optionally) specify a class that can handle the calculations.

This. There should be a template class that would have methods for generation & consumption. If this were standardized, then these instantiated classes could simply be plugged into the resource manager.

[Greys]

Solar panels vary in a 'daily' cycle that is largely but not entirely uniform from 'day' to 'day', if you define a day to be one complete movement around the parent body and thus one surface day is conceptually the same as one orbital day. It should be possible to keep a history of samples (say one sample every 2 seconds) and if you have a minimum of one day's worth then that ship is unloaded or when the player leaves the flightspace; calculate a curve from those samples and store it to the craft's persistence; then you could cheaply calculate production for solar dependent factors at any point from a simple request instead of adding new constant simulations.

Alternately it should be easy to use a spherical intersection comparison somethingerother to determine what percentage of the day you're exposed to the sun.

[Chris_W]

From my work with BioMass and TAC LS so far I fully agree that we need a clever resource handler.

Some thoughts on things:

Flow modes

I just had the thought "why not leave this up to the designers of the craft".

Idea being, a Tweakable that specifies a priority - or a variable in the part.cfg.

Priority could be twofold, for input and output.

A command pod would for instance hold..

Oxygen - Input prio 10, output prio 1 (first to store, last to give away)

CO2 - Input 1, output 10 (first to give away, last to store)

Whereas a greenhouse would beg for CO2 and give away oxygen.

Some canister somewhere on the craft could let the user decide how important it would be and default to 5.

Locking certain containers should still be possible as well (emergency batteries for instance that you unlock by hand later).

Activation

Current stock model makes no sense whatsoever.

I'd like to be able to specify if I want an on/off switch and the state the switch should be in on load.

Some things better start activated already. Other things I'd declare always active, like bacteria growth, without a switch.

Also resources should prevent the reaction from ocurring if they're not available and specified as vital.

Sometimes you can still get things happening with one resource less, just at a lower efficiency.

Output

Specifying what output should be generated is tricky sometimes.

Came across this when handling the bacteria from BioMass.. they "feed" on waste products and they generate two other products. One of them they should always generate, just because they exist onboard. The other one should be governed by the input. So what I'd like here is a method to say "this depends on efficiency, this does not and is always generated".

Units

I'd very much like to be able to specify "units per time" more efficiently.

Allowing to specify an amount and a time would greatly ease things.

units = 1, time = day.. much easier to handle than "0.0000117 per second".

Update time scaling

Other than the obvious, a given amount of time between updates per reaction would make the whole thing more enjoyable.

If something creates 1 unit per day you most likely want to reduce it every 6 hours by a fraction if it is food, yet by a much smaller amount and every second if it is oxygen or electricity. Even Kerbals are not permanently eating.. their microbacteria in the lab on the other hand don't have much else to do, so they sure are.

Just my two cents on what I experienced thus far, might not be much but.. may help thinking.

[BioSehnsucht]

Solar panels vary in a 'daily' cycle that is largely but not entirely uniform from 'day' to 'day', if you define a day to be one complete movement around the parent body and thus one surface day is conceptually the same as one orbital day. It should be possible to keep a history of samples (say one sample every 2 seconds) and if you have a minimum of one day's worth then that ship is unloaded or when the player leaves the flightspace; calculate a curve from those samples and store it to the craft's persistence; then you could cheaply calculate production for solar dependent factors at any point from a simple request instead of adding new constant simulations.

Alternately it should be easy to use a spherical intersection comparison somethingerother to determine what percentage of the day you're exposed to the sun.

Could probably just check the max power generation of all deployed panels when craft goes on rails and store it (so we don't have to compute it again later), assume craft is optimally pointed at sun, and for each timestep calculate based on distance from sun and really simple approximation check of whether the craft can 'see' the sun from it's location (and whenever the time step is such that the craft may have changed state between being in a shadow and not, back-compute some number of sub-steps between the last step and this one to figure out when it changed state, otherwise the entire time step it was either in light or not).

[Greys]

That would be precisely what I was trying to avoid, new constant simulation, an approximated curve is marginally less accurate but ends up as a simple lookup based on a percentage of orbital offset from zero position.

[JordanL]

That would be precisely what I was trying to avoid, new constant simulation, an approximated curve is marginally less accurate but ends up as a simple lookup based on a percentage of orbital offset from zero position.

Yeah, a lookup table that's created once it can be for the vehicle is actually going to be more accurate in most cases, simply because there is less room for error/bugs. This sort of this probably should be done on a curve lookup.

[Greys]

The biggest issue that I can foresee with my limited abilities is that we won't be able to make the stock modules and partmodules use our manager; I may be wrong about that, and I will be wrong gladly; but if I'm not wrong it would mean that this alternate resource manager (ARM) would need to also contain either natively or as a pack-along sibling project, a set of replacement modules for anything in KSP that consumes or creates resources.

as far as I recall that's

ModuleEngine

ModuleAlternator

ModuleGenerator

ModuleDataTransmitter

ModuleDeployableSolarPanel

ModuleLight

ModuleRCS

ModuleReactionWheel

ModuleResourceIntake

ModuleEnviroSensor

ModuleScienceLab?

ModuleWheel

Plus some modules that some people would probably like to have consume resources like ModuleControlSurface and ModuleLandingLeg/Gear, maybe even ModuleDockingNode; those probably aren't permanent magnets helping us snap ships together.

Taking that into account, this project has a lot of potential, as I can name major known flaws in the behavior or performance of almost everything in that list.

[Sakata]

I agree that this project would be very good for many mods.

Biomass is one that comes to mind, but others would also benefit from a common DLL, so that they can focus on their mod, and not the resource bugs/features in the default game. If a sufficiently robust system were made, then many mods would be able to focus more on other things, instead of each modder having to write their own libraries to deal with resource calculations which I'm sure many have had to do over and over. Its like re-inventing the wheel...

This IMHO would be awesome!

[Papa_Joe]

I'm new to unity and KSP modding, but not new to coding.

I'm just beginning to grapple with these very questions as I try to implement a "realistic" resource transfer from part to part in Ship Manifest. I want a realism mode that would simulate the time, flow characteristics and mechanical sounds for moving resources that can be moved, based on flow modes, flowstate and connected path.

My concerns are how to best handle the duration based on amount to move and some flow rate, and have it sync with whatever sound(s) I'm choosing for that resource. I'm thinking it probably needs to be an event on the part, but, not sure how to sync it to a second based duration. I'm now perusing various mods to learn how all this is currently being accomplished. A library is exactly what is needed for the community. I would contribute.

[landeTLS]

Would it be possible to include a handler for the nullref exception when you try to request a resource that the vessel cannot store. For example when you use the ISRUScoop () component from kspi.

PS:

Any snippet to do a check of resource to request against vessel.activeresources(etc. Sorry on my phone. Cant copypaste from the api) before requesting would be greatly appriciated btw.would probably go in the partmodule class under OnUpdate () probably (just guessing since thats when ypu switch resource and online the extraction) Havent got it working myself both due to lacking experience with c# and lacking documentation on that api

Edited December 31, 2013 by landeTLS

[Papa_Joe]

Would it be possible to include a handler for the nullref exception when you try to request a resource that the vessel cannot store. For example when you use the ISRUScoop () component from kspi.

PS:

Any snippet to do a check of resource to request against vessel.activeresources(etc. Sorry on my phone. Cant copypaste from the api) before requesting would be greatly appriciated btw.would probably go in the partmodule class under OnUpdate () probably (just guessing since thats when ypu switch resource and online the extraction) Havent got it working myself both due to lacking experience with c# and lacking documentation on that api

It is a bit of brute force approach, but this snippet collects all the parts and organizes them by Resource. the end result is a Dictionary object that can be interrogated by PartsByResource.Keys.Contains(resource.info.name), that provides a list of parts that contain the desired resource.

        // Provides a list of resources and the parts that contain that resource.
        private Dictionary<string, List<Part>> _partsByResource;
        private Dictionary<string, List<Part>> PartsByResource
        {
            get
            {
                if (_partsByResource == null)
                    _partsByResource = new Dictionary<string, List<Part>>();
                else
                    _partsByResource.Clear();

                foreach (Part part in Vessel.Parts)
                {
                    // Part has Resources, now where to put them...  (may be more than one place... 
                    foreach (PartResource resource in part.Resources)
                    {
                        bool vFound = false;
                        // is resource in the list yet?.
                        if (_partsByResource.Keys.Contains(resource.info.name))
                        {
                            vFound = true;
                            List<Part> eParts = _partsByResource[resource.info.name];
                            eParts.Add(part);
                        }
                        if (!vFound)
                        {
                            // found a new resource.  lets add it to the list of resources.
                            List<Part> nParts = new List<Part>();
                            nParts.Add(part);
                            _partsByResource.Add(resource.info.name, nParts);
                        }
                    }
                 }

                return _partsByResource;
            }
        }

Edited December 31, 2013 by Papa_Joe

[landeTLS]

It is a bit of brute force approach, but this snippet collects all the parts and organizes them by Resource. the end result is a Dictionary object that can be interrogated by PartsByResource.Keys.Contains(resource.info.name), that provides a list of parts that contain the desired resource.

        // Provides a list of resources and the parts that contain that resource.
        private Dictionary<string, List<Part>> _partsByResource;
        private Dictionary<string, List<Part>> PartsByResource
        {
            get
            {
                if (_partsByResource == null)
                    _partsByResource = new Dictionary<string, List<Part>>();
                else
                    _partsByResource.Clear();

                foreach (Part part in Vessel.Parts)
                {
                    // Part has Resources, now where to put them...  (may be more than one place... 
                    foreach (PartResource resource in part.Resources)
                    {
                        bool vFound = false;
                        // is resource in the list yet?.
                        if (_partsByResource.Keys.Contains(resource.info.name))
                        {
                            vFound = true;
                            List<Part> eParts = _partsByResource[resource.info.name];
                            eParts.Add(part);
                        }
                        if (!vFound)
                        {
                            // found a new resource.  lets add it to the list of resources.
                            List<Part> nParts = new List<Part>();
                            nParts.Add(part);
                            _partsByResource.Add(resource.info.name, nParts);
                        }
                    }
                 }

                return _partsByResource;
            }
        }

Cool. Thank you so much ill try it tonight

[landeTLS]

        // Provides a list of resources and the parts that contain that resource.
        private Dictionary<string, List<Part>> _partsByResource;
        private Dictionary<string, List<Part>> PartsByResource
        {
            get
            {
                if (_partsByResource == null)
                    _partsByResource = new Dictionary<string, List<Part>>();
                else
                    _partsByResource.Clear();

                foreach (Part part in Vessel.Parts)
                {
                    // Part has Resources, now where to put them...  (may be more than one place... 
                    foreach (PartResource resource in part.Resources)
                    {
                        bool vFound = false;
                        // is resource in the list yet?.
                        if (_partsByResource.Keys.Contains(resource.info.name))
                        {
                            vFound = true;
                            List<Part> eParts = _partsByResource[resource.info.name];
                            eParts.Add(part);
                        }
                        if (!vFound)
                        {
                            // found a new resource.  lets add it to the list of resources.
                            List<Part> nParts = new List<Part>();
                            nParts.Add(part);
                            _partsByResource.Add(resource.info.name, nParts);
                        }
                    }
                 }

                return _partsByResource;
            }
        }

Ok i may be a c# noob(i am) but copying this directly into a class causes an error here

foreach (Part part in Vessel.(error--->)Parts(<---error))

does it need to be encapsulated into a method? I am finding the definition "Vessel.Parts" when i do a goto definition in the ide

Also. Would it be possible to do something like this using a method with a bool returnvalue:

If ActiveVesselCanStoreResource(resourcename)

{ resource extractioncode here } else

{ do something else here }

EDIT:

i figured out doing some debugging that the nullref was not because of requesting resource not storable on vessel in the code, appearently the api is able to handle that, what it doesnt handle is trying to request a resource that is not defined, which was my problem. i seem to have defined either too many resorces in the cfg or have some other issue. but its working now(removed the gas resourses and have used the same definitions as the solid ones eg. propellium instead of propelliumgas) no nullrefs

Edited January 2, 2014 by landeTLS

[seanth]

A few thoughts as a coder that has not done any coding for KSP. Some general, some specific:

1.) Squad's generator module is truly and deeply broken when it comes to using more than one input resource. That alone pretty much guarantees that modders needs to roll their own generator code or use ones made by others.

2.) There are lots of mods out there with great, useful features. For example, I use Majiir's KethaneConverter a lot right now. However, these useful bits are often bundled in with code that might not be wanted. It would be supremely useful if things were more modular, even if it meant including multiple dlls with mods.

3.) To my knowledge, there is no generator module that takes into account the amount of a resource and performs an adjustment on the base output based on that. In my specific case, there is no way to scale oxygen output with amount of photosynthetic biomass. Ideally, the more biomass present, the greater the amount of O2 produced. Put another way: current generators I know of just take a fixed amount of a resource A per unit time and turn it into resource B. I would love something that takes the amount of resource A, has a baseline conversion rate, and then uses the total amount of A to scale the amount of B accordingly. If I am wrong here, please oh please point me in the correct directions. See also point 2.

4.) This is a stretch, but it would nice if we could code right in a cfg using something lightweight (lua? I dunno what else would be considered light weight. Python seems heavy). Then people could roll their own code without needing C# and making a dll.

That's all I have. I mean, I have very specific things I'd love to see--and may end up doing some coding, though I do enough of that and would rather not--but the above points hit my basic thoughts.

Edited January 1, 2014 by seanth
improved readability

[landeTLS]

Nr 3 is something that would be really nice to have. I remember trying to do something like that in a part config with a complex line of kethaneconverter modules. But couldnt get it working. Something that would be able to accept a powercurve(like the solarpanel module) in the cfg would probably be the best approach.

[Aatch]

Ok, so this doesn't get too fragmented, there are currently two projects related to resources going on: http://forum.kerbalspaceprogram.com/threads/61605-Modding-Team-Wanted-the-Resource-Expansion-Module-Project and http://forum.kerbalspaceprogram.com/threads/64595-Open-Resource-System-%28ORS%29-Mod-Resource-API-version-1-0-0

We probably don't want to end up with a million different ways to handle resources, so it'd be nice to work together to make sure there's only really one canonical way to do this.

Edited January 6, 2014 by Aatch
accidentally a word

[Greys]

The ORS is completely unrelated to this; it's an open resource spawning and collection system similar to Kethane

The other 'thing' is a bunch of nonsense and is mostly attempting the same thing.

This thread is looking to replace how resources in parts that are on ships behave, and how other parts that need to create or consume those resources do so.

[Aatch]

Ah yes, I see that now, just a little difficult to figure out which system is doing what. Still, probably a good idea to keep related projects in mind while working on this.

[Majiir]

The biggest issue that I can foresee with my limited abilities is that we won't be able to make the stock modules and partmodules use our manager; I may be wrong about that, and I will be wrong gladly; but if I'm not wrong it would mean that this alternate resource manager (ARM) would need to also contain either natively or as a pack-along sibling project, a set of replacement modules for anything in KSP that consumes or creates resources.

As of KSP 0.23, we can once again override Part.RequestResource(). In other words, we can rewrite the resource handling code for any moduleâ€â€past, present or future. I recommend caution to anyone who feels like doing this, because it's a core part of the game and only one mod can reasonably override resource behavior at a time. That said, I expect that whatever solution comes out of this thread will use it.

[Hyomoto]

I didn't see this mentioned but I feel this should really be at the top of the list. Some method that can avoid modders creating their own versions of the same resources. The user doesn't want to deal with your uranium and his uranium, he just wants the uranium to work. I'm not sure if that's more of a common courtesy for the user situation, but something like the ore dictionary with the Minecraft Forge API. This sort of ties into a 'plain language' approach as well. KWatts or TACOxygen isn't useful information for the user, it just serves to scatter the concept.

Ideally it wouldn't just an API for creating and handling new resources, but also for tying together existing ones to promote a higher level of compatibility between mods.