Multi processor space probes with kOS

[TheCardinal]

Computers and space have fascinated me for as long as i can remember. The combination, in the form of automated probes, even more so.

In the real world it is not uncommon that a probe develops one or more problems. My goal is to develop a system consisting of software and processors with kOS which is as fault tolerant as possible in KSP.

You might ask yourself: Why a multiprocessor system, Isn't one processor enough? Well, frankly one processor is definitely not enough. First of all, errors (in code and hardware) WILL happen. Error can't be caught by kOS. Any error will stop the processor executing the program. With just one processor, your probe is essentially dead and a total loss. kOS does allow the user to control the craft after such a fault occurs. That means that there must be at least one other processor on board which takes care of communication and such. In the system i want to (will!) develop, there will be no hidden processor(s) like that. If a processor halts, it must be reset bij another processor on the probe and not by typing commands in the terminal window.

Another reason why using multiple processors are required has to do with computerarchitecture. Any computer, from large mainframecomputers to even mere PC's, has controllers which take care of specific tasks. In theory, the CPU could do all those tasks but it would make the computer very slow. Most probably, it would render the computer impractical. With controllers, the CPU can delegate certain tasks and continue with it's own workload. It makes the computer faster and more fault tolerant. After a controllerfailure one function becomes unusable but not the entire computer.

The third reason is that the complete system can be less complex and more modular.

 

I will be using several mods, some are really required, some are not.

The most important are:

• Remote tech
• Universal Storage - Provides an easy way of placing multiple processors on a craft (i don't like to use multiple HAL9000's for several reasons)
• kOS (1.0.0) - Will be upgraded to release 1 as soon as some bugs in that relase are solved.
• MechJeb 2/MechJeb 2 embedded - Will only be used to display flightdata and will be removed when a kOS processor does that.
• Scansat - For science subsystem and testing.
• Kaboom - Used as selfdestructmechanism.
• Procedural parts - Will only be used if it is unavoidable. (has been removed)
• KSP 1.1.3 64 bits - The 64 bits 1.0.0 kOS version works correctly (the 0.21.1 version did not).

Mods which are not essential (but can provide eye candy or might be usefull):

• Aviationlights - lighting the probe and possibly also as "indicatorlights".
• Agext / Modactions - For easy programming specific tasks like opening/closing solarpanels
• HullCamVDS
• BD Armory - FLIR and IR camera
• Launchnumbering
• KerbalAlarmClock
• KerbalJointReÏnforcement

I probably will remove (parts of those) mods when they're not used to save memory space. I will be adding some modified parts as well (a universal container as kOS processor f.e.).

 

Edited September 7, 2016 by TheCardinal

[MedwedianPresident]

You mean a system that adapts to problems? Like "If left engine fails lock gimbal of right engine and set thrust limiter to 80%"? Or "Override landing program if not enough fuel for return"?

 

Probes with kOS are interesting, but IRL manned spaceships are often automated too, by the way.

[TheCardinal]

2 hours ago, MedwedianPresident said:

You mean a system that adapts to problems? Like "If left engine fails lock gimbal of right engine and set thrust limiter to 80%"? Or "Override landing program if not enough fuel for return"?

 

Probes with kOS are interesting, but IRL manned spaceships are often automated too, by the way.

Primarily i'm focussing on a system which will react to processing problems. F.e. A specific processor halts/fails and then the system must perform such and such action. Decisions like "If left engine fails ..." and "Override .." usually are not decided by the probe itself but left to the technicians at mission control.

In the system to be developed each processor will be assigned a specific task, including what to do if the task cannot be performed due to processorfailure and/or some other causes.

[MedwedianPresident]

9 minutes ago, TheCardinal said:

Primarily i'm focussing on a system which will react to processing problems. F.e. A specific processor halts/fails and then the system must perform such and such action. Decisions like "If left engine fails ..." and "Override .." usually are not decided by the probe itself but left to the technicians at mission control.

In the system to be developed each processor will be assigned a specific task, including what to do if the task cannot be performed due to processorfailure and/or some other causes.

You mean that processors that are used for unimportant tasks (e.g. minor inclination correction, landing leg damper control, light control) will be used to perform important tasks (e.g. throttling engines and orienting, life support systems, takeoff and return from Mun, etc...) if the processors that were originally supposed to perform them were destroyed or their program was faulty?

[TheCardinal]

1 hour ago, MedwedianPresident said:

You mean that processors that are used for unimportant tasks (e.g. minor inclination correction, landing leg damper control, light control) will be used to perform important tasks (e.g. throttling engines and orienting, life support systems, takeoff and return from Mun, etc...) if the processors that were originally supposed to perform them were destroyed or their program was faulty?

In case of a hardware failure, yes. But such a decision would still be taken at mission control. In case of a faulty program, assigning that same program to another processor would only result in another processor failure.

Let me try to give an example. A probe with multiple processors. One processor ceases to function. The first question is: How do you detect that a processor has failed? Next question will be: Hardware/Software? Does a reboot of that processor solve the problem and how do you detect if it functions again? How many times should the probe attempt that approach? What other actions can/should the probe take? What kind of messages should the probe send so mission control can determine the problem and possible solution? What if there's no connection with mission control?  Should the probe attempt to run the program(s) of the failed processor in another (or a spare?) processor? What information can/should be extracted for a proper fault determination?

Since weight is always very important, you cant ádd a processor for every small task. In other words, ít is impractical to assign one processor to manage the oxygen, one for the liquid fuel, one for the monopropellent etc.. Instead you group tasks and use one processor to manage and control every resource.

[TheCardinal]

I've started with a first, simple, experiment. In careermode i've launched comsats with each five processors. Three of them are deactivated at the launch and one is busy with the ascent and circularisation. The fifth checks continueously if the altitude is above 60000. When that altitude is reached, this processor will jettison the fairings and deploy the solarpanels. Of course the fairings normally will be jettisoned when the apropriate stage is activated. To make sure the fifth pprocessor takes care of the fairings, the fairing symbol has been moved to a stage which only be activated when the altitude is way beyond 60000.

It proves the advantages of having multiple processor on a probe.

Edit: I've added the Kaboom! mod in order to have the option to have the craft selfdestruct if it deviates too much from it's flightpath during ascend. (The program will run in one of the processor's during ascend).

Edited August 13, 2016 by TheCardinal

[TheCardinal]

kOS 1.0 has just been released and i'm switching to that version of kOS. Having interprocessor communication is too advantageous for the development of a multiprocessor system.

[TheCardinal]

 

In the picture above an image of a five processor complex, consisting of a (small) central core and four radially attached side processors. Each processor has a specific nametag to ensure that the correct program is run in the appropriate processor. The central core and the sidemounted parts stem from the Universal Storage mod by @Paul Kingtiger and @Daishi. I'm using the models and textures of the US_1C10_Wedge_Quadcore and the US_1P140_Wedge_WRS. For each a new cfg has been written for their new function(s).

The processor-ring consists of

• the central core tagged GUI; memory 10000 bytes; ElectricCharge 5; is a command pod. It controls the other cores, staging, selfdestruct, program distribution.
• a side core tagged COM; memory 20000 bytes; ElectricCharge 50. Takes care of communication including writing a mission log to the archive.
• a side core tagged TLM; memory 20000 bytes; ElectricCharge 50. Takes care of telemetry data such as altitude, thrust etc.
• a side core tagged RES; memory 20000 bytes; ElectricCharge 50. Takes care of resource management, especially ElectricCharge.
• a side core tagged NAV; memory 20000 bytes; ElectricCharge 50. Takes care of the ascent, heading, direction, rotation etc.

Before launching, GUI runs a bootprogram which initializes the other processors and re-initializes GUI.

Each processor sends "I'm alive" messages every second to the messagequeue of GUI after initialisation. GUI constantly checkes it's messagequeue and updates the (last) message time of each processor. If that time exceeds 2 seconds, the GUI prints of a warning and sets the message time to a negative value. If a negative value is detected at the next check, GUI reboots the failing processor. If it's just a fluke, this will solve the problem, but when at the next check again a negative value is detected, it didn't solve the problem and GUI will set the bootfilename of the failing processor to a special bootprogram and reboot it again. This should solve the problem (for now) if there's a problem with the processorprogram. In case of a hardwarefailure, it will not solve the problem and at the next check a negative value will still be detected. In such a case, GUI deactivates the failing processor and removes it from the checksequence.

Next step will be to determine what additional steps should be taken in case of a soft/hardware failure of each processor, what to do if TLM fails (or COM or ...).

Paul KingTiger has been so kind to help me modify the cfg files in such a way that module manager takes care of model and textures files for both, so no additional files are needed. Installation of Modulemanager, kOS (v1.0.0) and UniversalStorage are required though.

The CFG files have now been updated:

US_1C3PO_Wedge_Quadcoreprocessor.cfg

US_1R2D2_Wedge_kOS_Processor.cfg

 

 

Edited August 29, 2016 by TheCardinal
added cfg files

[TheCardinal]

The GUI is the only processor which has a bootfile set in the VAB (GUI_INIT.KS). At launch, this is the (first) program to be executed. It assigns several parameters to all processors and initialises them. This approach is somewhat more flexible than assigning a bootfile for each processor in the processor complex in the VAB. Whenever a mission requires a different bootfile for one (or more) processor(s), altering the GUI_INIT.KS is enough. No need to edit the processors in the VAB.

//GUI_INIT.KS
//
//     FYI is a function in LIB_GENERAL.KS
//
FUNCTION HAS_FILE {PARAMETER n. PARAMETER v. SWITCH TO v. LIST FILES IN f. FOR file IN f {IF file:NAME = n {SWITCH TO 1. RETURN TRUE.}}SWITCH TO 1. RETURN FALSE.}

copypath("0:/libraries/Lib_general.ks",""). 
copypath("0:/boot/Safemode.ks",""). 
run "Lib_general.ks". 
set terminal:width to 40. 
set terminal:height to 12. 
FYI(" ").
FYI("--------   S T A R T   O F   M I S S I O N    L O G   --------").
FYI(" ").
FYI("Guidance is internal.").
WAIT 0.1.
FYI("Deactivating other processors.").
processor("NAV"):deactivate.
processor("RES"):deactivate.
processor("TLM"):deactivate.
processor("COM"):deactivate.
FYI("Copying general files to other processors.").
copypath("Lib_general.ks","TLM:/Lib_general.ks").
copypath("Lib_general.ks","NAV:/Lib_general.ks").
copypath("Lib_general.ks","COM:/Lib_general.ks").
copypath("Lib_general.ks","RES:/Lib_general.ks").
WAIT 0.2.
FYI("Initializing processor TLM").
copypath("0:/boot/TLM.ks","TLM:/TLM.ks").
set processor("TLM"):bootfilename to "TLM.ks".
processor("TLM"):activate.
wait 0.3.
FYI("Initializing processor COM").
copypath("0:/boot/COM.ks","COM:/COM.ks").
set processor("COM"):bootfilename to "COM.ks".
processor("COM"):activate.
wait 0.3.
FYI("Initializing processor NAV").
copypath("0:/boot/NAV.ks","NAV:/NAV.ks").
copypath("0:/libraries/Lib_NAV.ks","NAV:/Lib_NAV.ks").
set processor("NAV"):bootfilename to "NAV.ks".
processor("NAV"):activate.
wait 0.3.
FYI("Initializing processor RES").
copypath("0:/boot/RES.ks","RES:/RES.ks").
set processor("RES"):bootfilename to "RES.ks".
processor("RES"):activate.
wait 0.3.
FYI("Initializing self").
copypath("0:/boot/GUI.ks","").
set processor("GUI"):bootfilename to "GUI.ks".
wait 0.3.
FYI("Rebooting").
wait 2.
Reboot.

The file LIB_GENERAL.KS is copied and executed by each processor. The code ensures that each processor sends an I'm alive (IMA) message every second with function FIU. Messages are sent to the mission log with function FYI. The file also opens a terminal for each processor.

The file contains the following code:

//LIB_GENERAL.KS

FUNCTION FORMATMET {LOCAL ts IS TIME+MISSIONTIME-TIME:SECONDS. RETURN "[T+"+padZ(ts:YEAR-1)+"-"+padZ(ts:DAY-1,3)+" "+padZ(ts:HOUR)+":"+padZ(ts:MINUTE)+":"+padZ(ROUND(ts:SECOND))+"]".}
FUNCTION PADZ {PARAMETER t, l is 2. RETURN (""+t):PADLEFT(l):REPLACE(" ","0").}
FUNCTION FYI {PARAMETER n. SET a1 TO PROCESSOR("gui"). SET a2 TO a1:CONNECTION. a2:SENDMESSAGE(core:tag+";"+n). SET a1 TO PROCESSOR("com"). SET a2 TO a1:CONNECTION. a2:SENDMESSAGE(core:tag+";"+n).}
FUNCTION FIU {PARAMETER n. SET a1 TO PROCESSOR("gui"). SET a2 TO a1:CONNECTION. a2:SENDMESSAGE(core:tag+";"+n).}

set klok to time:seconds.
when time:seconds > klok + 1 then {set klok to time:seconds. FIU("IMA"). return true.}
CORE:PART:GETMODULE("kOSProcessor"):DOEVENT("Open Terminal"). 
CLEARSCREEN.

 

Edited August 27, 2016 by TheCardinal

[TheCardinal]

During testing, i noticed that the coreparts didn't work together with RemoteTech (RT). No connection with mission control was established, no matter what i tried. After Google-ing i found out that the parts needed a signalprocessor (??!).  RT has several configuration files in it's directory to enable other mods with probecores to work with it. So i created a configuration file for the modified/new parts and added that to RemoteTech. That solved the connection problem.

One other thing that i noticed in KSP is that if the US_1C3PO_Wedge_Quadcoreprocessor gets deactivated (the command pod), the craft no longer is classified by KSP as a craft but becomes debris. Even if the core is activated again, the craft remains debris. It means that the craft type then has to be altered manually in KSP.

RemoteTech_US_KOS_probes.cfg

 

Edited August 29, 2016 by TheCardinal

[Drew Kerman]

On August 9, 2016 at 3:09 AM, TheCardinal said:

KSP 1.1.3 32 bits - The 64 bits version unfortunately doesn't play nice with kOS otherwise it would have been the 64 bits version.

What's this now? First I've heard of such a thing

[TheCardinal]

I had problems with kOS 0.20.1 and it's prerelease candidate combined with KSP 1.1.3 64b even without mods. The combination gave me problems i could only solve by using K.S.P. 1.1.3 32b. I haven't tried KSP 64b with kOS 1.0.0 (yet).

Reddit thread

[TheCardinal]

I've tried KSP 64b with kOS 1.0.0 and the problems i had, didn't occur, so i've switched to KSP 64 bits.

Edited September 7, 2016 by TheCardinal

[TheCardinal]

Waiting for kOS for KSP 1.2.

Edited October 10, 2016 by TheCardinal