[1.1.3] kIPC - Inter-Process(or) Communication between kOS and KRPC - v0.2.0-beta now available!

[dewin]

So I finally started playing with the kOS mod a couple of months ago, and started writing a comprehensive maneuver planning library for it.  This hit a brick wall pretty quickly when I tried to implement an interplanetary transfer planner in it using a Kerboscript port of Alexmoon's Transfer Window Planner and encountered processing times of... about 15 minutes.

Then I decided it was finally time to play with kRPC.  Should be easy enough, I write Python professionally anyways.  But then it occurred to me:

• I have a ton of existing kOS code that already works and I'd spend some time just rewriting it.
• kOS has some advantages over kRPC when it comes to craft control -- namely latency.

I've started writing my first-ever KSP plugin to try to implement a bridge between the two.  The timing couldn't be better either -- I discovered that kOS's external API support is so new, it's not even in a released version of kOS yet.  So I've compiled that build of kOS myself from source (which hopefully won't be a requirement by the time I release, I hear kOS 1.0.0 is right around the corner).

Note: The below developer build will NOT work without (not-yet-released) kOS 1.0.0.  It should work with the latest release candidate in the kOS thread.

Now includes client libraries (with thanks to @djungelorm for assistance)

A beta build is available at https://github.com/dewiniaid/ksp-kipc/releases/

API documentation for this release can be located at https://github.com/dewiniaid/ksp-kipc/blob/v0.2.0-beta/doc/index.md

Licensed GPLv3.

Old history is spoilered below.

Spoiler

Design/Architecture:

Most of the below is no longer accurate as I'm going to be more or less mirroring kOS's upcoming communications API 

My original idea was a sort of message queue.  I'm foregoing that for now due to possible memory management concerns -- if a kOS script decided to send its position every frame and nothing was listening to the other end it'd get messy pretty quickly and I don't want to deal with the idea of buffers too much.  So what I'm using instead (unless I get better suggestions) is somewhat similar, minus the queue:

• Each kOS Processor has a single "inbound" and "outbound" message.
• "Acknowledging" a message clears it out.
• From the kOS side, there are functions to retrieve the current processor's inbound message ("Receiving") if any, and set ("Sending") or cancel its outbound message
• From both kOS and kRPC, there are API functions that set or cancel any processor's inbound message ("Sending" to that processor) or retrieve its current outbound message ("Receiving" from that processor)
• The API won't let you send a message to a processor that already has a message waiting.  On the kOS side, it also won't let you send a message if your existing sent message is waiting.  This is to prevent a message from being overwritten before its seen by its intended destination.  Methods exist to cancel messages in both of these cases.
• Since mixing kOS and KRPC with multiple message senders/etc introduces the potential of race conditions and other pitfalls of asynchronous coding, there are API methods to do things like "Cancel a processor's incoming message, but only if I'm the sender" or "Acknowledge a message, but only if it's this message ID."

Message Data:

As far as I can tell (maybe a KRPC developer (paging @djungelorm ) can chime in?), there isn't a good way to accommodate the transfer of variably-typed data over KRPC.  I considered a standard serialization format (i.e. just sending JSON) but that would end up with its own problems (like needing to converting a reference to a vessel back to an actual vessel).  So instead:

• A message always has -- as its sole data element -- a collection of some form.
• Collections all have the same representation, but they have a property to determine what kind of collection they are intended to represent (list, dict, queue or stack).
• Collections have a property to return a list of their member argument types and a bunch of methods to retrieve the Nth argument (for reading a message) or append an argument of a specified type (for writing a message).
• This unfortunately means a *lot* of method calls to retrieve or send all of the arguments of a message, particularly if the 

I considered -- as an alternative -- having a property for 'all of the string parameters', one for 'all of the integer parameters', etc.  But that seems like bad design (even though it'd mean for lower protocol overhead).

I suppose I could go back to the JSON serialization route, but add metadata that states "These fields are actually vessel IDs", etc. to the result and a means to resolve them.  Hmm...  (Update: that's exactly what I'm doing).

 

Edited July 23, 2016 by dewin
0.2.0

[komodo]

Very neat! I have been wanting to play more with both of these addons, as, as you say, they each do different things very well.

I also can understand the difficulty with interfacing the two different communication methods, so that is very clever. I don't have any pointed feedback, as I haven't actually had a chance to use it as of yet, though ><

[dewin]

I opted to do JSON serialization anyways with some customization; the greater part of the last couple days has been trying to get a JSON library to actually work with KSP.

But now, I've got this:

PRINT "--Running kIPC Test Script--".
PRINT "Testing basic type serialization.".
FOR value IN LIST(True, False, 42, 1.234567890987654321, 1e-17, 1e+17, "A String") {
	PRINT value + " => " + addons:kipc:serialize(value).
}


PRINT "Testing reference type serialization.".
FOR value IN LIST(ship, core:part, body) {
	PRINT value + " => " + addons:kipc:serialize(value).
}

PRINT "Testing collections.".
LOCAL s IS STACK().
s:push(1).
s:push(2).
s:push(3).
s:push(4).

FOR value IN LIST(
	LIST(1,2,3),
	s, 
	QUEUE(1,2,3,4,5),
	LEXICON("only_strings", 1, "more data", 2),
	LEXICON("mixed", 1, 2, 3),
	LEXICON(9,8,7,6),
	LIST(LIST(1,2,3),LEXICON("foo", "bar"))
) {
	PRINT addons:kipc:serialize(value).
}

PRINT "Testing references in collections.".
LOCAL foo IS LIST("foolist").
LOCAL bar IS LIST("barlist", foo).

PRINT addons:kipc:serialize(LIST(foo, bar)).
PRINT addons:kipc:serialize(LIST(bar, foo)).

PRINT "Possibly crashing momentarily...".
WAIT 5.
foo:add(bar).

PRINT addons:kipc:serialize(LIST(foo, bar)).
PRINT addons:kipc:serialize(LIST(bar, foo)).

Which produces this:

--Running kIPC Test Script--
Testing basic type serialization.
True => true
False => false
42 => 42
1.23456789098765 => 1.2345678909876543
1E-17 => 1E-17
1E+17 => 1E+17
A String => "A String"
Testing reference type serialization.
SHIP("Test Rocket") => {"type":"vessel","data":"b422057f-e7de-4c1e-806f-6907e19f6e9a"}
PART(kOSMachine1m,uid=4294806512) => {"type":"part","data":187191025}
BODY("Kerbin") => {"type":"body","data":1}
Testing collections.
{"type":"list","data":[1,2,3]}
{"type":"stack","data":[1,2,3,4]}
{"type":"queue","data":[1,2,3,4,5]}
{"type":"dict","data":{"only_strings":1,"more data":2},"keys":[],"values":[],"sensitive":false}
{"type":"dict","data":{"mixed":1},"keys":[2],"values":[3],"sensitive":false}
{"type":"dict","data":{},"keys":[9,7],"values":[8,6],"sensitive":false}
{"type":"list","data":[{"type":"list","data":[1,2,3]},{"type":"dict","data":{"foo":"bar"},"keys":[],"values":[],"sensitive":false}]}
Testing references in collections.
{"type":"list","data":[{"type":"list","data":["foolist"],"ref":2},{"type":"list","data":["barlist",{"type":"reference","data":2}]}]}
{"type":"list","data":[{"type":"list","data":["barlist",{"type":"list","data":["foolist"],"ref":3}]},{"type":"reference","data":3}]}
Possibly crashing momentarily...
{"type":"list","data":[{"type":"list","data":["foolist",{"type":"list","data":["barlist",{"type":"reference","data":2}],"ref":3}],"ref":2},{"type":"reference","data":3}]}
{"type":"list","data":[{"type":"list","data":["barlist",{"type":"list","data":["foolist",{"type":"reference","data":2}],"ref":3}],"ref":2},{"type":"reference","data":3}]}
Program ended.

Notable here is: All of the more complicated types include enough metadata to reconstruct them, and... a list that contains itself can be properly represented without breaking the game due to recursion.  (Fun fact: PRINTing that same list instantly crashes KSP).

 

[dewin]

Over the last couple days I've been rewriting the serialization code (and implementing the deserialization code) to be a bit more modular and generally less terrible.

I've also learned that the upcoming probably-tomorrow kOS release includes a inter-processor/inter-vessel communication bit, so I'm scrapping a giant portion of my plugin and rewriting it to solely be interoperability between kOS and kRPC... so all of the documentation that's listed here is most likely now wrong.  On the bright side, this should drastically reduce development time.

Edited July 8, 2016 by dewin

[dewin]

Super-early dev build is now available in the first post.

If you ignore the bold text and complain that it doesn't work, you're in the wrong forum

[dewin]

Version 0.1.1-dev2 is now available; links in the first post.

0.1.1-dev2 (2016-07-12)

• Added KIPC.GetMessages and KIPC.CountMessages for better control and information about the message queue.
• Added KIPC.ResolveBodies and KIPC.ResolveVessels to handle multiple bodies/vessels at once.
• Added KIPC.GetProcessor to retrieve the kOSProcessor of a single part (compare to KIPC.GetProcessors which receives all processors on a given vesse)
• Added KIPC.GetPartsTagged to find parts with a given kOSNameTag.

[dewin]

Version 0.2.0 beta is now available; links in the OP.

• Added client libraries for C#, C++ and Java.
• Added preliminary support for CKAN and KSP-AVC. This release does not include MiniAVC and does not use the internet to check for version updates, but other providers of MiniAVC/KSP-AVC might

Barring any major issues, this is the same version that I'll have in the Release thread -- I'm just delaying that until kOS's final release.

[gowthamn]

Very nice explanation of use cases in git. And KOS version 1 is out. Probably you can update the 1st post? Looking forward to use it.

Edited September 8, 2016 by gowthamn