djungelorm

[1.4.x,1.3.x,1.2.2] kRPC: Remote Procedure Call Server (v0.4.7, 27th July 2018)

Recommended Posts

@ djungelorm: happy about the changes, real good improvements already, and good plans for further.

And, even more important, you are telling users what to expect from kRPC, so they can (at least in theory) take their precautions if trying to use it outside the protection offered by their home networks.

Share this post


Link to post
Share on other sites
The main use case I had in mind for the plugin was for use just on the local machine. However, TCP sockets happened to be the simplest cross-platform and language-agnostic way of doing interprocess communication - but appear to have opened a can of worms! Ideally (for local use) it should use something like unix pipes (for performance reasons as well as security!) although I'm not sure what the Windows equivalent would be. Anyone know?

i want to say named pipes is what they go by on windows. its pretty much the same thing as whats used on *nix. good luck finding a library that works on all platforms though. tcp/ip beats it out in this regard.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365590%28v=vs.85%29.aspx

ive tried to use them before for other projects but i was never satisfied with the results. its just so much easier to use the loopback address (127.0.0.1) under tcp/ip. i kinda like the idea of being able to use network assets if i want to (like being able to talk to a tablet or a raspberry pi). you might want to keep the loopback as the default address though. it should be pretty secure since it will never leave the local machine.

Edited by Nuke

Share this post


Link to post
Share on other sites

@Dio>

Maybe I misinterpreted the order you had listed as a "list of things that should be in" versus "priorities of what should be considered".

The lack of authentication AND having a window come up and say "Hey, is this connection from this other device OK?" was NEVER thought of to be taken off the table for this mod, but, there are exploits that get around authentication entirely, depending on the program.

The other thing is, at a hardware level, encryption might not be done at the NIC level, but at the CPU level. Some NICs are extremely dumb devices, meaning, they receive information from the CPU, which determines everything about the package (Except headers and such) and out it goes, which is where my concern is about the encryption. The CPU still needs to handle it, and on a multi-proc, multi-threaded CPU system not a big deal. But only if it is outside of KSP. There are some NICs where you tell the driver "Use this key with this protocol, and everything will be zen" and the CPU just pumps the raw data to the NIC and the NIC handles everything. Even if a separate application were to be running that would do the encryption/decryption and keep network communication limited to LOCALHOST for unencrypted traffic, while, anything out on the network goes through encryption, OK, I'm fine with that. But it'd need to be a completely different program, outside of KSP.

@Nuke>

I THOUGHT about named pipes as well, but the drawback on that is, as you mentioned, but its not all that fun to play with. Windows also has an inter-process communication protocol as well, that isn't named pipes, works on a client/server kind of concept, sits 100% local to the machine, but for the life of me, I just can't freak'n think of it. I was playing with it it back in the Win 9x days and after my experience with it, I gave up with it.

LOCALHOST for default listening should be the way to go. With the third-party app I mentioned above if you want to be security conscious, go for it. But if you want the plain text, the option to listen to a physical NIC on a real IP, then it should be allowed as well.

@All> I'm glad no one has battered me with "What the hell is up with that long post?" or I get that post reported... At least people took notice. YAY!

Share this post


Link to post
Share on other sites
... but, there are exploits that get around authentication entirely, depending on the program.

...

... But it'd need to be a completely different program, outside of KSP.

Agreed. Very good points.

Exploits certainly exist, and even more measures than suggested til now would be needed to further limit them. Though, I am not considering system-wide vulnerabilities that should not be this plugin business to amend. But certainly, if you spot any other measure that would significantly improve security with this plugin, please tell.

Understood what is your point with encryption load. Yes, there may be machines running KSP that only have dumb NICs and not multithreading. If I'm right, you're suggesting a different program, outside KSP, handling encryption and traffic to/from kRPC, could be a better solution. It's kind of an interesting sys architecture that should work. I suspect something useful for this purpose may already exist, need not be developed; but you may already have something up your mind?

Share this post


Link to post
Share on other sites
@Nuke>

I THOUGHT about named pipes as well, but the drawback on that is, as you mentioned, but its not all that fun to play with. Windows also has an inter-process communication protocol as well, that isn't named pipes, works on a client/server kind of concept, sits 100% local to the machine, but for the life of me, I just can't freak'n think of it. I was playing with it it back in the Win 9x days and after my experience with it, I gave up with it.

LOCALHOST for default listening should be the way to go. With the third-party app I mentioned above if you want to be security conscious, go for it. But if you want the plain text, the option to listen to a physical NIC on a real IP, then it should be allowed as well.

msdn's list of the means of interprocess communication on windows.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365574%28v=vs.85%29.aspx#base.using_com_for_ipc

problem is all this stuff only works on windows. there may be a *nix equivalent for some of those, but as i said earlier good luck finding a cross platform library. im 100% in support of using tcp/ip and defaulting to localhost.

Share this post


Link to post
Share on other sites

Thanks for the pointer Nuke. Also, handling multiple clients with TCP/IP is a breeze compared to some of the other options, so I'll stick with it.

Share this post


Link to post
Share on other sites

Ahhhh yeah... Dynamic Data Exchange. DDE. That was what I was playing with. Must have blocked that outta my memory or something. ;)

But yes, this is a Windows Only thing. The only other thing that could possibly work on *Nix systems is to create a FIFO file and stream the data in and out. You'd probably need two files though, one for client->Server and another Server->Client. I don't think FIFOs are two way capable.

Share this post


Link to post
Share on other sites

Looking good so far, nice work. Having tried several times to get IronPython running in KSP, I feel your pain and appreciate your solution.

One thing to think about for the far future:

(0) Add RPCs for creating in-game GUIs; imagine the possibilities: Display KER-like readouts, or even send commands to the script via GUI buttons. The latter might even incorporate RT2's control lag system; I _think_ I heard Cilph mention that he's working on an API for other mods to use, I suggest talking to him on #kspmodders.

One thing for the slightly nearer future:

(1) For basic autopiloting (especially, pointing prograde/retrograde/maneuver-grade/etc), I'd recommend looking into how Mechjeb does that, and probably even out-right copying most of the code; no need to re-invent the wheel. In fact, both RT2 and kOS simply use Mechjeb's code. Your MIT license as opposed to MechJeb's GPL might pose a problem though... If I were You I'd simply switch over to GPL anyways.

And... a few more immediately-relevant things, mostly regarding your distribution:

(2) Your shebang line in the example python script is #!/usr/bin/env python. Some more modern distros such as Arch Linux already use python3 as standard python interpreter, so many scripts that use this shebang line break. I'd heavily recommend writing #!/usr/bin/env python2 instead.

(3) I'd appreciate it if you'd ship the auto-generated proto.py file... It would save the trouble of having to download protobuf-csharp-port and figure out/run the Makefile.

(4) You should probably publish more detailed build instructions... I'm too lazy to figure out the Makefile right now, but simply changing KSP_DIR to my path in the Makefile didn't fix the "can't find Assembly-CSharp" errors. Probably I'll have to copy the whole project into my ~/kerb/GameData folder, but that shouldn't really be neccesary.

(5) I managed to break the Control class by using it in an interactive python interpreter and pressing ^C while it was executing a command... Though I admit that adding KeyboardInterrupt exception handlers everywhere wouldn't be worth it.

(6) You'll want to look into toolbar integration for the UI.

Edit after reading the full thread:

Network sockets are perfectly fine IMHO, just limit to localhost or ask the user on connection attempt (as you're already doing) will be enough security. Don't artificially limit yourself to platform-specific solutions.

FIFOs are 1-way indeed. On POSIX you'd want to use UNIX sockets (which work and are used exactly like network sockets, but instead of connecting to an IP/Port you open a file).

P.S. wow this post has evolved into a wall of text over a few edits, and I guess this P.S. message isn't making it better :)

P.P.S. nice to meet a fellow Linux developer, who shows the boldness of delivering a Makefile with his C# project.

Edited by mic_e

Share this post


Link to post
Share on other sites

(0) Add RPCs for creating in-game GUIs...

Nice idea! I had in mind that the you'd just output stuff to the terminal or something like an R script to draw graphs, but this would be a great inclusion too. Displaying data to a custom window would be easy. I'm not so sure about sending commands to client scripts - the architecture is designed for the request/response to happen in one direction, so isn't set up for clients to receive commands.

There are probably workarounds though. Its not pretty, but we could have the clients actively polling the server for new commands.

(1) For basic autopiloting (especially, pointing prograde/retrograde/maneuver-grade/etc), I'd recommend looking into how Mechjeb does that...

I agree - it would be a lot of wasted effort coding an autopilot that MechJeb already does very well. krpc.dll should provide an API so that other plugins can expose functionality as RPC calls, e.g. autopilot commands from MechJeb. I haven't found the time to look into this in any detail yet though.

And for the more immediate stuff:

(2) Fixed!

(3) + (4) I've updated the readmes with build instructions, and included dependencies in the repository - so it should be much easier to figure out now!

(5) Fun. I need to revisit the python client code - it doesn't do clean disconnection from the server yet.

(6) Yep! On the todo list.

P.P.S. nice to meet a fellow Linux developer, who shows the boldness of delivering a Makefile with his C# project.

:) I'm doing all my development in MonoDevelop on Ubuntu, and it's nice having a makefile so I can just type "make ksp" into a terminal and have it deploy the plugin to gamedata, launch ksp and switch to an active vessel. And it handles all the jiggery pokery of compiling protocol buffer definitions...

Share this post


Link to post
Share on other sites

I might try to automatically export the public KSP API through the RPC server, but this needs more research...

It looks like you are sending the field values as part of the response to the client. If those values change on the server, the client wouldn't know. So what about on the server keeping a map from object IDs (which you make up) to the actual objects. The client can have a peer object, and when you call a method or get/set a field on the client peer object, it sends a message to the server to actually perform the operation. All state is maintained on the server, so there's no chance of the client thinking a field is one value but actually it has been changed on the server.

Then it might be possible to write a script or some C# code to create all the client peer object classes instead of creating them manually.

I'm working on some proof of concept code if you're interested.

Btw, I'm not a C# developer, nor do I have much experience with networking code, so if any of that sounds like gibberish just let me know :)

Share this post


Link to post
Share on other sites
I'm not so sure about sending commands to client scripts - the architecture is designed for the request/response to happen in one direction, so isn't set up for clients to receive commands.

(0) My basic idea: Use a multi-threaded client. One thread does a blocking RPC for waits for user input, and the other does its usual stuff. That way, you can let the script writer worry about this stuff. Disclaimer: I didn't really look at your C# or Python code, so I don't know whether this would be possible/easy. That being said, polling is something truly horrible, and you should avoid it at all costs.

krpc.dll should provide an API so that other plugins can expose functionality as RPC calls

That's a great idea, but

(1.0) I think you'd need quite the critical userbase to get the authors of the big plugins like Mechjeb to expose their functionality via it.

(1.1) There's many other plugins (Selene, kOS, Jebnix, ...) that probably also want to an API like this, so the API should probably be its own plugin, and I imagine it would be a rather large project.

(1.2) I'd prefer not having to install Mechjeb for my main save (Yes, I'm one of those guys who refuse to use Mechjeb, and among the 'hey I want to write my own autopilot' faction that will basically be your userbase, I think a lot of people are)

In conclusion, I'd really recommend stealing Mechjebs code and copying it into the core plugin - This is basic functionality, and without it, there's no way I'm gonna use kRPC in my main save.

Ideally the game should simply expose a method to change the heading that the SAS is locked to, to prevent every mod developer from having to entirely re-implement SAS.

(stuff)

I think you'll want to look into System.Reflection on the C# side. Overloading the getter/setter methods on the python side sounds like a good idea.

Edited by mic_e

Share this post


Link to post
Share on other sites
It looks like you are sending the field values as part of the response to the client. If those values change on the server, the client wouldn't know...

Proxy objects (i.e. objects whose properties auto-update from the server) is definitely something I want to include. I think it can be implemented solely on the client side though (in the python code) which would allow the server to be kept stateless which keeps things simple (ignoring state being held by the game of course!). Although if we do do it on the client side we might get some fun issues when there are multiple clients!

Then it might be possible to write a script or some C# code to create all the client peer object classes instead of creating them manually.

Yep definately. The python client library currently does this for RPC calls. There is an RPC call that lists all the available methods and their parameter/return types. The python uses this info to create a bunch of objects representing the RPC calls, to make them appear like normal python methods. We could do something similar to create auto-create objects for the parameter/return values.

I'm working on some proof of concept code if you're interested.

Sure! Be happy to take a look at what you come up with.

Share this post


Link to post
Share on other sites
(0) My basic idea: Use a multi-threaded client ... That being said, polling is something truly horrible, and you should avoid it at all costs.

Yeah that could work. The server would need to be extended slightly to not hang the game when an RPC is blocked, but I don't think that'd be hard to do.

Agreed that we should avoid polling! This reminds me of an idea I had where you use an RPC call to request a stream of e.g. flight data from the server. Then you don't have to keep requesting it from the server. Would save the overhead of making repeated calls to poll for it. But that idea can wait...

(1.0) I think you'd need quite the critical userbase to get the authors of the big plugins like Mechjeb to expose their functionality via it.

(1.1) There's many other plugins (Selene, kOS, Jebnix, ...) that probably also want to an API like this, so the API should probably be its own plugin, and I imagine it would be a rather large project.

Yeah I didn't expect that the big plugins would jump to modify themselves to include this stuff! But it might be possible to write a thin wrapper over plugins like mechjeb to expose their functionality through the RPC server (e.g. include a krpc-mechjeb.dll with kRPC). I had a quick look at the code for mechjeb and a lot of its functionality is exposed as public classes. This way we can expose features from the big plugins without needing to modify them.

(1.2) I'd prefer not having to install Mechjeb for my main save (Yes, I'm one of those guys who refuse to use Mechjeb, and among the 'hey I want to write my own autopilot' faction that will basically be your userbase, I think a lot of people are)

Totally agree, MechJeb should not be a prerequisite for kRPC. I don't want to use it in my scripts either, but it might be nice to include so that people starting out writing autopilot scripts can use bits of MechJeb, then slowly replace them with their own code (that uses just basic autopilot such as setting headings).

kRPC should definately be useful in its own right - without needing to rely on having other plugins. I think a good plan would be to ship 2 dlls: krpc.dll provides just the RPC server and an API for adding functionality to it (which I've just implemented - I'll add details to the OP), and krpc-services.dll provides a set of useful functionality (basic autopiloting, getting flight data etc.)

That way kRPC will be useful on its own, and if this stuff takes off and other plugins want expose their functionality via it they could just include krpc.dll

Share this post


Link to post
Share on other sites

Sounds like a pretty good concept; it'd also solve some licensing issues for the Smart SAS code - simply put that in its own service provider DLL and GPL only that DLL.

Also, the Makefile still doesn't work, it didn't find the references: I needed to create a symlink ln -s ~/kerb/KSP_Data/Managed lib/ksp

I think your Makefile could auto-create that symlink depending on the KSP_DIR variable.

test -d lib/ksp || ln -s "$(KSP_DIR)/KSP_Data/Managed"

Also, I needed to install inkscape (you should add that to your apt-get example command, or instead use something more common/lightweight like imagemagick (I have no idea whether imagemagick supports svgs, but in my experience it can do absolutely everything)

Apart from that, I really like your build system... the most fun so far I've had building a KSP mod.

Oh, and I also suggest a test in the install recipe:

test -d "$(KSP_DIR)/GameData"

Ooooh... looking at example.py, the actual project has evolved nicely... but it gave me a python exception after clicking 'Allow': http://pastebin.com/EG8df23Z

After hacking around that issue, there also seems to be a problem with setting throttle... which I'm not motivated to look into right now. Ideally, the python-side interface would allow something like

Controls.throttle *= 0.95

and

Controls.SAS = True

on the .NET side, service providers could hint that something is a getter/setter method pair for a field that should be called 'SAS'.

on the python side, you could use the built-in 'property'.

Edited by mic_e

Share this post


Link to post
Share on other sites

Also, the Makefile still doesn't work...

...

Apart from that, I really like your build system... the most fun so far I've had building a KSP mod.

Thanks! I was worried my build scripts were a little arcane... I've fixed the errors you found, and incorporated your suggestions into the build script. A quick googling suggests imagemagick's handling of svg files isn't great, so I'll stick with inkscape for now. I've added it to the readme along with install instructions.

Ooooh... looking at example.py, the actual project has evolved nicely... but it gave me a python exception after clicking 'Allow': http://pastebin.com/EG8df23Z

The client code on the master branch is a bit broken at the moment (as I added the ability to pass value types - int, float, string etc. - as parameters to procedures in the server code, but the client code hasn't been updated to use it yet).

I'm, currently rewriting the client code to make it more robust and work with this new feature (the previous implementation was just a quick and dirty first attempt anyway). I'll let you know when I've merged the changes back into master. (If your interested it's on this branch: https://github.com/djungelorm/krpc/tree/feature/python-client).

Ideally, the python-side interface would allow something like

Controls.throttle *= 0.95

and

Controls.SAS = True

on the .NET side, service providers could hint that something is a getter/setter method pair for a field that should be called 'SAS'.

on the python side, you could use the built-in 'property'.

Yeah that would be a nice feature! I'll add it to the todo list.

Share this post


Link to post
Share on other sites

I've finished my rewrite of the client code, so the example script should work properly now. I'll try to not make any breaking changes to the master branch on github in future...

I've also updated the OP with a download link to the latest built binaries.

Share this post


Link to post
Share on other sites

So, let me compile every info I have found here...

This mod allows us to remote control KSP ships from outside the game, using every application that supports TCP serialization. It accepts multiple connections. In this sense, its like Orb::Connect. So, the first thing that comes to my head is tablet based RasterPropMonitor

The second thing that would be very interesting is an external map view program. The program will also have maneuver editor and better maneuver node, with MechJeb autopilot and KSPTOT-like trajectory optimization. Using this, we could automate tedious tasks like periapsis kicks easier

Share this post


Link to post
Share on other sites

Following the progress in the git repo, this seems to be coming along really nicely.

In fact, I think it has a stable enough interface to write 3rd-party service providers. I'm gonna try and implement a heading service provider, which reports and holds different kinds of headings in surface/orbit/target/... mode, and report back any issues.

One thing I already know I want: Support for enums. Think ksp.heading.mode = ksp.heading.SURFACE instead of ksp.heading.mode = 3.

Edit: Another thing that would be awesome: Support for default arguments (specify default arguments in C#, and they get transferred to Python)

Edited by mic_e

Share this post


Link to post
Share on other sites
Following the progress in the git repo, this seems to be coming along really nicely.

Thanks :) I'm quite pleased with the way the services api has turned out.

... I'm gonna try and implement a heading service provider, which reports and holds different kinds of headings in surface/orbit/target/... mode, and report back any issues.

That'd be excellent, thanks. I have a design document I wrote a while ago for "essential" services such as this, and have put it on the wiki here https://github.com/djungelorm/krpc/wiki/SpaceCenter-Service

You might also want to take a look at this branch on github, where I've implemented bits and pieces of it https://github.com/djungelorm/krpc/tree/feature/basic-services

Enums are supported by protocol buffers, so it should be possible to include them. I'll have a look into default arguments too, I think that should be an easy to implement.

Some other things kRPC can't do are calling constructors and function overloading. I think constructors are worth supporting, but am not so sure about overloading. Python for example doesn't have overloading, so I'm not sure how it'd work on that end.

Edit: Would be nice to add support for named arguments too.

Edit again: Named arguments and optional parameters have now been implemented!

Edited by djungelorm

Share this post


Link to post
Share on other sites

My Autopilot service is basically done; the important parts, found in ReferenceRotation.cs and AutoPilotAddon.cs, are taken and modified from RT2, which took the relevant code form kOS, which took and modified the relevant code from MJ2.

If it was a larger project, I'd GPL it, but you (and everyone else) can have it and do with it whatever you want, under the condition that you credit me. Note, however, that the MJ2/kOS/RT2 parts are GPL'd.

An example session would be:

ksp=krpc.connect()

ksp.Autopilot.Engage("surface", 90, 45) #set east heading at 45 deg pitch

while ksp.Autopilot.ErrorYP > 1: #while the error in Yaw/Pitch direction is > 1deg

time.sleep(1)

ksp.Autopilot.Engage("orbitv") #set heading to orbital velocity vector

ksp.Heading.Mode = "surface"

while True:

print((ksp.Heading.Yaw, ksp.Heading.Pitch, ksp.Heading.Roll))

Note that the Autopilot is not without flaws yet: When docking two craft it will freak out reproducably. However, I'm currently not motivated to debug that as I have a lot of other stuff to do.

I've uploaded the source files here.

All in all, using your API was a pleasant experience; would develop against again.

Oh, and an other request for the python client code: Automatically convert property and function names to lower case or at least camel case, to match python code standards.

Also, I've no idea how docstrings work in C#, but if those would somehow appear in python help(), that would be pretty neat, too.

Edited by mic_e

Share this post


Link to post
Share on other sites

How do I install this? The content inside the gamedata folder go inside KSP gamedata folder, but how about python and schema?

Share this post


Link to post
Share on other sites
How do I install this? The content inside the gamedata folder go inside KSP gamedata folder, but how about python and schema?

You're correct about the game data folder. To install the python module, you just need to add the path to the python folder to your module search path (e.g. PYTHONPATH), then you can do 'import krpc' from a python script. I plan to implement an installer (likely using python pip) to make installing it simpler. The top-level schema directory isn't needed - not sure why I included it!

Share this post


Link to post
Share on other sites
My Autopilot service is basically done...

Cool! I'll take a closer look at the code when I have some free time. I'm tempted to make the kRPCServices dll LGPL so I can reuse bits of MJ/RT2 etc.

Oh, and an other request for the python client code: Automatically convert property and function names to lower case or at least camel case, to match python code standards.

Good idea. Although I'm slightly worried that it'd be confusing having different things for the C# and python names. Maybe provide both?

Also, I've no idea how docstrings work in C#, but if those would somehow appear in python help(), that would be pretty neat, too.

The documentation is stripped out by the compiler, so we can't get hold of it via reflection. I think the only way to do this would be to generate an XML documentation file from the code (which most C# compilers can do for you) and distribute that with the dll.

Share this post


Link to post
Share on other sites

hi dj (and all the others her),

found KRPC and its exactly what i wanted to start for some time to get some sort of remote control and using additional screenspace (sucks to sit in front of multiple screens and having all that plugin information stuff clogging the view). luckily u did this, thx for that.

i try to build on KRPC to first get a simple webserver showing flight information. my later plans would include scriptable control (like KOS but thru a web app) etc, sky is the limit ...

my idea is to also use other devices like laptop displays, tablets etc for additional views on the KSP instance ... so a webapp is it.

i will use groovy/grails and GWT for it as this is my bread & butter and it scales nicely.

i successfull protoc'd and implemented an equivalent to python/krpc.py in java, will post after some cleanup.

i've got one problem thou which i found while playin around with the stuff:

if i just use your original stuff (KRPC on one the KSP host and example.py w. changed address on my development machine) to remote control KSP on another machine the KSP instance gets extremly laggy (around one frame per second and worse).

i just added a long sleep after the connect and it seems the connect alone puts KSP down. disconnecting from either side will bring KSP to senses again.

KSP.log shows nothing unusual.

KSP runs flawless on the machine otherwise. both machines are linux, sorry no windows here.

i will test further and try to find whats the cause but maybe you have an idea ?!?

wbr,tja...

Share this post


Link to post
Share on other sites

hi again,

after some further testing the picture changed a bit.

the second machine i used for testing has some older hardware with just enuf juice to do KSP @ 1080p/30FPS+ - which brought the issue to light as the performance on this machine is horrible as soon as a client connects via KRPC.

but the issue is there on my faster development host too, regardless of "remote" or "local" KRPC. i didnt pay attention to the FPS drop in my past experiments but i measured drops to 5FPS+ from my usual smooth 60+.

i even installed windows 7 and KSP on a spare disk but the problem is there in windows too.

i wonder, is nobody out there experiencing this ? everyone plays KSP on perfect HW ?!? :confused:

Share this post


Link to post
Share on other sites
This thread is quite old. Please consider starting a new thread rather than reviving this one.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.