[WIP] Jebnix - A kOS Alternative

[woodywood245]

At the same time, given that this is "Jebnix," I think it might be fun and helpful to have a handful of *nix-like utilities to play with if people want to.

[nanobot]

What if the "shell" just compiled the entered lines on-the-fly and uses the same backend? It should be fast enough. To me, dropping the interactive mode (which I would really like to have) for this reason would be overkill.

[woodywood245]

What if the "shell" just compiled the entered lines on-the-fly and uses the same backend? It should be fast enough. To me, dropping the interactive mode (which I would really like to have) for this reason would be overkill.

The one major issue with that is that the compiler and the program counter would have to talk to each other, just like an interpreter does. It wouldn't simplify things very much. That's pretty much what I'm doing now.

Consider this basic example:

1   set x to 0.
2   while x < 5
3   {
4       set x to x + 1.
5       print x.
6   }
7   print "Finished!".

Consider what is happening when it's translated into a pseudoassembly, and then executed without the compiler talking to the program counter.

// When the whole thing is compiled and run
// Assembly:
main:
    mov x, 0
    jmp while_loop
    print "Finished!"

while_loop:
   add x, 1
   print x
   cmp x, 5
   jl while_loop
   ret

// Output:
1
2
3
4
5
Finished!

// Compiling one line at a time. The <-- indicates where the program counter is.
// When line 1 is compiled alone:
main:
    mov x, 0   <--

// Then lines 2 and 3 are compiled:
main:
    mov x, 0

while_loop:
    // this is where the block should go
    cmp x, 5          <--   // this will begin an unending loop
    jl while_loop

// Then line 4 is compiled:
main:
    mov x, 0

while_loop:
    add x, 1       <-- // on the next pass, the counter winds up here. The loop will end within 17 cycles, which at even at 100 Hz is less than a 5th of a second. 
    cmp x, 5
    jl while_loop

// Then lines 5 and 6 are compiled
main:
    mov x, 0

while_loop:
    add x, 1
    print x
    cmp x, 5
    jl while_loop
                       <--  // outside the loop, waiting for an instruction

// Line 7 is compiled
main:
    mov x, 0
    jmp while_loop
    print "Finished!"

while_loop:
   add x, 1
   print x
   cmp x, 5
   jl while_loop
   ret               <-- // this is the new instruction. the call stack is empty, however, resulting in an underflow

The way to fix this would be to halt execution whenever a control block is about to be written by the user, then compiling the block when the user closes the block. This, however, presents its own set of problems. Consider if I'd left out the period on line 4. Jebnix wouldn't let the user know about the syntax error until compilation. By that time, they've already written the line, and it may be buried deep inside a long block of code which now has to be rewritten from scratch.

I'll admit that it's late, and I may not be able to think of an obvious solution to this problem, but this is how things look to me at the moment.

[diomedea]

Are you planning to include debugging tools with the utilities for Jebnix? I am totally in favor of the compiled version, but need to debug. At least a step/trace with an output display.

I admit I don't see a real difference, from a coder perspective, among an interpreter and a JIT compiler/launcher with debug options on (bear with me, I don't write compilers).

But some hurdles you wrote about (e.g. the program counter) are certainly there in both an interpreter and a debugger.

[woodywood245]

Are you planning to include debugging tools with the utilities for Jebnix? I am totally in favor of the compiled version, but need to debug. At least a step/trace with an output display.

I admit I don't see a real difference, from a coder perspective, among an interpreter and a JIT compiler/launcher with debug options on (bear with me, I don't write compilers).

But some hurdles you wrote about (e.g. the program counter) are certainly there in both an interpreter and a debugger.

I will be including a debugger, which will include stepping, stack trace, and variable values, as well as the values of the evaluation stack, all with symbols (the virtual machine will be a stack machine). You will also be able to do a memory dump.

I will also be including the console debugger that I use to test everything, which is simply a Windows Console application that emulates the machine in the game, except without a rocket attached to it.

[nanobot]

I see what you meant, woodywood245. I agree with you that there should not a machine code generation invoked every time the input gets partially parsed, since this can change the semantics of a program in a serious way.

What about this:

- The interpreter expects user input line for line and tries to parse this line, starting with an empty parsing stack.

- The input is tried to be parsed into a <statement> node. If the parser needs more tokens, print a symbol like > that prompts the user for another line (easiest way to do this is a blocking readNextToken() method)

- If the input is a valid statement, execute it.

This can be generalized for the whole language grammar. Most important for this is to specify what exactly is an executable piece of code.

Assuming you have a simple formal grammar like this:

<statement> -> <expr> := <expr>

<statement> -> while (<expr>) <statement>

<statement> -> { <statement-list> }

<statement-list> -> ε | <statement><statement-list>

Then a <statement>, defined like above, is the minimal executable piece of code.

I will demonstrate a parsing process using an interactive shell.

The parser should start with expecting a <statement> node.

The <statement> node then requires reading a { or a "while" or an expression.

Let the input of the user be:

> while (

The parser sees the "while" keyword and then knows it has to use production 2. (I think that a recursive descent parser is a good option for this kind of user input interpretation).

While resolving the right hand side of production 2, it consumes the tokens "while" and "(" and then requires an <expr>. However, since the sequence of remaining tokens is empty, all the parser will do is again printing a prompt:

>

Assume the user will now enter the expression and closing ")" :

> (3 < 4) == !false )

Then the parser can parse the <expression> node and consume the ) token, requiring a <statement> (but knowing that this one is "one level deeper" than the original one).

After this statement was also parsed, the production

<statement> -> while (<expr>) <statement>

is finished and the parser returns to the original "<statement>" call, then producing the required machine code.

In this grammar, also a function definition could be modeled as some kind of statement, but one with no side effect or output except that it internally stores the new function.

I think I will now first study the parser that you use, I worry everything I wrote is already obvious for you.

EDIT

Ok, I see that at least on github, the Compiler and CompiledScript files are empty.

So if you want, I can give you some directions on how a multi-stage compiler (tokenizer -> syntax parser -> assembly code generator) could be designed.

Edited April 28, 2014 by nanobot

[Camacha]

I understand the problem For me the live mode/interactive mode is helpful for reasons explained, but it is not something critical. If there might be another way of quickly testing code without having to go back and forth, that should work fine too. I would just hope to avoid a situation where you need to go through a number of steps each try, as that is something that would drive me a bit crazy in kOS sometimes. Of course, the lack of accurate error messages and often sparse or ambivalent documentation amplified the problem a lot.

So no, I would not be terribly disappointed if the interactive mode disappeared. It is a luxery and one mostly needed to compensate for other problems. As long as the points I mentioned are taken into account (which I feel they are), things should be fine.

[woodywood245]

I've just added an Official Features List. This list is not complete (I still need to add stuff to it), but it contains a good idea of what I've got done so far and what I need to do still. This only applies to the compiled version, which I consider experimental.

https://github.com/griderd/Jebnix/wiki/Official-Features

[woodywood245]

This is a sample of the documentation I am writing for this project. This page describes the Vector structure in KerboScript++. https://github.com/griderd/Jebnix/wiki/Vector-Structure-(KerboScript)

Is there anything else you'd like to see in the documentation that will make working with KerboScript++ easier?

P.S. The other wiki pages, except for Official Features, are out of date by several weeks because of the changes I've made. However, I will be updating them as I go.

[Masked Turk]

Is it really worth developing an alternate to kOS instead of contributing resources/ideas to the existing system that has an audience? Don't get me wrong, I'm not knocking your work, but it does seem a shame to fragment what is probably already a fairly small userbase.

[diomedea]

Is it really worth developing an alternate to kOS instead of contributing resources/ideas to the existing system that has an audience? Don't get me wrong, I'm not knocking your work, but it does seem a shame to fragment what is probably already a fairly small userbase.

Information Technology is filled with solutions that have turned obsolete and, though great efforts were made to keep them alive, ultimately had to be abandoned because new ones performed much better. Notable are MS DOS, parallel ports, floppy disks. Even if very used, ultimately those technologies had to be considered a legacy, and be removed to allow progress.

The original kOS is really an example of the above. Its language has so many quirks and shortcomings, it was really needed to improve. But, as often happens with new things in IT, different and sometimes opposite solutions emerge.

There is of course the new kOS Scriptable Autopilot System, by Erendrake. Originally started as a maintenance update to kOS, it now implements a revamped kerboscript syntax. But it strictly adheres to the original kOS code in many ways, and as a result it is sometimes even slower to compute.

Jebnix is not yet ready for use, therefore you don't see as much of a user base with it yet. But it has some great potential because it completely abandons the kOS code and implements its own (still able to interpret kerboscript, but also to run it in compiled form). Performance gains are expected to be impressive, and further improvements in the language are coming as well.

[erendrake]

Information Technology is filled with solutions that have turned obsolete and, though great efforts were made to keep them alive, ultimately had to be abandoned because new ones performed much better. Notable are MS DOS, parallel ports, floppy disks. Even if very used, ultimately those technologies had to be considered a legacy, and be removed to allow progress.

I would say that there are many more examples of Engineering projects that are total rewrites and never see the light of day, I know I have been part of my fair share of these

The original kOS is really an example of the above. Its language has so many quirks and shortcomings, it was really needed to improve. But, as often happens with new things in IT, different and sometimes opposite solutions emerge.

There is of course the new kOS Scriptable Autopilot System, by Erendrake. Originally started as a maintenance update to kOS, it now implements a revamped kerboscript syntax. But it strictly adheres to the original kOS code in many ways, and as a result it is sometimes even slower to compute.

We do not strictly adhere to anything if we think we can improve it. Truly very little code remains of the addon that Nivekk left us. If you had used it in the last month you would notice that the new kOS is considerably faster than the old one.

Jebnix is not yet ready for use, therefore you don't see as much of a user base with it yet. But it has some great potential because it completely abandons the kOS code and implements its own (still able to interpret kerboscript, but also to run it in compiled form). Performance gains are expected to be impressive, and further improvements in the language are coming as well.

Scripts run with kOS today are compiled and run much faster than before thanks to a lot of hard work by our small team. we are adding new features almost every day such as a vector drawing tool and an improved file editor that we just issued a pre-release for.

I welcome anyone who wants to develop a mod with their own time. Someone bringing new ideas to the table is exciting for everyone in the modding community. I wish woodywood245 the best of luck with his project.

[woodywood245]

Aaaannnd we're back!

I went AWOL for a while. As I'd mentioned in April, I'd made some major life changes, and I was working in relation to those. In short, I quit school and I'm starting a business. As a result, I've been doing a quite a lot of work in that area to get it going.

However, I haven't stopped working on this. I've just performed a commit a few minutes ago, which fixed a few known problems in the code. Hopefully I'll get a few things done over the next couple days.

[woodywood245]

A sample of the WinForms Jebnix Debugger. This includes the new video memory viewer.

[diomedea]

So happy to see you are back (again), and keep making progress with Jebnix. Wish all is good IRL for you .

If the disclaimer with the github repo is still current, Jebnix is not yet ready for independent testing. Would you point what steps still are required to make the first beta version (that I would like to run) ? Not asking timings or schedules, unless you want to tell.

[FennexFox]

Are there some mods like 'Mechjeb embedded' which embed Jebnix function to all command pod?

[woodywood245]

So happy to see you are back (again), and keep making progress with Jebnix. Wish all is good IRL for you .

If the disclaimer with the github repo is still current, Jebnix is not yet ready for independent testing. Would you point what steps still are required to make the first beta version (that I would like to run) ? Not asking timings or schedules, unless you want to tell.

I am at the point where the control flow from compilation to execution is complete. There are a lot of things that need to be implemented in both the compiler and runtime. I also need to set up vehicle control and some additional data types. Then I'll implement a KSP version of the console. That's the short version.

[woodywood245]

Are there some mods like 'Mechjeb embedded' which embed Jebnix function to all command pod?

Jebnix is not complete. So no, not yet.

[Assossa]

Hey woodywood, this mod looks amazing. I was wondering if i could help develop it. I wouldn't be a huge help, but hopefully I can help implement some of the features faster.

Thanks for considering, I am an experienced java and c# developer by the way.

[woodywood245]

I'm back! Again. As usual, life kinda went on a rollercoaster ride, and I haven't had as much time to work on this as I'd hoped. However, I'm getting back into it, reading the code, fixing stupid mistakes, removing bugs, and implementing features. Or in some cases, completely re-implementing features (WHAT WAS I THINKING!?)

Over the last couple days, I've fixed some problems in compiler's implementation of the Shunting-yard algorithm, fixed issues with the PRINT keyword, re-implemented SET, fixed a keyboard input issue, and added a handful of extra architectural stuff. I'm not done yet, though.

- - - Updated - - -

Hey woodywood, this mod looks amazing. I was wondering if i could help develop it. I wouldn't be a huge help, but hopefully I can help implement some of the features faster.

Thanks for considering, I am an experienced java and c# developer by the way.

Thanks for the offer. I'll think about it, but to be honest, I'm a bit of a control freak. I also don't currently have a clear development roadmap because I've been off the project for so long. So as far as that goes, I wouldn't exactly know what to have anyone else work on, because I'm not sure what I'm working on until I'm halfway into it.

[VentZer0]

I am curious how this will turn out. I'd like to suggest in terms of functionality that you also keep people in mind who are more into planes and like to autopilot them. Like me for example

Currently I am working on a autopilot in kOS for my drone planes (which are targets for my missile tests) and I have serious issues with steering the plane into turns and have them turn at ~0 vertical speed, ALT hold while turning. I would love if you could implement some smart PID-controller function. That way it would also be very easy to have a SAS alternative tailored to your craft, since SAS and FAR/NEAR do like eachother, planes start to oszillate extremely at some point with FAR or NEAR.

With a PID function it that we could tie into the controls this behaviour could be avoided.

Edited February 13, 2015 by VentZer0

[fatcargo]

Please confirm is Jebnix still developed ? From what i saw at github, last activity was in July of 2014, with one branch updated 10 days ago.

[lodestar]

I've been experimenting with kOS and it's great that an alternative is coming up, but how hard can it be to embed a Lua or Python interpreter in KSP, instead of writing a new language from scratch?

[fatcargo]

As far as languages are concerned, my preference is Pascal. I does not allow various ambiguities both in program structure and use of variables. If i remember right, there is "engine" for Pascal Script, which is open source, so reviewing for implementation in Jebnix should be a matter of time and effort, not royalties nor side-stepping vulture squadrons protecting intellectual property.