Insight on Mod Development Expectations

[REDACT3D]

Do any of you folk have any relevant insight on mod development for ksp2

I imagine it is still unity but HDRP and hope that it still uses the same kind of files used in ksp1. but no idea.

 

Any insight on this topic?

 

I quite enjoy makin' parts. and I notice the wing change eliminating the need for tiny variations.  but could add more complexity. the same goes with the multiplayer.  I imagine it will have navigation more automated and more consistent with the abilities of the majority of players.  maybe this will eliminate the need for navigation mods for example.  

Edited November 29, 2022 by REDACT3D

[GoldForest]

16 minutes ago, REDACT3D said:

Do any of you folk have any relevant insight on mod development for ksp2

I imagine it is still unity but HDRP and hope that it still uses the same kind of files used in ksp1. but no idea.

 

Any insight on this topic?

 

I quite enjoy makin' parts. and I notice the wing change eliminating the need for tiny variations.  but could add more complexity. the same goes with the multiplayer.  I imagine it will have navigation more automated and more consistent with the abilities of the majority of players.  maybe this will eliminate the need for navigation mods for example.  

Can't really 'insight' about something we know nothing about. All we've gotten is KSP 2 will be 'more moddable' and possibly easier. That's it. Nothing has been talked about on the modding side of KSP 2, afaik anyway. 

We don't even know if "Gamedata" is still used or if they moved to a new system of loading parts. 

Edited November 29, 2022 by GoldForest

[MechBFP]

They are moving to a LUA based system, last we heard a few years ago, so I expect it to be completely different.

[GoldForest]

39 minutes ago, MechBFP said:

They are moving to a LUA based system, last we heard a few years ago, so I expect it to be completely different.

They also said the LUA system was also optional and can be completely bypassed if you want to use the direct/old method. 

[MARL_Mk1]

When a developer says they'll support modding officially, I'd expect at least some tools (even if it's later on) to be released publicly. A planet builder would do wonders for modded planet packs, and we've seen some small clips of them using their own on the vlogs already.

Edited November 29, 2022 by MARL_Mk1

[Nazalassa]

1 hour ago, MechBFP said:

They are moving to a LUA based system, last we heard a few years ago, so I expect it to be completely different.

Dumb question, but what is a LUA system? I know a game that uses the Lua programming language for mods, but nothing else that is named "LUA".

(Except License User Agreement, but I don't think it's that.)

[GoldForest]

1 hour ago, Nazalassa said:

Dumb question, but what is a LUA system? I know a game that uses the Lua programming language for mods, but nothing else that is named "LUA".

(Except License User Agreement, but I don't think it's that.)

Lua isn't an acronym. It's the actual name of the programming code.

And Lua is basically a middle man coding system, from what I understand. It also slower than directly talking to the game from what I understand as well.

[Nazalassa]

Ah, so that's the programming language. OK.

Spoiler

Like in Minetest. The engine is written in C++, but the 'mods' (the game actually) are written in Lua.

[K^2]

5 hours ago, Nazalassa said:

Ah, so that's the programming language. OK.

  Reveal hidden contents

Like in Minetest. The engine is written in C++, but the 'mods' (the game actually) are written in Lua.

Yeah. The advantage of going with Lua (or some other properly sandboxed language) over native modules is that it's much safer to make them portable. So hopefully, there won't be any problems joining a MP game where a host is using a custom part with a Lua script attached, as the script can be safely downloaded. Or have the scripted parts and missions available on the Workshop.

Though, have we had a specific confirmation that we'll be able to use Lua for modding? When it was first mentioned, it was only confirmed for missions, but that was a while ago, and I'm sure I could have missed additional info.

[GoldForest]

1 hour ago, K^2 said:

Yeah. The advantage of going with Lua (or some other properly sandboxed language) over native modules is that it's much safer to make them portable. So hopefully, there won't be any problems joining a MP game where a host is using a custom part with a Lua script attached, as the script can be safely downloaded. Or have the scripted parts and missions available on the Workshop.

Though, have we had a specific confirmation that we'll be able to use Lua for modding? When it was first mentioned, it was only confirmed for missions, but that was a while ago, and I'm sure I could have missed additional info.

It was confirmed that modding could use Lua or bypass it, which I feel most mods will choose to bypass since Lua can be slower in some use cases. If it is slower, which I've heard it is, I doubt modders will want to increase an already long load time for the sake of "portability".

[K^2]

37 minutes ago, GoldForest said:

If it is slower, which I've heard it is, I doubt modders will want to increase an already long load time for the sake of "portability".

I mean, it depends on how much work you need done. If you have a few lines of script you need to make your component works, which is going to be most mods, I suspect, both the compile (loading) time and performance hit are going to be absolutely insignificant.

If you have a huge complex library of stuff, yeah, you want a module, and keep any Lua interactions down to hooks if needed at all.

Either way, I'm glad they're giving us both options. That's fantastic to hear.

[Gotmachine]

Nothing was ever confirmed regarding LUA modding.
All we know is early 2020 tidbits from one random guy on the KSP 2 team.
Who was saying that they had experimental LUA bindings on top of the C# API used in the context of some internal prototyping tools, which may or may not be used in the final product.

Edited November 30, 2022 by Gotmachine

[K^2]

55 minutes ago, Gotmachine said:

Who was saying that they had experimental LUA bindings on top of the C# API used in the context of some internal prototyping tools, which may or may not be used in the final product.

We've had a job posting in February of this year for a Mission Designer, which explicitly required knowledge of Lua. So we know that these experimental bindings turned into the way missions are done for the game at the very least. I have not personally seen where the confirmation of Lua for modding comes from, so just going along with what @GoldForest says. It would make perfect sense to allow Lua scripts on parts if there's already a support for it on the missions side, because that does expand what you can do with modding in multiplayer, but it'd be nice to see a link to an interview or an article that states that this was, indeed, implemented.

For reference, here's the thread about the posting. Unfortunately, the posting itself is no longer available, and the only surviving bit seems to be the part I quoted about Lua. (Wayback machine was of now help, since this is a dynamic page.)

[Gotmachine]

2 hours ago, K^2 said:

So we know that these experimental bindings turned into the way missions are done for the game at the very least.

This is a lot of extrapolation. So, yeah, this just confirms that those LUA bindings still existed in february and are still used for internal developments.

For reference, this is the only source we have (it's actually from 2019) :

I see very little incentive for them to put together a whole user facing LUA-based modding API and framework, when thanks to the game being an Unity game they basically get full modding support for "free".

What I realistically envision is some level of LUA embedding for specific configuration files use cases. Being able to have a fully functional programming language at the config files level can be very useful in some situations.
There are many KSP 1 mods that provide in one form or another some level of logic embedding in config files (ModuleManager and ContractConfigurator for example).

But anything is possible. Indeed, if they already have most of the LUA bindings they (we) need already in place and have used them in production for some final configuration systems, the gap to a LUA based modding framework might not be that much extreme, although the API coverage needed to be able to do anything useful is quite huge. It doesn't only need to cover the KSP API, but also large chunks of the Unity API.

A LUA modding framework would maybe somewhat lower the perceived barrier of entry to modding (although if this is actually good is debatable). There are some possible benefits like hot-loading (to be fair that's also possible with C#), but there are many downsides.
Beside API coverage limitations, LUA is slow, it's a very barebone language with a very barebone standard library. My opinion is that it would make modding harder, not easier, and without putting words in others mouth, I doubt any serious KSP 1 modder would disagree on that.
Overall, my feeling is that this is a lot of work for the game devs and not that much benefits for the modders. As a modder, if I were to choose what I want, I'd rather have a good built-in CKAN/ModuleManager than a redundant LUA modding API.

Don't get me wrong, having a scripting language with good API coverage would be an interesting feature. It would basically mean that KSP 2 has a built-in KOS, or at least something quite close from it, and that opens up a lot of possibilities if the LUA system itself is correctly exposed from the C# API.

Edited November 30, 2022 by Gotmachine

[K^2]

14 minutes ago, Gotmachine said:

I see very little incentive for them to put together a whole user facing LUA-based modding API and framework, when thanks to the game being an Unity game they basically get full modding support for "free".

The incentive is that if I make a mod with a native plugin, I can take full control of your computer if I'm a malicious type and convince you to downnload my mod. That means that if you join a multiplayer server, having the server share native mods with you is a huge security hole. This creates all sorts of complication.

Lua runs in a sandbox. If properly set up and properly patched, there are no known ways to exploit it for an attack, meaning you can just download all the mods from the server when you join a multiplayer game. Likewise, for sharing modded parts, you can utilize things like Steam Workshop, have modders upload their mods there, and have zero risk when people download these.

Lua is by far not the only way to do this, of course, but there is also no reason to add yet more complexity for this if Lua is already being used.

16 minutes ago, Gotmachine said:

This is a lot of extrapolation. So, yeah, this just confirms that those LUA bindings still existed in february and are still used for internal developments.

You don't put it on a job requirement like this unless it's actually heavily used. So yes, it might be entirely internal, with no user-facing interface, but we know that Lua engine is in and it's heavily used by the game.

Once you have that, creating bindings for a mod is actually very easy in C#. C# supports custom attributes. So any function you create for use with modding API can simply have something like [KSP_Lua_Function] attribute in front of it and it's now available in the Lua API. That's it. Last studio I've worked at, we did this for Python bindings in our dev tools. It was all internally facing, since we didn't ship our tools, but it made writing Python script for our tools and content validation a cakewalk.

So again, this is hardly confirmation, but we know that the Lua engine is in the game, there was interest in and at least some work done to expose Lua to modders, the additional workload is trivial - as the modding API C# functions have to be written anyways, and the binding attributes have to exist for internal use already, and there is a strong incentive to make this available to modders to avoid security problems when sharing mods.

At this point, my confidence in this being a feature usable in KSP2 modding is well past 50%, though still far from certainty until officially confirmed, of course.

[Zozaf Kerman]

Roblox uses lua for scripting, if that helps any.

[linuxgurugamer]

50 minutes ago, Gotmachine said:

Beside API coverage limitations, LUA is slow, it's a very barebone language with a very barebone standard library. My opinion is that it would make modding harder, not easier, and without putting words in others mouth, I doubt any serious KSP 1 modder would disagree on that.

Ummm, I do disagree.  While Lua maybe slower than native, if the game is compiled into native code, the speed loss won't be noticeable compared to interpreted c# byte code

[Gotmachine]

1 hour ago, K^2 said:

So any function you create for use with modding API can simply have something like [KSP_Lua_Function] attribute in front of it and it's now available in the Lua API.

I'm quite skeptical. For that to work, you also have to create bindings for all types used as a parameter or return type, and consequently bindings for the methods in those types too. That mean a huge proportion of the BCL and Unity API surface.
In a way or another, you have a ton of work to do if you want to expose a vaguely useful LUA modding API.

1 hour ago, K^2 said:

That means that if you join a multiplayer server, having the server share native mods with you is a huge security hole.

While we have zero confirmation that LUA modding exists, we have a lot of confirmation that C# modding exists, and there is technically no way to do such sandboxing if they allow C# modding.
And frankly, if they try, it would be a deal-breaker and a major failure on their "KSP 2 will be at least as moddable as KSP 1" statement.
I can guarantee you that the first thing the modding community would do is to move on to BepInEx or similar and build it's own C# based no strings attached modding ecosystem. By definition, modding doesn't want to limit itself to what the developer has made.

If you read carefully what was reported by Snark in 2019, I think the likely scenario is quite clear : embedded LUA scripts in config files in order to be able to define custom logic within the scope of the configured subsystem and its specific functionality.
This seems like a very sensible choice to me, but extending that to a general purpose modding API that cover the full KSP (and consequently BCL & Unity) API surface seems a total waste of resources for very questionable benefits.

53 minutes ago, linuxgurugamer said:

While Lua maybe slower than native, if the game is compiled into native code, the speed loss won't be noticeable compared to interpreted c# byte code

Hu, what ?

C# is compiled to machine assembly, it's not interpreted on the fly.
Lua can either use a static interpreter reading byte code, or be compiled to machine assembly too.

Either way, in practice, Lua mods would be much slower than C# mods.
Not so much because it is intrinsically slower (which it is), but mainly because of the overhead of calling into the C# API while mutating the methods parameters on every call. The overhead here would be very significant.

Edited December 1, 2022 by Gotmachine

[K^2]

1 hour ago, Zozaf Kerman said:

Roblox uses lua for scripting, if that helps any.

They use a custom implementation of it. It started of as an off-the-shelf implementation, but they've since replaced it with an entirely custom Lua engine that does some very clever optimizations and JITing.

59 minutes ago, linuxgurugamer said:

Ummm, I do disagree.  While Lua maybe slower than native, if the game is compiled into native code, the speed loss won't be noticeable compared to interpreted c# byte code

Ok, there are a lot of misconceptions here.

Lets start with C#. C# scripts for Unity components first get converted to Common Intermediate Language (CIL). There are three ways that can run. Unity provides a transpiler that takes CIL and converts it into C++ code (IL2CPP) that is then compiled to native code. That is your best performance. Still not as good as writing C++ code yourself, but very close. However, it's not always an option for plugins, as the C# code for IL2CPP has to be available when you build the game. If you're making your own plugin, you can absolutely just write all of your performance-critical code in C++ or have your code likewise transpiled into native, and then bind it from C# layer. But any C# code you ship with your parts is going to be interpreted from CIL directly. So that leads us into two other ways Unity runs C# code, and it's by using either .NET Common Language Runtime (CLR) or Mono. Former is available only on MS platforms, and the latter is available for the other platforms KSP2 is likely to ship on. Both will optimize CIL further and use JIT (Just in Time) compilation to run the code. This is slower, but not by a lot. Typically, you'll get about 20%-30% performance of native with CLR and a little bit worse with Mono.

Next, Lua. Lua is compiled to Lua bytecode. There are ways to convert that to native code, but that completely defeats the purpose of using Lua in the first place. The only benefit of using Lua is running it in an interpreter sandbox. The typical way to run it is using Lua's own runtime, which is a pure bytecode interpreter. It is comparatively slow, about 10x slower than CLR and Mono on average. There are projects that are designed to bring in on-demand optimization and JIT to Lua, like the aforementioned Roblox runtime and the LuaJIT project. The latter is publicly available, but it's relatively new and I don't know how good their sandbox security guarantees are. Since the only reason to even entertain Lua code for KSP2 mods is safe portability of code, odds are, if we're getting Lua in KSP2 for mods, it's the slow, official version.

Now, 20-50 times slower than native is still blazing fast in the absolute terms. Most mods I've seen that have custom code for custom parts does not need full native performance. A few percent of native is fine. If you want to write a custom IK solution for a Canadarm part mod that automatically moves modules on your station, Lua is plenty fast enough. If you want to run dynamic trajectory estimates, you'll want to be careful about how you write your code, but you can get away with Lua. The place where Lua will start struggling is anything to do with graphics. But I'm not sure we'll be able to do much with custom displays without C# anyways.

15 minutes ago, Gotmachine said:

C# is compiled to machine assembly, it's not interpreted on the fly.

See above. This is only the case if you specifically enable IL2CPP, which converts to C++ (which never becomes assembly in modern systems), and this only works for code that's available when you build your project.

If you want to import things dynamically, best you can do is interpreting CIL or Lua bytecode, depending on the language.

17 minutes ago, Gotmachine said:

I'm quite skeptical. For that to work, you also have to create bindings for all types used as a parameter or return type, and consequently bindings for the methods in those types too.

No. You use language reflection and it's automatic. You literally just have a line you paste in front of a function you want to be accessible from Lua.

18 minutes ago, Gotmachine said:

and there is technically no way to do such sandboxing if they allow C# modding.

Of course there is. You simply don't allow a client to connect to a host unless the client already has all the plugins that the server is running. That's how it's done in literally every moddable multiplayer game. You both must have the mod installed.

Edited December 1, 2022 by K^2

[Gotmachine]

3 minutes ago, K^2 said:

So that leads us into two other ways Unity runs C# code, and it's by using either .NET Common Language Runtime (CLR) or Mono. Former is available only on MS platforms, and the latter is available for the other platforms KSP2 is likely to ship on. Both will optimize CIL further and use JIT (Just in Time) compilation to run the code. This is slower, but not by a lot. Typically, you'll get about 20%-30% performance of native with CLR and a little bit worse with Mono.

Nope, Unity uses (a custom fork of) Mono on all platforms.

And IL2CPP performs very marginally better than Mono in terms of execution speed. The main reason it exists is because it allow Unity games to run on platforms where JIT VMs aren't allowed (namely iOS), but it also has the advantage of producing a smaller distribution and using slightly less memory (because it strips unused BCL classes and doesn't need the JIT VM), which is relevant on mobile platforms in general.

14 minutes ago, K^2 said:

best you can do is interpreting CIL

CIL is not statically interpreted. That's not how the JIT works. It is compiled to assembly when the application start. Which means that technically, you can get as much performance from C# as in C or C++.
C# in practice is (slightly) slower than equivalent C++ code because of the overhead of the high level constructs of the BCL and of the GC.
But at least on modern CoreCLR, if you desire so you can get quite close to the metal with the most low level C# constructs and get similar performance as C/C++.

[K^2]

1 minute ago, Gotmachine said:

CIL is not statically interpreted. That's not how the JIT works. It is compiled to assembly when the application start. Which means that technically, you can get as much performance from C# as in C or C++.

There is not a single modern language that compiles to assembly. Lets get that out of the way first. It's just not done. You can ask some C++ compilers to dump assembly code, but even that, in practice, is usually achieved by disassembling output binary. Assembly is not a good choice for intermediate language, and so it is simply not used.

JIT turns bytecode to machine code during runtime. That's literally it's job. Please consider reading up on JIT and how it's actually used. A compiled C# program is stored as CIL and does not turn into machine code until execution, and only as necessary. A typical JITed interpreter will still elect to run bytecode in pure interpreter mode until it becomes a performance penalty over spending time to JIT the code. This is a standard feature of CLR, Mono, V8, LuaJIT, JVM, and basically any serious JITed interpreter.

I stand corrected on use of CLR. Older versions of Unity were relying on CLR on MS platforms, but yes, it looks like recent versions have switched to Mono entirely.

[linuxgurugamer]

Actually, based on some benchmarks done by  a former Microsoft developer, Dave Plummer, it appears that C++ much faster than C#.  

You can review the results here, along with all the source code used:  https://github.com/PlummersSoftwareLLC/Primes

The first of the series of videos can be viewed here:  

 

[Gotmachine]

4 minutes ago, K^2 said:

A typical JITed interpreter will still elect to run bytecode in pure interpreter mode until it becomes a performance penalty over spending time to JIT the code.

Nope. CLR and CoreCLR aren't and never were interpreters. Only Mono has (had ?) an (optional) interpreter, which isn't used in Unity.
They fully compile the whole application on startup, with the downside being a vastly increased start delay.
Which is why stuff like NGen or NativeAOT existed, and why CoreCLR has introduced tiered compilation where jitting is initially done with a minimal amount of work done on optimizations to get quick app startup, then it takes some times to do some deeper analysis and re-jit identified hot paths.
The compact framework JIT is a bit different and does on-demand jitting, not compiling stuff until it actually used. The micro framework JIT however is an interpreter, but it's almost a completely separate product.

20 minutes ago, K^2 said:

Older versions of Unity were relying on CLR on MS platform

Nope. Unity has forever been using Mono.

[Gotmachine]

34 minutes ago, linuxgurugamer said:

Actually, based on some benchmarks done by  a former Microsoft developer, Dave Plummer, it appears that C++ much faster than C#.  

I won't go into that pointless debate. Low level modern C# is just as fast as low level modern C++.
High level "usual" C# code tend to be slower than equivalent "usual" C++ code because the BCL abstractions are much higher level and offer handy features like automatic exception handling, managed memory and some level of thread safety.
If you choose to deliberately to get ride of those high level features (which you can in modern .NET), C# code becomes again in same ballpark as C++.

Edit : actually, did you look at the results on the GH repo you linked ?
The fastest C# implementation (very low level) is 34% faster than the fastest C++ implementation.
The "normal" C# implementation ("tannergooding") is twice slower.
Which kinda illustrate my point nicely. I suspect that a middle-ground C# implementation not resorting to unsafe code but using .NET6/7 low level constructs would be roughly equivalent as C++.

Edited December 1, 2022 by Gotmachine

[Vl3d]

This is a great thread. Thank you - to all the modders! You have molded KSP into what it is for each of us.