Tool Portmod Package repository for KSP2

[bmw]

Edit: to clarify, as my intentions may not have been clear from the initial post, I'm mostly looking for feedback here and would like to know if there is any interest in using Portmod

I've put together a KSP2 mod package repository to use with Portmod

https://gitlab.com/portmod/ksp2

I haven't actually tested it properly as I don't have a copy of KSP2 yet, but it happened to be a great way to test some improvements I've been making to  Portmod's package generator tool, mostly because ksp mods have direct downloads, standard licenses and very good metadata available, so I could quickly test several new mods without having to enter much manually, and without having to mess with nexusmods for other games (ksp stuff is a pleasure to work with by comparison).

But in case anyone's interested, here it is. There aren't any KSP2-specific guides on the Portmod wiki yet, but you can in short:

1. Install Portmod: https://portmod.readthedocs.io/en/stable/install/index.html
2. Setup a prefix with

   portmod init <prefix_name> ksp2 <ksp_directory>

   (your choice of prefix name, Portmod stores configuration globally rather than being local to a particular directory like CKAN, so you use 'portmod <prefix_name>' to run commands for a given prefix)

3. Start installing mods (See https://portmod.readthedocs.io/en/stable/basic-usage.html)

As mentioned in the Portmod thread, there isn't great support for game engine versioning yet, but it's being worked on.

[VITAS]

i welcome your effort. However i think you should be aware that CKAN has a checkbox for mod authors to opt into it on SpaceDock.  You might want a similar opt in method for any mod Repo you build.

If you want and are willing to commit the needed code this could be added to SpaceDock.

Edited March 10, 2023 by VITAS

[bmw]

I don't see any way of reading that data from the spacedock API. There's just a field when posting a new mod. Is this a permissions thing, or is it just a way to trigger the generation of a package?

Taking a quick look at the spacedock code, (this being what I found) I don't think there's really an equivalent for portmod at the moment, both because there isn't a server to notify, and because while importmod can generate packages automatically, I'm not sure the package generation is quite ready to be completely unsupervised. I've done it in a limited capacity in the past (with the exception of the portmod/ckan repo for ksp1), but the general idea has been that generated packages are marked as untested and any stable package versions will be preferred over untested packages during dependency resolution.

I noticed that one mod which I packaged includes a netkan file, which should be sufficient for completely automated package generation since it includes dependencies and installation data (lacks optional features I assume since CKAN also does, but so far I haven't seen any of those anyway). Some of the others at least have swinfo.json, which has dependencies, but not install paths, so importmod would have to rely on heuristics which can easily fail, particularly if stuff like a builtin mod loader changes how things install.

[VITAS]

There one on ckan side that runs on SpaceDock and notifies them. You want to talk to @HebaruSanabout how ckan works and integrates (it might even help you not to reinvent the wheel)

The main point im trying to make is: Mod Authors have to approve of distribution channels.

[bmw]

8 minutes ago, VITAS said:

The main point im trying to make is: Mod Authors have to approve of distribution channels.

I sort of understand what you mean, but Portmod's repositories are just metadata repositories. I'm not redistributing the mods (except sometimes when they have a license permitting it, but I haven't set up anything like that for ksp2), and all information about the mods points to the original homepages and documentation (i.e. I'm not doing anything that legally requires me to get the mod author's permission).

I'm aware that CKAN has had some issues in the past with mod authors complaining about being overwhelmed by misdirected user complaints when CKAN metadata is incorrect (though I may be missing some of the details), is that the sort of thing you're referring to?

Portmod takes a much more conservative approach to packaging (based on Gentoo's model for package stabilization). Generated packages, as mentioned, start as "untested", and the idea is that they get tested manually, marked as "testing", then stabilized after some time has passed and the stability has been independently verified (in practice however the distinction between testing/stable has been ignored, essentially making everything perpetually "testing", since there haven't been sufficient contributors to make the the extra stabilization stage beneficial). More experienced users (who in theory know better than to complain upstream) can set Portmod to "accept" testing packages and then report any issues they find to help with the stabilization process, while less experienced users who just want things to work will just get the stable packages by default.

[linuxgurugamer]

2 hours ago, bmw said:

I sort of understand what you mean, but Portmod's repositories are just metadata repositories. I'm not redistributing the mods (except sometimes when they have a license permitting it, but I haven't set up anything like that for ksp2), and all information about the mods points to the original homepages and documentation (i.e. I'm not doing anything that legally requires me to get the mod author's permission).

CKAN is also just metadata repositories, yet some mod authors object to their mods being indexed by CKAN.

Frankly, CKAN is well known in the KSP community, and setting up an alternate method will just lead to balkanization and fracturing, which we really don't need.

[bmw]

13 hours ago, linuxgurugamer said:

CKAN is also just metadata repositories, yet some mod authors object to their mods being indexed by CKAN.

What I've read mostly indicates that the objections to mods being indexed is a matter of things specific to CKAN, more than a general problem (e.g. stability, as mentioned, as well as for Principia it seemed to be because CKAN can't do anything about certain external dependencies). I'm not opposed to only adding mods with explicit permission, I was just asking about the background as the motivations and implementations are not very clear to me (e.g. the spacedock "Add to CKAN button" is presented just as a way to add it to CKAN, there's no mention that you have to give permission, and while I have run across the fact that CKAN only includes mods with explicit permission when reading CKAN issues, I don't see any official description of that policy in the README or FAQ).

13 hours ago, linuxgurugamer said:

Frankly, CKAN is well known in the KSP community, and setting up an alternate method will just lead to balkanization and fracturing, which we really don't need.

I'm not looking to fracture the community, just wondering if there is any interest in Portmod (maybe I didn't communicate that as well as I could have in the initial post). If so, it might be possible for them to coexist without issues. This isn't a zero-sum game; multiple package managers can each bring their own benefits to the community.

Portmod could be used, for example, to compile mods from source. I don't know enough about the KSP modding architecture to know if there is an advantage to that for users, but if ABI changes occur without API changes, it could provide better compatibility (setting it up could be complicated though, particularly on Windows).

Portmod also has a number of packaging features which CKAN lacks (last I checked anyway). E.g. to name a few:

• Support for multiple source archives in a single package file
• Support for tar.{gz,bz,xz} by default in addition to zip, and other archive formats by installing tools itself (e.g. the bin/7z package for openmw)
• Support for platform-specific installation behaviour
• Support for optional mod features (including full support in dependency resolution and fetching)
• More or less arbitrary installation scripts in case mods are more complicated (fortunately doesn't seem like that's usually an issue for KSP)

But if there isn't any interest in directly using Portmod's packaging environment, I could just set it up as a CKAN-meta client like I did for KSP1, assuming there aren't any major objections to that, as it would at least keep the package metadata unified and shouldn't fracture anything.

[linuxgurugamer]

There are very few mods that CKAN can't deal with, Principia is one, Sound Track Editor is another.

12 hours ago, bmw said:

"Add to CKAN button" is presented just as a way to add it to CKAN,

If you are adding a mod to Spacedock and you select that checkbox, that is the mod author giving permission to have it indexed by CKAN

12 hours ago, bmw said:

Portmod could be used, for example, to compile mods from source. I don't know enough about the KSP modding architecture to know if there is an advantage to that for users, but if ABI changes occur without API changes, it could provide better compatibility (setting it up could be complicated though, particularly on Windows).

Ummm, I have no idea how you can compile a mod like that.  KSP uses C# code, and it takes a little bit to set up an environment to be able to compile KSP code.  So far, it looks like KSP 2 will eventually support that, and possibly other languages as well.  Since the developers have not yet released any information on modding KSP2, nor provided any guidance, any work done right now is subject to extreme changes

 

Among other things, the CKAN people spend significant time verifying that the indexing is good, any dependencies are specified, proper permission and licensing has been specified, etc.  It's not just a program.  Also, CKAN is both a GUI-based mod manager (which most people use) as well as a CLI manager.  From what I just read (correct me if I'm wrong), Portmod is only CLI

 

[Grimmas]

I've never heard of Portmod or have seen a single game whose mods use it. I don't even know the original project that it's forked from. Is it just some obscure weekend FLOSS project or has it seen actual use somewhere? 

[bmw]

On 3/11/2023 at 11:32 PM, linuxgurugamer said:

If you are adding a mod to Spacedock and you select that checkbox, that is the mod author giving permission to have it indexed by CKAN

That is obvious by association, but it doesn't seem obvious to me from the context provided on that page that the grant of permission is needed (and it's not really explained anywhere I could find in the CKAN docs). For me, being more familiar with the open source software world where asking the authors permission before packaging software for a Linux distribution is unheard of, having to grant permission, particularly for a mod which already has a permissive license explicitly allowing people to do what they want with the mod (under the conditions of the license) is not something I would expect.

On 3/11/2023 at 11:32 PM, linuxgurugamer said:

There are very few mods that CKAN can't deal with, Principia is one, Sound Track Editor is another.

There are also some mods which CKAN doesn't really handle that well (and maybe I'm just a purist; from a practical perspective, CKAN generally works fine. I'm just pointing out that an advantage of Portmod is that it can relatively clearly and concisely combine various options into a single package rather than having to duplicate lots of data and split things across multiple packages. CKAN's method is certainly a good way of keeping the spec concise). E.g. AdvancedFlyByWire has separate packages for Windows/OSX/Linux, while Portmod could package that as a single combined package. The same is true for CKAN packaging variants of texture mods for each size separately, or different configuration files meant as compatibility for different mods, packaged separately from the main mod and with a package for each different alternative.

Maybe one possibility for Portmod and CKAN coexisting is that CKAN could be used for the vast majority of mods that work well with it, and Portmod could be used alongside it for installing mods which are more complicated. I don't know if that's the best option, I'm just hypothesising.

On 3/11/2023 at 11:32 PM, linuxgurugamer said:

Ummm, I have no idea how you can compile a mod like that.  KSP uses C# code, and it takes a little bit to set up an environment to be able to compile KSP code.  So far, it looks like KSP 2 will eventually support that, and possibly other languages as well.  Since the developers have not yet released any information on modding KSP2, nor provided any guidance, any work done right now is subject to extreme changes

Portmod can install software as well as mods, and as long as it's possible to compile the mod from a script of some form, it should be possible to automate installing the compilers and libraries and compiling it with Portmod. Some patching may be required if mods expect libraries to be in unusual places.

• On Linux binary packages can be borrowed from a Linux distribution. Alpine Linux seems particularly well suited since it builds software against musl and has a lot of statically linked packages which would reduce the number of dependencies.
• On macOS it might be possible to leverage homebrew packages, or homebrew itself.
• On Windows it would probably be more difficult since Portmod's sandbox restricts writes to the registry, so unless the tools have portable versions, it may not be possible to install them using Portmod in its current incarnation. However I've looked at using Docker in the past (I didn't test on Windows but it worked fine on Linux), which should work in this situation, though it's somewhat more complicated to install than Sandboxie, the current sandboxing tool.

On 3/11/2023 at 11:32 PM, linuxgurugamer said:

Also, CKAN is both a GUI-based mod manager (which most people use) as well as a CLI manager.  From what I just read (correct me if I'm wrong), Portmod is only CLI

I'm not sure what point you're trying to make here. I'm not trying to suggest that Portmod is universally superior to CKAN, or that it should replace CKAN entirely. It's a different piece of software with different advantages and disadvantages, and yes, it's CLI-only (personally, I prefer CLI package managers to GUI ones; I don't consider it a fatal flaw if that's what you're implying, though I admit that may mean some people may not be comfortable using it).

On 3/12/2023 at 7:59 AM, Grimmas said:

Is it just some obscure weekend FLOSS project or has it seen actual use somewhere? 

I started working on it four years ago for OpenMW mods. Some small amounts of work have been done for a few other games, but never really advertised much or maintained. It's not particularly well-known, largely because I spend most of my time tinkering with software rather than advertising it online, though lately I've been trying a little more to do so. Adoption in the OpenMW community has been relatively slow but steady, however it is hindered by few contributors and a lack of consistency in mods, which makes packaging take more time and require a lot of manual checks (nexusmods being the most common distribution site for mods doesn't help).

Edited March 13, 2023 by bmw
clarify wording