Jump to content

The Comprehensive Kerbal Archive Network: Call for mod participation

pjf

By pjf
in KSP1 Mods Discussions

 Share

Recommended Posts

pjf

pjf

Posted
Posted (edited)

We've released! Head on over to the official release thread, as this thread is now closed.

The Comprehensive Kerbal Archive Network

SN102Oa.png

As some of you know, we've had a team of people working on the Comprehensive Kerbal Archive Network, or CKAN. The CKAN aims to provide a way to install and manage mods and their dependencies in a reliable, reproducible fashion. It does this through the use of strong metadata (with both human readable and machine readable specs), with many lessons and conventions drawn from both the Debian Policy Manual, and the CPAN Meta Spec, both of which are successful projects with similar operation, and which deal with tens of thousands of software packages and their dependencies.

If you've come from a Debian or Ubuntu background, then the CKAN is "apt-get for KSP mods", but with a GUI as well as a command-line tool. If you've ever facepalmed because users couldn't follow install instructions, installed incompatible mods, installed outdated mods, didn't install the dependencies, or installed things to the wrong place, then the CKAN aims to make your life easier.

The CKAN is still being developed, and is pretty close to a v1.0 release. There's a working client under active development, and we're in the process of gearing up for an official release now. We absolutely welcome testers and contributions, but there is absolutely no guarantee that things won't break.

The client works on all operating systems (targeting Mono 4.0), can detect steam installs and KSP version, and is able to calculate which mods can be installed on your system, and actually install (and upgrade, and remove) them for you. However the CKAN can only install mods if it has the metadata it needs, and that's where you come in.

CKAN Metadata - We need your help

We need the help of mod-authors to make their mods CKAN indexable. If you're already releasing KerbalStuff, then this is as simple as writing a CKAN file once, and including it in the zip file you distribute. CKAN files are just JSON, and if you're releasing on KerbalStuff then it can be a very small JSON file, as the KerbalStuff APIs can provide us with the rest of the information we need, which means you can include it once and never have to update it again. This is what a simple CKAN fragment looks like:

{
    "spec_version"   : 1,
    "identifier"     : "DogeCoinFlag",
    "$kref"          : "#/ckan/kerbalstuff/269"
}

The "$kref" token is a special token indicating that the CKAN toolchain is expected to fill in remaining felds (version, license, author, homepage, etc) from externally supplied data (in this case, from KerbalStuff). By default, if no install instructions are provided, the CKAN installer will look for the top-most directory in the zip file that matches the mod identifier, and install that to GameData. This works with the majority of common mods.

However CKAN files can contain a lot of data, for example, they can depend upon, recommend, or suggest other mods, install tutorials or ships, or list additional resources: 


{
    "spec_version"   : 1,
    "identifier"     : "ExampleMod",
    "$kref"          : "#/ckan/kerbalstuff/1337",
    "depends" : [
        { "name" : "RealSolarSystem" },
        { "name" : "ModuleManager", "min_version" : "2.5.0" },
        { "name" : "FerramAerospaceResearch" }
    ],
    "recommends" : [
        { "name" : "RealismOverhaul" },
        { "name" : "Toolbar" }
    ],
    "resources" : {
        "github"   : {
            "url"      : "https://github.com/ExampleUser/ExampleMod"
        }
    },
    "install" : [
        {
            "file"       : "ExampleMod",
            "install_to" : "GameData"
        },
        {
            "file" : "ExampleTutorial",
            "install_to" : "Tutorial"
        }
    ]
}

When we declare the CKAN client ready for release, we'd like to support as many mods as possible, which means collecting mod metadata now. We're very dedicated to working with mod authors to make that happen.

Getting the client

Go to our releases page! Releases run on Windows, Linux and Mac, and have no dependencies other than .NET/Mono 4.0.

Adding your mod to the CKAN

Use this awesome guide on our wiki! You can update it if you have hints or tips!

Get involved!

If you'd like to see CKAN development as it happens, have a chat, or even see what you can do to help, you can join us in #ckan on irc.esper.net. The CKAN github repo is our focal point for tracking issues, contributions, making releases, and getting things done. We welcome contributions and discussions that are in-line with our code of conduct, and we always have plenty of things we're working on. If you'd like to join us, we'd love to see you!

Many thanks!

~ pjf

Edited by pjf
Link to release thread
Link to comment
Share on other sites
jordanjay29

jordanjay29

Posted
Posted

This is what I've been waiting for. I'm not a modder, so I'll patiently sit back until this is ready for public release, but I've been saying for months that KSP needs a mod manager with dependency management. It just takes someone (like the OP) stepping up to propose the standard for it to make it happen.

You should also consider broadcasting this in Addons Development and on the KSP subreddit for more exposure. Eventually modders will get the picture, but multiple avenues to reach them is always a good idea.

Link to comment
Share on other sites
pjf

pjf

Posted
  • Author
Posted

Ippo: Hooray! I look forward to it! :)

jordanjay29: Aye, good idea. Although we're not really stirring up the publicity engine fully until we've got a solid, user-targeted release. That's coming along pretty quickly, though; I'm already managing all my mods via the CKAN.

Justin: We've got a 4.0 target because the code makes extensive use of dynamic objects (which are *really* useful when working with JSON), and also because I'm still relatively new to C#, and all the features I kept reaching for were only introduced in 4.0. I fear if I was targeting 3.5 from the get-go then we wouldn't yet have a usable client.

If it ends up that a 4.0 target is a big hurdle after our first release then we can look at shifting back to 3.5, but I suspect that most people will find installing the .NET/Mono 4.0 framework a relatively easy step.

~ Paul

Link to comment
Share on other sites
pjf

pjf

Posted
  • Author
Posted

Just a quick note to say:

* Your mod will start to appear in the CKAN even without you having uploaded it to KS. We've got a special deal where we can download files from the future.

* I've just about got github support working. :)

* You can download a test client and play if you like, just make sure you report any issues you might encounter. (`ckan version` and `ckan gui --debug` are useful commands here)

:)

Link to comment
Share on other sites
pjf

pjf

Posted
  • Author
Posted (edited)

Thanks Paul! I *love* your work, so I'm glad you're excited. :)

Right now you *can* install mods based upon the presence of other mods:

{
    "file" : "GameData/YourMod/Optional",
    "install_to" : "GameData",
    "requires" : "SomeOtherMod"
}

However this is discouraged, and we may even officially deprecate it. The reason is that the user may install SomeOtherMod in the future, in which case they *won't* get the extra components from yours. Likewise, if the user uninstalls SomeOtherMod in the future, our dependency system doesn't detect that parts of your mod should be uninstalled with it.

Where possible, the preferred way to make a component that conditionally activates on another is using ModuleManager, with the `[:NEEDS]` token. This simplifies the install, and handles the situation there the additional mod is installed (or potentially removed) in the future.

For more complex operations, relationships combined with provides can be very expressive. For example, if you have a mod that requires life support, has custom components for each life support mod, and a base of common components which are used by everything, one could use this sort of arrangement:

* MyMod depends upon MyModLifeSupport

* MyModTACLS provides MyModLifeSupport, depends upon MyMod and TACLS, conflicts with MyModLifeSupport

* MyModECLSS provides MyModLifeSupport, depends upon MyMod and ECLSS, conflicts with MyModLifeSupport

* MyModSnacks provides MyModLifeSupport, depend supon MyMod and Snacks, conflicts with MyModLifeSupport

This guarantees that anyone installing MyMod must also select a life support system to go with it (and the latest dev build of the GUI does this), but you can't have *two* modules providing MyModLifeSupport installed at the same time. We have a special rule that mods can't conflict with themselves. (If you're familiar with Debian packaging, this is pretty much identical.)

Likewise, while the spec currently defines an `optional` keyword for install stanzas, its use is also discouraged (and may be deprecated), as we can't query the user about optional components for unattended installs. The recommended way of doing this is to have another package which depends upon, and is either recommended (installed by default) or suggested (not installed by default) by the parent package. Those package may reference the same underlying zipfile, so there's no need for additional downloads or author packaging, but the result is that users can install and uninstall optional components via the client, and other mods can also depend upon or recommend optional components of other mods.

~ Paul

Edited by pjf
Spelling and grammar fixes
Link to comment
Share on other sites
nlight

nlight

Posted
Posted (edited)
It may be good to have a JSON validator tool which use the official schema for validation.

There is such a tool available here. It requires Perl and Java to run. We're looking into implementing JSON validation in the CKAN client in the future.

Edited by nlight
Link to comment
Share on other sites
nli2work

nli2work

Posted
Posted (edited)

looks cool

for me to get RetroFuture to take advantage of CKAN I should include the following correct? Included in the KerbalStuff upload... as "RetroFuture.ckan"?

{
    "spec_version"   : 1,
    "identifier"     : "RetroFuture",
    "$kref"          : "#/ckan/kerbalstuff",
    "depends" : [
        { "name" : "Firespitter" },
        { "name" : "ModuleManager", "min_version" : "2.5.1" }
    ],
    "recommends" : [
        { "name" : "FerramAerospaceResearch" },
        { "name" : "RasterPropMonitor" },
        { "name" : "BDArmory"},
        { "name" : "ProceduralDynamics"},
        { "name" : "SmokeScreen"}
    ],
    "resources" : {
        "homepage"   : {
            "url"      : "http://forum.kerbalspaceprogram.com/threads/84054"
        }
    },
    "install" : [
        {
            "file"       : "RetroFuture",
            "install_to" : "GameData"
        },
    ]
}

Thanks!

Edited by nli2work
Link to comment
Share on other sites
Justin Kerbice

Justin Kerbice

Posted
Posted (edited)
There is such a tool available here. It requires Perl and Java to run. We're looking into implementing JSON validation in the CKAN client in the future.

ok.

You also forgot "json-schema-validator-2.2.5-lib.jar" as dependency, I guess.

(Such tool can be used with Cygwin for example, which may sightly differ from a "standard" Linux install, + there is a few chance the java package is install or available as it is not in the standard/base package).

By the way, using CHILD_ERROR ($?) in the script may be bad as java lib not found, exec error etc shows a "JSON validation" error to the user, who is not likely to find the true cause of error.

Using $?>> 8 may be better (+ check the doc of the java stuff to know relevant error codes).

Edited by Justin Kerbice
Link to comment
Share on other sites
pjf

pjf

Posted
  • Author
Posted (edited)

@Justin: I'd *love* something more friendly than our current validator. It's written primarily to target our continuous integration system (travis-ci), and so it's not exactly user-friendly; we don't have to worry too much about exit statuses as *any* error indicates that we need human attention. One of the gotchas for validators is we need to use v4 of the JSON Schema spec, and the java validator is the best one I've found out there. (The dependency is in the git repo, so we recommend just cloning that).

Going forward, we're hoping people can write significantly reduced descriptor files, which get most of their info from KerbalStuff or GitHub. These use a special reference that's described in the latest version of the spec.We're also hoping to make it easier for mods which use a standard directory structure to not require install instructions, which will make things easier still! I'm in the process of writing up better (and *much* simpler) contributor guidelines now, but I've got my hands full with a burst pipe as well, so that may be delayed a little.

@nli2work: That looks *awesome*. Technically you can leave out the homepage (and resources entirely) if you want to use the homepage you have listed on KerbalStuff. I'm going to test your descriptor now to see how it goes. Are you on github at all? (It'll make it easier for me to CC you in on the testing there.)

~ Paul

Edited by pjf
I can speall nli2work's handle, honest!
Link to comment
Share on other sites
pjf

pjf

Posted
  • Author
Posted

@nli2work:

I've got your mod working great with the following:


{
    "spec_version"   : 1,
    "identifier"     : "RetroFuture",
    "$kref"          : "#/ckan/kerbalstuff",
    "depends" : [
        { "name" : "Firespitter" },
        { "name" : "ModuleManager", "min_version" : "2.5.1" }
    ],
    "recommends" : [
        { "name" : "FerramAerospaceResearch" },
        { "name" : "RasterPropMonitor" },
        { "name" : "BDArmory"},
        { "name" : "ProceduralDynamics"},
        { "name" : "SmokeScreen"}
    ],
    "install" : [
        {
            "file"       : "RetroFuture",
            "install_to" : "GameData"
        },
    ]
}

However support for Firespitter is currently an open issue. If you need all of Firespitter, then the code above is correct. If you just need the Firespitter dll, then it's likely that will be called `FirespitterCore`. Because the dll isn't released like a normal mod, it's proving a bit hard to provide packaging for it. This is a good thing for us to be discovering now. :)

Also, do feel free to chime in on any of the issues on github, especially those marked as policy. We're trying to figure out the best way to do a lot of things, and while most policy decisions are relatively easy, there are some which I know we need to get right now because they'll be hard to change in the future.

~ Paul

Link to comment
Share on other sites
pjf

pjf

Posted
  • Author
Posted

Just a bit of a preview, for a *typical* mod (which is one that's on KerbalStuff, and has zipfile which contains a directory with the same name as the module), you can soon get away with:


{
    "spec_version"   : 1,
    "identifier"     : "DogeCoinFlag",
    "$kref"          : "#/ckan/kerbalstuff"
}

That's the *entire* file, and it doesn't even need to be embedded in your zipfile at all, we can just have it in our BootKAN/NetKAN system which does a lot of the automated production of metadata.

That file above will get all the metadata from the KS, and then traverse the download zipfile for a directory called `DogeCoinFlag` (so it will do the right thing work with: `GameData/DogeCoinFlag`, `DogeCoinFlag-1.02/GameData/DogeCoinFlag`, and just `DogeCoinFlag`).

I'm thinking of changing $kref to allow identifiers like "#/ckan/kerbalstuff/269", which means we no longer need an external index of where to locate said data.

More tomorrow when I have time to write!

~ Paul

Link to comment
Share on other sites
nli2work

nli2work

Posted
Posted

However support for Firespitter is currently an open issue. If you need all of Firespitter, then the code above is correct. If you just need the Firespitter dll, then it's likely that will be called `FirespitterCore`. Because the dll isn't released like a normal mod, it's proving a bit hard to provide packaging for it. This is a good thing for us to be discovering now. :)

~ Paul

I'll change it to "FirespitterCore" as RF requires only the DLL. if the flag changes to some other name I'll update it accordingly. I'm still getting familiar with github and figuring out how it works.

Thanks!

Link to comment
Share on other sites
Khatharr

Khatharr

Posted
Posted

Awesome! I'm glad to see this finally happening. I bought Don't Starve on Steam the other day and the way the Steam workshop integrates with the game is really enviable. This also saves me some trouble, since I recently started work on an information aggregator for mod update statuses (to get around the search-through-every-post-of-every-thread-for-every-update problem). This system will solve the problem much more neatly and I don't have to think about how to find a free host for the aggregator, lol.

Link to comment
Share on other sites
pjf

pjf

Posted
  • Author
Posted

@nli2work: Oops, our validation bot doesn't like RetroFuture. In particular, it's very particular about the license, and because KerbalStuff allows free range strings it's not always able to figure out which license is on which mod.

I don't suppose you can change your KerbalStuff license to "CC-BY-NC-SA-4.0"? There's no need for a new release, I can just run the indexer again when done.

Many thanks!

~ Paul

(Also, for those interested, we're popping out new test releases every day or two now.)

Link to comment
Share on other sites
pjf

pjf

Posted
  • Author
Posted (edited)

And our bugfix release (v0.15.1 - Hydrazine Princess) is now out! Not only is the can't-download bug fixed, but all operations are now transactional. If you try to upgrade a mod and it fails (and mod upgrades are still pretty buggy, so they probably will), or you try to install six related mods and one of them barfs, then everything is put back exactly as it was before. I can't explain just how excited I am about this, but I am *incredibly* impressed with C# transactions right now.

Edited by pjf
speeling
Link to comment
Share on other sites
Guest
This topic is now closed to further replies.
 Share
Go to topic listing
×
×
  • Create New...