Combining efforts on proper mod management framework / tools / platform

[TeddyDD]

So our ideas are similar. I thought about three packages like this:

There is R4.0c package (for KSP 0.22)

first version of this package for KSP 0.23.5 appears:

R4.1 (virtual, itself does not contain any files) > depends on R4.0c+community-fix-1 (another package, not installed manually, contains only the fixed files) > depends on R4.0c

USER: Great, Install it.

Manager downloads and installs what you need. and marks the R4.1 package as willed by user.

New version appears

5.0

USER: Cool, Install it!

Manager sees that the package has two dependencies. That dependencies were automatically installed and no other mod need them. They can be safely deleted. (In case of trouble prompt the user.)

Deletes three packages and install the new version - 5.0

Generally, my idea is to use sub packages.

I understand that you do not want to remove any packages when upgrading. It's a good idea because it simplifies this process.

In that case, I'm interested in how R4.0c-community-fix-1 package could look like according to your idea?

Mod file with the patches and fixes applied? Or some sort of meta data?

In any case, I agree with you that the mod with community fixes (those essential to work) should be treated as newer version of this mod.

Edit: I used the additional virtual package so it is be recognized by the manager as the next version of R4.0c and to law and order But of course it could be done without it:

R4.0c

R4.0c-community-fix-1 (or R4.1 for a pretty version number) depends on R4.0c

Edited August 27, 2014 by TeddyDD

[keks]

The idea as a whole consists of three major parts:

• The mod repository
  
• Description files aka meta-data
  
• The client applications
  

The repository:

The repository is represented a as bunch of git repositories containing meta-data for a set of mods.

A mod release is represented as a 'tag' in the repository and (optionally) has the actual release archive attached to it.

In addition to the actual mods, a special 'index' repository exists, listing all mods known to the repository as git-submodules.

Once a mod gets updated, so does its submodule reference in the index.

This will be explained later on, under 'The actual manager application'

The description files:

The description file(s) contain all information the client-application needs to manage a given mod:

• Download link.
  The actual URL to obtain the release archive from - basically the download link from the forums
  
• Legal information like author, copyright, license, ... to display in the client
  
• Dependencies on other mods and KSP version requirements
  
• A complete list of files to install from the release archive
  

The client application(s):

This part actually consists of two separate parts:

developer tools:

The developer tools should assist the maintainer/developer in maintaining the mod and providing him with some common tools to use for modding. Like commonly used libraries, templates, a blank module boilerplate, etc...

The actual manager application:

This application does most of the work. It maintains a list of repositories to get mods / updates from. It keeps track of mods installed and checks them against the repositories for updates.

The client application makes use of git to track the repositories index-repo. When the index has updated on remote, the client also updates its submodules to get the latest meta-data available.

This way a single 'git fetch' can tell if there are any updates available, efficiently reducing the number of API requests required to keep clients up-to-date.

Adding a mod to the repo:

• Some developer "D" puts up a mod on the forums
  
• Some maintainer "M" (may be the developer himself) downloads the release-archive
  
• M creates a new git-repository on GitHub
  
• M creates the 'meta.yaml' file and commits it
  
• M creates a new version tag (e.g. 'v1.0') and pushes it to GitHub
  
• M attaches the release-archive to the release-tag on GitHub
  

At this point the mod is actually already fully usable. What's left to be done is adding it to the index by some "trustworthy" person "T":

• T pulls the current index
  
• T adds the new mod as submodule to the index
  
• T commits and pushes to GitHub
  

Now the new mod is live and automatically available to all clients as new "latest" version.

Such an example repo could look like this (I already linked this earlier):

https://github.com/ksprepo-alt

https://github.com/ksprepo-alt/index

https://github.com/ksprepo-alt/deadlyreentry/blob/aa8720d9366fefda8f1785e2340450fd7f6c1d92/meta.yaml

The meta.yaml linked points to two independent distinct sources for the release archive to download from.

As shown by the md5-checksum the files are identical. The repo here simply acts as a mirror. No files have been altered.

The 'contents' section lists all files included in the archive, that are to be extracted into the 'GameData' directory by the client-application.

Files contained in the archive, but not listed under 'contents' are to be silently ignored (e.g. Documentation, Sources, etc...).

The meta.yaml file also contains proper license and copyright information to be displayed by the client-software in some way.

The repo contents (in most cases only containing the meta.yaml file) are also properly licensed (in this example MIT) and should be accompanied by a proper README-file, auto-generated from the meta-data.

This makes sure our repo can be freely used/forked by others. Such an example README can be seen here: https://github.com/ksprepo/ksp_module-manager/blob/master/README.md

Hotfixing a mod:

When in need to create unofficial hotfixes (like community-made fixes) we can make use of the same scheme:

• M downloads community fix
  
• M pulls current mod repo
  
• M updates 'meta.yaml' to include the community fix' changes
  
• M creates a new tag (e.g. v1.0+ksprepo-1) and pushes it to GitHub
  
• M attaches community-fix.zip to the new release tag
  

The actual download url for the mod itself remains the same.

When the client updates the mod to the new release 'v1.0+ksprepo-1' it follows the usual update-procedure (see 'Updating a mod') but also respects changes introduced by the patch-set.

It effectively combines them before applying it, where the patch-set always takes precedence over the original release.

Updating a mod:

When updating a mod the client compares its local state (for example the contents of the GameData directory) against the file lists of the mods it has installed. It will take care of updating files, as well as removing obsolete ones.

When it detects a conflict (for example a user locally modified a file) it prompts the user to decide what to do.

This is basically exactly what git does when you pull from a remote into your working tree. Download changeset, check conflicts, apply changes.

@Spyhawk:

Does this satisfy your needs for now? As I said I do not have the time to start working on a proper spec until at least Sunday, but I took the time to quickly summarize the current state of our discussion.

Disclaimer: I may have missed something in the hurry here, as I really should be asleep by now... have to get up for work in less than 5 hours again...

Very basic reference implementation: https://github.com/ksprepo-alt/kspmm-prototype

Edited September 16, 2014 by keks

[keks]

[...]I understand that you do not want to remove any packages when upgrading. It's a good idea because it simplifies this process.

In that case, I'm interested in how R4.0c-community-fix-1 package could look like according to your idea?

Mod file with the patches and fixes applied? Or some sort of meta data?

It could look something like this:

Original v5.2 file: https://github.com/ksprepo-alt/deadlyreentry/blob/aa8720d9366fefda8f1785e2340450fd7f6c1d92/meta.yaml

[...]

download:
    - url: 'https://github.com/NathanKell/DeadlyReentry/releases/download/v5.2/DeadlyReentryCont_v5.2.zip'
      md5: 'f70f51778cb8c026bbccd07523016477'

    - url: 'https://github.com/ksprepo-alt/deadlyreentry/releases/download/v5.2/DeadlyReentryCont_v5.2.zip'
      md5: 'f70f51778cb8c026bbccd07523016477'

contents:
    '03d2659490e744b2641ca47ebe6e93f8': 'DeadlyReentry/DeadlyReentry-RealChutes.cfg'
    'bbe1bd3cb63cba5b630ae9c82bc2f011': 'DeadlyReentry/Sounds/fire_damage.wav'
    '01ea0cb76541c1f16f8c71ba09d04098': 'DeadlyReentry/Sounds/gforce_damage.wav'
    'aa0ed53b8b89a0366c53777ff76e23eb': 'DeadlyReentry/Parts/UP_decoupler_2/model000.png'
    ....

patch:
    - download:
            - url: https://github.com/ksprepo-alt/deadlyreentry/releases/download/v5.2-cf-1/5.2-community-fix.zip
              md5: 'bbe1bd3cb63cba5b630ae9c82bc2f011'
      contents:
            '03d2659490e744b2641ca47ebe6e93f8': 'DeadlyReentry/DeadlyReentry-RealChutes.cfg'
      remove:
            - 'DeadlyReentry/Sounds/gforce_damage.wav' 

In this example the patch would be committed to the repo as version/tag 'v5.2+ksprepo-1'.

'+ksprepo' being our suffix, '-1' being the version counter (I took Debian's approach as reference here).

When the clients sees there is an update available, it simply applies everything under 'patch' in the order specified over the original release.

Internally it simply merges 'contents' and all 'patch/.../contents' to one single list (ofc. also respecting the 'remove' entries) and treats the generated list as a normal release.

In the example above that would replace 'DeadlyReentry/DeadlyReentry-RealChutes.cfg' with a new version and delete 'DeadlyReentry/Sounds/gforce_damage.wav'.

So we actually do not distribute patched releases, we distribute only differential updates / patches.

Edited August 28, 2014 by keks

[TeddyDD]

Hotfixing a mod: [..]

This description convinced me. Forget about my previous post. Now I understand what you mean, and damn ... this is simple and great!

I have a few comments, but this is rather details.

I think that the client should care about hashing files when installing mod and then save checksums somewhere.

Package file should be simple enough to make it possible to create it manually (excluding the checksum xD)

It is still a matter to discuss. If you want these checksums I will not argue about them

I do not like skipping files like readme, or example crafts etc. That king of files in the archive must be marked and manager will decide what to do with them.

This will be simple:

contents:
    '03d2659490e744b2641ca47ebe6e93f8': 'DeadlyReentry/DeadlyReentry-RealChutes.cfg'
    'bbe1bd3cb63cba5b630ae9c82bc2f011': 'DeadlyReentry/Sounds/fire_damage.wav'
    '01ea0cb76541c1f16f8c71ba09d04098': 'DeadlyReentry/Sounds/gforce_damage.wav'
documentation:
   "imagine I'm checksum": "DeadlyReentry/readme.txt"

etc.

To hash archive better use SHA1 (just in case)

The whole project is starting to look more and more real.

I need to start learning C# in order to do something useful

Edited August 28, 2014 by TeddyDD

[keks]

I think that the client should care about hashing files when installing mod and then save checksums somewhere.

Package file should be simple enough to make it possible to create it manually (excluding the checksum xD)

It is still a matter to discuss. If you want these checksums I will not argue about them

That's where the dev-tools come into play. The meta.yaml file can mostly be generated automatically using a simple script.

This would also greatly reduce the possibility to mess up things wen creating/updating the meta-data.

I do not like skipping files like readme, or example crafts etc. That king of files in the archive must be marked and manager will decide what to do with them. [...]

You're correct. Some users may prefer to have them installed to their usual location. We could split up contents into categories and let the user decide which categories to install:

contents:
    core:
        '03d2659490e744b2641ca47ebe6e93f8': 'DeadlyReentry/DeadlyReentry-RealChutes.cfg'
        'bbe1bd3cb63cba5b630ae9c82bc2f011': 'DeadlyReentry/Sounds/fire_damage.wav'
        '01ea0cb76541c1f16f8c71ba09d04098': 'DeadlyReentry/Sounds/gforce_damage.wav'
    crafts:
        ...
    docs:
        ...

This way we can easily add categories over time.

To hash archive better use SHA1 (just in case)

MD5 is a bit faster than SHA1. Given a large amount of mods/files and a slow client-pc this could greatly reduce computation time when updating mods.

The probability of two hashes colliding is something about 1/(2^128) for both, MD5 and SHA1.

As we only use it as checksum (and not as crypto-function) it really does not matter if we chose MD5 or SHA1 as they both create 128bit sums.

If we really encounter problems with collisions, we can still switch to a longer checksum.

Now I really have to get my sleep. have a good night

[Justin Kerbice]

@keks, and others: things have become quite serious glad to see that, but in the other hand, debian packaging system was not a one day work and it has evolve with time and features needed. So I suggest doing the same: a first basic step which allow a single mod per package to work without even dependencies, then adding feature one at a time.

Including license text is a pure waste of space and bandwidth, as there is just a few licenses for thousands of mods, license name/version is more than enough. + add a link to the full license. OR you'll get something as ridiculous as some software with license data is bigger than useful data.

and this:

Some maintainer "M" (may be the developer himself) downloads the release-archive

creates a new git-repository on GitHub

creates the 'meta.yaml' file and commits it

creates a new version tag (e.g. 'v1.0') and pushes it to GitHub

attaches the release-archive to the release-tag on GitHub

looks to be a lot's of work which can be done by a script (create repo, create a dummy meta.yaml file, create release tag, at least)

[keks]

Including license text is a pure waste of space and bandwidth, as there is just a few licenses for thousands of mods, license name/version is more than enough. + add a link to the full license. OR you'll get something as ridiculous as some software with license data is bigger than useful data.

And how would you then handle custom licenses like? Many authors add further restrictions to their license or simply create their own license.

Proper licensing is a must, and nowadays 1 kByte of data really is not that much

By the way, I in fact did only put in name and version of the license when a unmodified standard license was used:

license:
              - name: "CC BY-NC-SA 4.0"
                url: "http://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.txt"

looks to be a lot's of work which can be done by a script (create repo, create a dummy meta.yaml file, create release tag, at least)

Well, it actually is not really a lot of work. Everything can be done in less than 2 minutes if you're quick at typing. But As I already said:

That's where the dev-tools come into play. The meta.yaml file can mostly be generated automatically using a simple script.

This would also greatly reduce the possibility to mess up things wen creating/updating the meta-data.

In addition to generation of the meta-data file, uploading, tagging, etc can ofc. also be done via a simple script.

[TeddyDD]

Including license text is a pure waste of space and bandwidth, as there is just a few licenses for thousands of mods, license name/version is more than enough. + add a link to the full license. OR you'll get something as ridiculous as some software with license data is bigger than useful data.

Agree. Besides package licence must be clearly separated from mod licence.

I suggest

package-licence:
    -name: "MIT License"
     url: "http://opensource.org/licenses/MIT"

instead

license:
    - name: "MIT License"
      plaintext: &mit_license |
       The MIT License (MIT)
       Permission is hereby granted ...

[Justin Kerbice]

And how would you then handle custom licenses like? Many authors add further restrictions to their license or simply create their own license.

Proper licensing is a must, and nowadays 1 kByte of data really is not that much

Standard licenses can use a link to the related website, custom one have instead plain text, as simple as that .

so license could used: name, link (optional), text (optional) but one of the two last is mandatory.

[Spyhawk]

Does this satisfy your needs for now? As I said I do not have the time to start working on a proper spec until at least Sunday, but I took the time to quickly summarize the current state of our discussion.

Much better, thx! Will reread this summary tonight and might come up with additional comments.

Standard licenses can use a link to the related website, custom one have instead plain text, as simple as that .

so license could used: name, link (optional), text (optional) but one of the two last is mandatory.

I'd even say a link is too much. Simply use the name of the license (like "license: MIT"), or for custom license, simply use "license: custom". If the user needs to know more, he could still check the mod forum URL, the mod archive, etc.

[keks]

Standard licenses can use a link to the related website, custom one have instead plain text, as simple as that .

so license could used: name, link (optional), text (optional) but one of the two last is mandatory.

That's acutally exactly what I am doing in my examples ;-)

Don't mix up the license of the repo with the mod license. The mod license is below repo license under the 'legal/original' key.

PS: I also just noticed that I messed up some white-space there... 'source' is actually ment to be on the same level the other keys under 'original' are

I'd even say a link is too much. Simply use the name of the license (like "license: MIT"), or for custom license, simply use "license: custom". If the user needs to know more, he could still check the mod forum URL, the mod archive, etc.

Agree. Besides package licence must be clearly separated from mod licence.

It actually is a requirement to include a proper license when redistributing content. For GPL licensed content, you have to either ship the full license text or provide a link where users can download the full text. I don't know about CC and BSD, but I'm pretty sure they handle it in a similar way. Also for example BSD-2-Clause (and others) requires you to include the original license notice. The same goes for MIT-license. You have to include the original copyright/license notice.

I did not make this clear earlier, but my example file file plaintext/url entries are mutually exclusive, but one of them is always required.

Mods using unmodified license texts can simply supply the license name and URL, where modified licenses must provide the full license text and/or the original license they build on. For example some developers here use the BSD-2-Clause, but add further restrictions.

Instead of putting in plaintext license into the meta-file, you could also put your license into a file and then use the 'url' key to point to that file. It really does not matter IMHO.

But fact is, we must include those unchanged license/copyright notices and must provide a proper full-text version of said license. This is a requirement most licenses impose upon derived work, and I see no reason to not make this mandatory for all mods. I'd rather have a link too much in there than always having to figure out if I need it or not.

Edited August 28, 2014 by keks

[Spyhawk]

It actually is a requirement to include a proper license when redistributing content. For GPL licensed content, you have to either ship the full license text or provide a link where users can download the full text. I don't know about CC and BSD, but I'm pretty sure they handle it in a similar way. Also for example BSD-2-Clause (and others) requires you to include the original license notice. The same goes for MIT-license. You have to include the original copyright/license notice.

I was only referring to meta-data (the .yaml files). Licenses can be shipped in the archives themselves, or if needed included in a general dummy packages as done by Linux distributions. But please, not in the meta-data (apart from the license name).

[keks]

I was only referring to meta-data (the .yaml files). Licenses can be shipped in the archives themselves, or if needed included in a general dummy packages as done by Linux distributions. But please, not in the meta-data (apart from the license name).

Well, I still see no reason to not put it in there, but OK. I'd suggest to putting it into a file called 'LICENSE' (to conform with GPL requirements) located in the root of the repo.

Edit:

I updated the example repo i linked ealier:

Such an example repo could look like this (I already linked this earlier):

https://github.com/ksprepo-alt

https://github.com/ksprepo-alt/index

https://github.com/ksprepo-alt/deadlyreentry/blob/master/meta.yaml

Edited August 28, 2014 by keks

[Spyhawk]

Well, I still see no reason to not put it in there, but OK. I'd suggest to putting it into a file called 'LICENSE' (to conform with GPL requirements) located in the root of the repo.

Because the meta-data is just that, meta-data. It's the interface between the server side and client side of the "package management". Why would dull copies of licenses be there?

Also, the licenses should already be included in the archives releases. It's not the package manager job to do the legal stuff if the upstream haven't done it properly (again, I see little value anything other than the .yaml/.json files with the bare strict minimum required to allow the package manager to work).

[keks]

Also, the licenses should already be included in the archives releases. It's not the package manager job to do the legal stuff if the upstream haven't done it properly.

Problem is most devs do not include a proper license in their release archive. Instead they mention license on the forums only. That's a problem for us, because when not shipping a proper license, we actuall violate the dev's copyright.

As we agreed on not modifying the original release, shipping it outside the release archive really is the only way that's left.

[micha]

Good to see the discussion moving along nicely.

Also, I think it was just a terminology issue - the ideas of a "community-fix" package and a "patch-package" are exactly the same (in my mind) - that is, from the package manager's perspective, they are both "real" packages and will follow the same installation/upgrade paths. I'm not sure why people thought there was a difference.

This is why I said the technicalities (what and how exactly do we upgrade from one version of a mod to another) need to be defined, but at a high level, it's simply a progression of versions and installs/upgrades.

Licenses are a necessary evil unfortunately, and yes, we must include them if the upstream mod doesn't.

Also, I'm away until Sunday.

[keks]

Good to see the discussion moving along nicely.

Also, I think it was just a terminology issue - the ideas of a "community-fix" package and a "patch-package" are exactly the same (in my mind) - that is, from the package manager's perspective, they are both "real" packages and will follow the same installation/upgrade paths. I'm not sure why people thought there was a difference.

Well, there acutally is a little difference. Official packages only contain the original release archive. Unofficial patches contain the original release archive and a the unofficial patch-set that gets applied over the original release. I explained my proposal earlier: http://forum.kerbalspaceprogram.com/threads/85989-Combining-efforts-on-proper-mod-management-framework-tools-platform?p=1378736&viewfull=1#post1378736

From perspective of the user, there is no difference between them. That's correct.

[TeddyDD]

I'd suggest to putting it into a file called 'LICENSE' (to conform with GPL requirements) located in the root of the repo.

Very good solution, actually a compromise between simple packages and copyright laws.

EDIT:

I thought of our prototype Python application.

If it would be good enough (I mean very good) we could try to use cx_Freeze to create standalone executables for Window, Mac and Linux.

Of course Python program usually has slightly worse performance. But developers time is more valuable than computer time.

Python is a good language, and if this way would work I see no reason to rewrite everything in C#

Edited August 28, 2014 by TeddyDD

[keks]

Python is a good language, and if this way would work I see no reason to rewrite everything in C#

Python definitely is a nightmare to build GUI (or basically any multi-threaded) applications with. Python does not support running multiple threads in parallel due to it's GIL.

So when the application is working, the GUI hangs. The only way around this is multi-processing which has it's problems on it's own...

Also writing it in Pyhton would add the dependency on the python interpreter and libraries.

Even if cx_freeze was working reliably (which it does not) it would be at least lets say "adventurous".

I really prefer building the final application in a proper language which is properly supported on all of our target platforms without the need of strange hacks and workarounds.

We'd have to rewrite the application anyway, because the prototype is just meant to be a prototype.

Rewriting an application from scratch usually leads to cleaner, more sane code as you won't (or should not) do the same mistakes again, you did when writing your prototype

Edit:

However, if you guys really want to use Python instead of C#/Mono, I could accept IronPython, which is a .Net/Mono implementation of the python specification.

IronPython compiles python code down to .net Bytecode which in turn can be run using the .Net/Mono runtime.

But that does not really feel right to me. Mods are written in C#, so should our application be written in C#. No need to introduce a new language here, IMHO.

Edited August 28, 2014 by keks

[TeddyDD]

I would argue... but just tried to build cx_Freeze against Python 3.4 No way :> What a crap xD

Personally, I wish we wrote this in my favourite Golang xD But guess what? There isn't native Go GUI library and using bindings is... funny :> (It is not a proposal of course, just my adoration for this language)

I just hope that Mono/C# stuff will work properly on all systems

Rewriting an application from scratch usually leads to cleaner, more sane code as you won't (or should not) do the same mistakes again, you did when writing your prototype

Yeah... true

Edit: IronPython is still python 2.7 :| Since we want to use Mono/.NET IMO better let's stay with C#

Edited August 28, 2014 by TeddyDD

[keks]

I just hope that Mono/C# stuff will work properly on all systems

Well, KSP does. At least kind of. As long as we stick to Mono only, things should be fine.

External libraries will most certainly bring along some problems though.

Edited August 28, 2014 by keks

[TeddyDD]

Keywords are "kind of" I tried to play on Linux once and it was awful

But I think We can manage to do it

[keks]

I'm actually running KSP on Linux perfectly fine. It's only some mods that are "Windows only", but they usually mention that on their forum posts.

(The .23.5 x86_64 build was a huge mess though, always running into memory issues and segfaults...)

[TeddyDD]

That "once" was on 0.23.5 I have got nvidia optimus card what also makes trouble (ksp doesn't like optirun)

Also I use a lot of mods. Actually, that's why I became interested in this topic

[Justin Kerbice]

Of course Python program usually has slightly worse performance. But developers time is more valuable than computer time.

Python is a good language, and if this way would work I see no reason to rewrite everything in C#

Not agree, "worse" compared to what ? assembly (the one no one use anymore, almost).

Python is quite fast in fact and it is pre-compiled/use a bytecode interpreter (as Perl for example) which can be more than enough, especially for a tool which doesn't care about speed.

Also, it is use for Asseto Corsa plug-in, just to say, and Kerbal Stuff too.

I may be interested to join .

@keks: are you sure about what you sad about python (horrible with GUI, not multithreaded, ...) ?

Do you know which language is used for Blender ?

Edited August 30, 2014 by Justin Kerbice