CKAN (The Comprehensive Kerbal Archive Network); v1.28.0 - Dyson

[Lewie]

Thanks guys! I’m not a techno wizard, so a ton of this is to me.

@Gargamel that’s my bad. I was a bit borderline on where to put it, I felt that it could fit under both ‘technical support’ and he ckan thread.

[Gargamel]

1 hour ago, Lewie said:

I was a bit borderline on where to put it, I felt that it could fit under both ‘technical support’ and he ckan thread.

Not a problem.  If you are asking a question about a tool or mod, it's usually best to ask directly within the existing thread on that topic.  If you aren't sure, post it where you think is best, and then report the thread asking for guidance, we'll take a look and help you figure it out. 

[Fougère_gs]

Bonjour à tous, je me suis rendue conte d'un problème que j'avais sur ckan, le ksp qui est sur ckan est en 1.9.1 alors que ma version steam du jeux est en 1.10.1 l'emplacement du jeu est aussi pas le même, sur ckan est est dans "C / programex86 /  Steam / Steamapps / common / KSP" et sur steam il est dans "D / Game / Steam / Steamapps / common". J'aimerais bien que quelqu'un me dépanne de ce problème.

 En le/la remerciant d'avance  

[Gargamel]

Quote

Hello everyone, I realized a problem I had on ckan, the ksp which is on ckan is in 1.9.1 while my steam version of the games is in 1.10.1 the location of the game is also not the same, on ckan is is in "C / programex86 / Steam / Steamapps / common / KSP" and on steam it is in "D / Game / Steam / Steamapps / common". I wish someone could help me out with this problem.

Thanking him / her in advance

Welcome to the Forums @Fougère_gs.  We ask that you post questions in English out side of the international forums.   Feel free to post in your preferrred language and then an automated translation to English if you need.  

Also your thread has been merged into the CKAN thread. 

[Mercor]

On 7/27/2020 at 8:04 PM, HebaruSan said:

See: https://github.com/KSP-CKAN/CKAN/issues/3115

Yeah, I posted in there and even did the investigation on my machine that probably got a bit clarity what was causing the issues. I believe we had a chat in discord while I tested different options and building it on my machine to see if anything was fixing it.

[Dunbaratu]

I have a question about how the CKAN bot crawler works.

The kOS mod is already in CKAN, but I need to add a new dependency when I make the next release (The next release will depend on ClickThroughBlocker.  It didn't need it before.)

I know what edits to make to the netkan file to describe the dependency, but I don't know how to get CKAN's bot to see the change.  Currently the .netkan file isn't part of the ZIP at all (it's edited "one folder up" from where the ZIP is cut from) so I assume the bot can't possibly see that information.  I'm guessing that previously the mod was manually added to CKAN with a pull request but I'm trying to figure out how to make it automate the updates by having the CKAN bot see the information somewhere inside the ZIP file, like it does for the .version file.  (This is a mod I inherited from others.  I was not the person who originally set it up in CKAN so I don't know what steps were taken at the time.)

So my question is this - if I just start using netkan.exe to convert the .netkan file into a .ckan file, and copy that .ckan file to GameData/kOS/whatevername.ckan, inside the ZIP will that automagically make the bot update kOS's CKAN info when I make a release  with that ZIP next time?  Or are there other manual steps I need to do first?

Up until now, we *have* been getting automatic updates in CKAN from the information in the ZIP's "kOS.version" file, but this is a thing the .version file doesn't have in it (dependency on another mod).

[HebaruSan]

49 minutes ago, Steven Mading said:

I know what edits to make to the netkan file to describe the dependency, but I don't know how to get CKAN's bot to see the change.

If you edit https://raw.githubusercontent.com/KSP-KOS/KOS/master/kOS.netkan, that would do it.

49 minutes ago, Steven Mading said:

Currently the .netkan file isn't part of the ZIP at all (it's edited "one folder up" from where the ZIP is cut from) so I assume the bot can't possibly see that information.

It does, though, because it's set up to crawl your .netkan file as what we call a "metanetkan":

• https://github.com/KSP-CKAN/NetKAN/blob/master/NetKAN/kOS.netkan#L4

Every 30 minutes,. CKAN's bot checks that file and attempts to generate new metadata from it.

49 minutes ago, Steven Mading said:

having the CKAN bot see the information somewhere inside the ZIP file, like it does for the .version file. 

... that can be done. Yes, if you put a .ckan file inside your ZIP file, it will be parsed for CKAN metadata. This has some unique advantages, insofar as it is guaranteed to always be associated with the correct download file (I do this with my own mods). But...

49 minutes ago, Steven Mading said:

Or are there other manual steps I need to do first?

Yes, we would need to remove any properties you intend to set from the .netkan file and your metanetkan, since they will take precedence otherwise. The "internal .ckan file" is the lowest priority.

The easy path is to release the new release, then update the .netkan file as needed. Bleeding-edge users upgrading in the meantime might get incomplete metadata, but ultimately everything would work out.

[Dunbaratu]

31 minutes ago, HebaruSan said:

If you edit https://raw.githubusercontent.com/KSP-KOS/KOS/master/kOS.netkan, that would do it.

It does, though, because it's set up to crawl your .netkan file as what we call a "metanetkan":

• https://github.com/KSP-CKAN/NetKAN/blob/master/NetKAN/kOS.netkan#L4

Every 30 minutes,. CKAN's bot checks that file and attempts to generate new metadata from it.

... that can be done. Yes, if you put a .ckan file inside your ZIP file, it will be parsed for CKAN metadata. This has some unique advantages, insofar as it is guaranteed to always be associated with the correct download file (I do this with my own mods). But...

Yes, we would need to remove any properties you intend to set from the .netkan file and your metanetkan, since they will take precedence otherwise. The "internal .ckan file" is the lowest priority.

The easy path is to release the new release, then update the .netkan file as needed. Bleeding-edge users upgrading in the meantime might get incomplete metadata, but ultimately everything would work out.

Thanks for the quick reply.

Will it be smart enough to NOT apply the new dependancy to older versions of kOS if I just continue with the .netkan file technique as it is using now?  Or do I need to switch to the .ckan file in the ZIP to get that kind of version-specific dependency?

[HebaruSan]

3 minutes ago, Steven Mading said:

Thanks for the quick reply.

Quote

Will it be smart enough to NOT apply the new dependancy to older versions of kOS if I just continue with the .netkan file technique as it is using now?  Or do I need to switch to the .ckan file in the ZIP to get that kind of version-specific dependency?

The bot processes only one version, the latest version, at a time. So if you release a new version and then update the metanetkan soon afterwards, old versions will be unaffected, and the latest version will take the changes as the bot sees them. Does that help?

Edited August 7, 2020 by HebaruSan

[Dunbaratu]

1 minute ago, HebaruSan said:

The bot processes only one version, the latest version, at a time. So if you release a new version and then update the metanetkan soon afterwards, old versions will be unaffected, and the latest version will take the changes as the bot sees them. Does that help?

So if do it in the reverse order from that, such that I merge the .netkan change into the master branch a few hours *before* I've cut that release from master, will it end up making the previous release of kOS get a dependency on ClickThroughBlocker (because the newer release wasn't there yet when the Netkan bot crawled every half hour and saw the kOS.netkan change in master branch)?

If so, then I have to change how I'm doing my PR for this change and split it into two PRs - one for the C# code that calls ClickThroughBlocker's API and another for the kOS.netkan dependency on ClickThroughBlocker that code caused.  (Right now they're the same PR which seems sensible at first unless it would cause the scenario I described in the previous paragraph).

It matters because my dependency requires a minimum ClickThroughBlocker version number to have the features LinuxGuruGamer just added a few months ago that I'm using.  If the previous release of kOS ends up falsely claiming it has a hard dependency on this new version of ClickThroughBlocker, that could prevent CKAN from letting you install it on older versions of KSP that only allow older versions of ClickThroughBlocker.  That's why I'm nervous about making sure I do this right.

[HebaruSan]

3 minutes ago, Steven Mading said:

So if do it in the reverse order from that, such that I merge the .netkan change into the master branch a few hours *before* I've cut that release from master, will it end up making the previous release of kOS get a dependency on ClickThroughBlocker (because the newer release wasn't there yet when the Netkan bot crawled every half hour and saw the kOS.netkan change in master branch)?

Yes. Whatever the bot finds when it attempts to inflate a mod, is what it will index.

[steve_v]

@HebaruSan How do I set environment variables when launching KSP from CKAN? Setting them in the "KSP command-line arguments" dialog as I would with any other "launch command" setting (steam, lutris, etc. etc.), e.g. 'LC_ALL=C ./KSP.x86_64' results in "Couldn't start KSP. Cannot find the specified file".

I could certainly launch it via a shell script I suppose, but that seems kinda redundant if it's only purpose is to set an env var...

[TranceaddicT]

3 hours ago, steve_v said:

@HebaruSan How do I set environment variables when launching KSP from CKAN? Setting them in the "KSP command-line arguments" dialog as I would with any other "launch command" setting (steam, lutris, etc. etc.), e.g. 'LC_ALL=C ./KSP.x86_64' results in "Couldn't start KSP. Cannot find the specified file".

I could certainly launch it via a shell script I suppose, but that seems kinda redundant if it's only purpose is to set an env var...

There's not much you should need in the CLI.

This is mine: KSP_x64.exe -single-instance -mm-dump

Since it looks like you're running linux try ./KSP.x86_64 -single-instance -mm-dump

(The 'mm-dump' give you a folder in the rootKSP that mirrors GameData but with the final cfg files as used in game.)

[Corax]

I know I'm not being helpful here, but that being redundant functionality is exactly why I set environment from the same shell script that backs up save games and ships, renames screenshots in a sensible pattern, updates and launches CKAN, deletes MiniAVC and other nonsense, corrects file permissions (all files are executable as decreed by Squad), and eventually also launches KSP... ; )

[steve_v]

[snip]

What I do have a use for is launching KSP with environment variables set, as such variables are used extensively for a great many things on POSIX systems, and have been since the 1980s.
These cannot be set by command-line options to KSP, because KSP knows nothing about them - they're environment variables, that is setting up the environment in which KSP is launched.
In my case, I need to set variables for mangohud (overlay), as well as pass some antialiasing and anisotropic filtering overrides to the graphics driver. I'd like to set LC_COLLATE to "en_US" too, so that save names are not sorted based on capitalisation.
I can probably think of others that I might want, and none of them are accepted in CKANs as part of a launch command. In fact it appears to reject anything that doesn't start with a (single) literal filename.

 

2 hours ago, Corax said:

I set environment from the same shell script that backs up save games and ships, renames screenshots in a sensible pattern, updates and launches CKAN, deletes MiniAVC and other nonsense, corrects file permissions (all files are executable as decreed by Squad), and eventually also launches KSP

I have a similar script that uses symlinks to provide switchable mod sets, compresses and restores backups, etc. etc. (actually it's more like ~300 lines of me playing with bash arrays and universal wrapper functions for dialog, kdialog and zenity because I got bor... Okay, I admit, it's 95% utility functions I should split out into a library ).
This script's purpose is to set up the game files and directory structure for both KSP and ckan, and it is intentionally a standalone tool. What if I want to just lazy-click "ckan.exe" from my GUI file manager? I'd need yet another shellscript just as a wrapper.
CKAN not supporting environment variables in it's launch command is clearly the result of a "file exists" check, and one that I would personally consider redundant (and a windoze-ism to boot). Every other application I have installed supports setting environment when launching external commands (and other shell syntax, as is intuitive on a posix system), why not this one?
 

Edited August 11, 2020 by Vanamonde

[HebaruSan]

59 minutes ago, steve_v said:

CKAN not supporting environment variables in it's launch command is clearly the result of a "file exists" check

Nope. It just calls .NET/Mono's Process.Start function.

https://github.com/KSP-CKAN/CKAN/blob/f0f02d68f52603c3c43295f94b5d942c93cd60f4/GUI/Main/Main.cs#L679-L724

Remember that CKAN is cross-platform. The (shell) syntax you want would be platform-specific.

Also remember that environment variables are inherited from parent process to child process. You could set them before/as you launch CKAN.

[steve_v]

22 minutes ago, HebaruSan said:

It just calls .NET/Mono's Process.Start function.

Bugger. I kinda suspected that limitation was from mono rather than something explicit in CKAN, oh well.
And mono apparently doesn't include such platform-specific functionality, even when compiled on and for that platform... Mildly disappointing, but not surprising.
 

22 minutes ago, HebaruSan said:

environment variables are inherited from parent process to child process. You could set them before/as you launch CKAN.

I know, that's my current workaround. It's annoying to have to remember not to just click ckan.exe from the file manager is all, passing vars in the launch command would be convenient.

 

Ed. @TranceaddicT From the CKAN source code:
 

// -single-instance crashes KSP 1.8 to KSP 1.9 on Linux
// https://issuetracker.unity3d.com/issues/linux-segmentation-fault-when-running-a-built-project-with-single-instance-argument

 

Edited August 11, 2020 by steve_v

[Vanamonde]

Some remarks have been removed. A fellow KSP player who does not give you the answer you wanted is still a fellow KSP player who is trying to help. There's no call to get snippy about it. 

And if you do see someone getting snippy, please just hit the report button on the post rather than reply yourself. Let the moderators deal with the unpleasantness while you get on with enjoying the forum. 

[HebaruSan]

33 minutes ago, steve_v said:

Ed. @TranceaddicT From the CKAN source code:

// -single-instance crashes KSP 1.8 to KSP 1.9 on Linux
// https://issuetracker.unity3d.com/issues/linux-segmentation-fault-when-running-a-built-project-with-single-instance-argument

I somewhat skimmed the above conversation; do we need to extend that workaround to 1.10?

[steve_v]

39 minutes ago, HebaruSan said:

I somewhat skimmed the above conversation; do we need to extend that workaround to 1.10?

You didn't miss much.

To answer your question: That would probably be a good idea.

KSP  (1.10.1) starts normally if I pass -single-instance from an interactive shell... But I still can't launch it from CKAN with that particular argument.

No visible segfault, all I get is:

$ mono ./ckan.exe 
Found path: /home/steve/Games/KSP_linux/./KSP.x86_64

That line is from KSP/Unity IIRC, but I get no window, no KSP.x86_64 process, and no further output.

I can't launch it from a bash script with -single-instance either, but this time I do get an obvious segfault:

$ cat ./ksp.sh 
#!/bin/bash
./KSP.x86_64 -single-instance

$ ./ksp.sh 
Set current directory to /home/steve/Games/KSP_linux
Found path: /home/steve/Games/KSP_linux/KSP.x86_64
./ksp.sh: line 2: 32303 Segmentation fault      ./KSP.x86_64 -single-instance

 

I'm not sure if mono/CKAN is eating stderr somehow and I'm just not seeing the segfault, but either way it looks like launching KSP on GNU/Linux with that particular argument is still broken.

Aside, if you are eating stderr, could you please, uh, not? It's kinda important information.

 

 

Edited August 11, 2020 by steve_v

[Some Random Rocketeer]

Just need a working link.

[Gargamel]

Welcome to the Forums @Some Random Rocketeer, I've moved your question to the CKAN thread, where you will find a working link in the OP. 

[HebaruSan]

3 hours ago, darthgently said:

What is your guess on when it will be on CKAN?  No rush, just don't want to keep checking until I need to

This has changed over time. A summary of the current state of the art:

Scenario	Time till update
SpaceDock with CKAN checkbox checked	Immediately(ish)
SpaceDock with an active version file	Within 30 minutes
GitHub or other hosts	Within 30 minutes
SpaceDock otherwise	Up to 24 hours

SCANsat is in the second category, hence the difference between v20.4's release date timestamp (17:27 UTC) and when it was indexed (17:44 UTC) is approximately 17 minutes:

• https://github.com/KSP-CKAN/CKAN-meta/blob/master/SCANsat/SCANsat-v20.4.ckan

The first group in the table is handled via a webhook notification mechanism by which SpaceDock instructs the CKAN infrastructure to check the mod for new releases right after an upload finishes, which only happens if the "Add to CKAN" checkbox is checked on SpaceDock for that mod. If the background jobs mentioned below are not active, this happens immediately, otherwise it waits for the background job to finish (up to 10 or so minutes).

The second and third groups are handled by a background job that checks those mods for updates every 30 minutes. The exact timing will vary based on the scheduling of the background job when the release is created and the position of the mod in the overall queue (currently alphabetical).

The fourth group is handled by a similar daily background job. The idea is that simple SpaceDock mods don't need to be checked every 30 minutes because we should receive instant notifications for them, but in case the webhook breaks down on either the SpaceDock or CKAN side, there's a fallback mechanism to catch up.

(There is a fifth group consisting of only my two mods, which are hosted on GitHub but are enabled for instantaneous updates via some special magic to which non-CKAN team members do not have access, but it's safe to ignore that for purposes of an overall summary.)

[darthgently]

2 minutes ago, HebaruSan said:

This has changed over time. A summary of the current state of the art:

Scenario	Time till update
SpaceDock with CKAN checkbox checked	Immediately(ish)
SpaceDock with an active version file	Within 30 minutes
GitHub or other hosts	Within 30 minutes
SpaceDock otherwise	Up to 24 hours

SCANsat is in the second category, hence the difference between v20.4's release date timestamp (17:27 UTC) and when it was indexed (17:44 UTC) is approximately 17 minutes:

• https://github.com/KSP-CKAN/CKAN-meta/blob/master/SCANsat/SCANsat-v20.4.ckan

The first group in the table is handled via a webhook notification mechanism by which SpaceDock instructs the CKAN infrastructure to check the mod for new releases right after an upload finishes, which only happens if the "Add to CKAN" checkbox is checked on SpaceDock for that mod. If the background jobs mentioned below are not active, this happens immediately, otherwise it waits for the background job to finish (up to 10 or so minutes).

The second and third groups are handled by a background job that checks those mods for updates every 30 minutes. The exact timing will vary based on the scheduling of the background job when the release is created and the position of the mod in the overall queue (currently alphabetical).

The fourth group is handled by a similar daily background job. The idea is that simple SpaceDock mods don't need to be checked every 30 minutes because we should receive instant notifications for them, but in case the webhook breaks down on either the SpaceDock or CKAN side, there's a fallback mechanism to catch up.

(There is a fifth group consisting of only my two mods, which are hosted on GitHub but are enabled for instantaneous updates via some special magic to which non-CKAN team members do not have access, but it's safe to ignore that for purposes of an overall summary.)

Thank your for the detail, very much appreciated

[HebaruSan]

21 minutes ago, HebaruSan said:

Scenario	Time till update
SpaceDock with an active version file	Within 30 minutes

handled by a background job that checks those mods for updates every 30 minutes. The exact timing will vary based on the scheduling of the background job when the release is created and the position of the mod in the overall queue (currently alphabetical).

A minor clarification on this as my thoughts finish collecting themselves. Including this group in the every-30-minutes passes is not done to handle new releases (though in a few cases it does do that as a side effect, as with SCANsat). Rather it is done because version files have a "remote version file" feature that gives the author the opportunity to change a mod's compatibility post-release, which must be reflected in the CKAN metadata, and there is no on-demand notification available for this. So in addition to the webhook, we need to check such mods regularly in case the remote version file has been updated.