Combining efforts on proper mod management framework / tools / platform

[keks]

@keks: are you sure about what you sad about python (horrible with GUI, not multithreaded, ...) ?

Do you know which language is used for Blender ?

Well I am no blender user, but AFAIK blender itself is written in C/C++ and only the scripts are Python. And that's exactly where Python (or even more often Lua) is often used at, and where it can play it's strengths.

When writing GUI you usually divide your application into at least two (mostly) independent parts: the GUI and the logic. Both run their own threads, so some long-taking operation in the logic does not block the GUI, but you can still easily share data between logic and GUI. Python cannot easily do this, because it can only run one thread at a time due to it's GIL [1]. That causes the GUI to freeze, even when you run it in its own thread. The only way around this is to use multiple processes, which in turn cannot easily share data anymore...

Edit: For the sake of completeness: The above is actually not 100% accurate. You could also use external (C/C++) libraries for displaying your GUI and manually control the GIL then. But then you must make sure your GUI code never ever manipulates python memory in any way, or bad things will happen... And that's something you cannot guarantee that easily if you still want to be able to interact with your GUI

End Edit

[1]: https://wiki.python.org/moin/GlobalInterpreterLock

Disclaimer: when talking about "Pyhton" I actually mean the "CPython" implementation

CPython is pretty fast, but not even close to a native implementation. That's why many Python modules are written in C. To get a rough idea of the speed differences of various languages take a look at this: http://onlyjob.blogspot.de/2011/03/perl5-python-ruby-php-c-c-lua-tcl.html .

But keep in mind that "speed" is also very dependent on the actual implementation and algorithms used. That's why Perl scores so high at the site I linked. Perl makes heavy use of highly optimized native modules (Perl modules actually written in C). It also was created to efficiently handle strings, and the "benchmark" used at the site I linked does measure string manipulation time. Perl has had 25 years to optimize for exactly that use case

Edited August 30, 2014 by keks

[Justin Kerbice]

mmm I see, I've never have a good feeling with Python (a lanaguage that's force you with the indentation is not good IMHO) but performances anyway are mostly irrelevant for the target application here, don't forget whatever the job of packaging/processing text files/creating file structure & dummy files can take 1 or 5 s, it will not change so much, it's not a physics simulation after all so the good thing to do is to take the fastest way to create/maintain/update code.

And there, interpreted languages get the first place, Perl mostly, especially with all the available modules to do almost anything you need to do on any OS you work on. Python main weakness IMHO is the lack of something like CPAN. And the GIL thing appears to me as quite a failure on its main purpose.

The benchmark you give is a flawed as possible, it quite confuse people and is not precise enough (differences between Perl 5.005.. and Perl 5.14 are huge for example, and it still perl 5, java embed it's horrible vm and everything so memory usage will always be against it, string it not the best object to test too). If the guy/girl have tested with arrays, it would have been a lot more... interesting (Perl is/was a dumb with that, as dynamic array = recreate a full new array each time an item is added + destroying the previous = sssslllloooooooooooooooooooowwwwwwwwwwwww)

[Spyhawk]

mmm I see, I've never have a good feeling with Python (a lanaguage that's force you with the indentation is not good IMHO) but performances anyway are mostly irrelevant for the target application here

Hmm, one of the strength of Python is forcing you to write readable code

I also agree with the performance being not relevant here. I'd suggest to write a prototype in Python, and if performance are so bad that the app is unusable to recode it in C#/Mono.

Python main weakness IMHO is the lack of something like CPAN.

Pip?

[keks]

mmm I see, I've never have a good feeling with Python (a lanaguage that's force you with the indentation is not good IMHO)

Hmm, one of the strength of Python is forcing you to write readable code

the GIL thing appears to me as quite a failure on its main purpose.

I fully agree with Spyhawk here. Python forces you to write readable code and gets rid of all the unnecessary braces.

I mean, you (should) indent your code anyway, no matter if you have to or not. So I see no reason why not to get rid of redundant braces that basically are of no meaning at all.

Also, the GIL can be a good thing. For example, you don't have to care about synchronization that much, because Python threads do not run in parallel anyway.

But that's a completely different topic and should not be discussed in here IMHO

The benchmark you give is a flawed as possible [...]

I already explicitly pointed that out in my post earlier. This benchmark is three years old and is not representative at all. It tests for one specific operation which shows a fundamental problem with Java's memory management, and shows how fast it could actually be if implemented right, like Perl does. This issue is nicely explained on the site I linked.

The reason I chose this benchmark is, that it pretty much corresponds with my personal experience with those languages. I've been using those languages for quite some time now, for projects of all kinds and scales, and I can safely say that there probably is no better language out there than Perl, when having to do lots of string manipulations. On the other hand, Perl is pretty much unusable in big projects, because of the limitations of the language features it provides "out of the box". You almost always would want to rely on some hacks like Moose (or it's deviates) which frequently break because of their dependency on B.pm and the like. Also, running Perl on Windows is a big mess. Python on the other hand is pretty portable, but very slow when multi-threading. In Python when for example implementing search algorithms in a multi-threaded approach, they are actually tremendously slower than a single-threaded implementation. This is caused by how Python handles threads and their synchronization and ofc. the GIL itself.

Perl is/was a dumb with that, as dynamic array = recreate a full new array each time an item is added + destroying the previous

That's actually not correct. Perl does not have the concept of Arrays as you know them from lets say languages like C++. Instead they basically are an over-sized array or pointers to multiple lists containing the actual data. This way all basic array operations (insert, fetch, push, pop, shift, unshift, ...) can almost always be done in O(1). There are only a few edge-cases in where the complexity grows to O(n). So actually Perl handles "Arrays" quite well. The reason why Java sucks so hard in the benchmark I linked is that it does have to allocate a new String() object after every operation and destroy the old one. This takes a lot of time. In addition to that, it does not re-use "freed" memory of destroyed objects.

but performances anyway are mostly irrelevant for the target application here, don't forget whatever the job of packaging/processing text files/creating file structure & dummy files can take 1 or 5 s, it will not change so much

I also agree with the performance being not relevant here.

Performance actually IS of concern here. When updating mods we have to calculate checksums for every file in the GameData directory tree. Depending on the actual implementation, this can be something that can take a few seconds, or a couple of minutes. AFAIK Python provides C-implementations of all major checksum algorithms, but I do not know how portable they are. Pure python implementations would take way too much time, especially on many small files. When using C# we could compute multiple checksums at the same time, taking advantage of multiple CPU cores and I/O wait times.

I'd suggest to write a prototype in Python, and if performance are so bad that the app is unusable to recode it in C#/Mono.

We already agreed to implement out prototype in Python. We also agreed on not introducing additional dependencies on client-side, so choosing Python for the actual release is out of question. .Net/Mono is already there, because KSP runs on it, so let's stick to that. It would (maybe) also attract more contributors, as mods are written in C#, not Python

Edited September 1, 2014 by keks

[TeddyDD]

I'd suggest to write a prototype in Python, and if performance are so bad that the app is unusable to recode it in C#/Mono.

I would second that If there was a way to deploy standalone Python 3 app for Win, Mac and Linux. But from what I know it is impossible to do.

For now I would stick to first idea (Python prototype > C# final app)

[Justin Kerbice]

Warning : this

I'd suggest to write a prototype in Python, and if performance are so bad that the app is unusable to recode it in C#/Mono.

comes from Skyhawk, not me, keks quotes him wrong and credit me for these sentence. I deny any involvement in this .

If your concern is checksum, best is to use the fastest checksum tool available in each platform like:

UNIX=md5sum

WINDOWS=somethingsum

MACOS=macsomething

MD5SUM= one of these (it could be a hash for example)

and a subprocess call like in Perl:

$checksum = `MD5SUM file`;

(with proper checks, before, while and after of course)

C# is not the only one to allow you to parallel-ize your code.

And before even thinking of language performance, thinking first and do smart things is a lot more better. Why calculating sum for EVERY files in GameData on update ?

You only have to check mod X folder(s), and even less: only files that need to be checked.

(Why for Jeb sake you'll have to calculate sums for files you don't care about like all stock parts/props/tex/... ?)

And... I have a big doubt, do you mean this tool is suppose to run as kind of plug-in INSIDE KSP ?

Whatever the language used, you can always made a executable which ship everything needed (Perl have the ugly PAR which make big, very big, executable from small scripts).

I think you're going the wrong way and maybe think a bit too much, the best would be to define already something to work on with a minimal features set, but WORKING and USABLE, not the best tool on Earth and Kerbin which is just a wish in your mind.

Iterative development model like KSP is great for complex projects.

[keks]

Warning : this comes from Skyhawk, not me, keks quotes him wrong and credit me for these sentence. I deny any involvement in this

My fault. Sorry for that!

If your concern is checksum, best is to use the fastest checksum tool available in each platform [...]

I'd like to keep external dependencies as minimal as possible, because they need maintenance. When sticking to language features only, we can rely on them not breaking that easily in a multi-OS environment.

Besides that, we could also rely on git's implementation here when using a C# implementation/wrapper.

C# is not the only one to allow you to parallel-ize your code.

I never said that. But I already mentioned several times that I do not think it is a good idea to introduce further dependencies when there effectively is absolutely no need for it. Why ship (and maintain!) another runtime when .Net/Mono is already available on all target systems?

And before even thinking of language performance, thinking first and do smart things is a lot more better. Why calculating sum for EVERY files in GameData on update ?

You only have to check mod X folder(s), and even less: only files that need to be checked.

(Why for Jeb sake you'll have to calculate sums for files you don't care about like all stock parts/props/tex/... ?)

Because when not checking all files for manipulation, your updates are not consistent. Maybe a user manipulated a file that is not part of the update-diff or does not want a file to be overridden/removed when updating a mod. And I never said we need to always calculate all checksums - it's just a worst-case scenario for people updating old installations. We also need to consider files created at runtime, not being part of the installation package. Do we delete, ignore or patch them? Stuff like that. Also, when knowing which files changed and which files did not, we can apply differential updates, effectively reducing update-time and fragmentation. No need to delete everything and unpack it again.

And... I have a big doubt, do you mean this tool is suppose to run as kind of plug-in INSIDE KSP ?

Nope. But I'd love to see it integrate Toolbar via some API some day, notifying you of updates But that's something to look at when things are working as expected.

Whatever the language used, you can always made a executable which ship everything needed (Perl have the ugly PAR which make big, very big, executable from small scripts).

Did you ever try that in real-life environments? I did (have to...) do that for Perl, Python and Ruby applications. It was a nightmare to get it working reliably. Freezes, segfaults, dependencies that are determined at runtime only, and stuff like that. Not to mention the bloat of always having to ship a complete interpreter and a ton of libraries in your executables. Also efficiently debugging such applications at runtime is nearly impossible.

If you still think it's a good idea, go try pack up a application of your choice and see how it runs on different systems.

(And here I'm talking of real applications, no "Hello World" stuff )

I think you're going the wrong way and maybe think a bit too much, the best would be to define already something to work on with a minimal features set, but WORKING and USABLE, not the best tool on Earth and Kerbin which is just a wish in your mind. Iterative development model like KSP is great for complex projects.

Well, I we already agreed on creating a basic prototype. But someone has to build it... As I said I currently do not have the time to build it right now. When I find the time to do so, I will build it, but I cannot say when that will be. Maybe next weekend.

Also nobody stops you guys from implementing the basic spec I posted earlier (and linked in the first post). Everything one needs to know to implement a prototype has been discussed on this thread here.

When replying to this thread here, I am usually actually doing stuff for my daytime-job, and just try to reply to your posts without having you guys waiting too long.

So again, if you feel like you want to implement a prototype application, please, feel free do so! The more references we have, the better. I will implement mine as soon as I find the time to do so.

Edited September 1, 2014 by keks

[Justin Kerbice]

So what you suggest regarding checking files and update only when needed is called 'rsync' and already used by patcher, there are even API to rsync I think.

And yes, I've already made a terribly ugly heavy package with PAR for a big Perl project on solaris if it was not already enough and that's sucks a lot.

Looks like language with "easy-and-fast" code have to be balanced by "ineffective-poor-bad" executable burden (except on unixes). Which is very sad because doing such features rich in "better-for-delivrey" languages is a lot more work. And I'm not sure C# have yet everything needed for easy and fast coding.

[keks]

So what you suggest regarding checking files and update only when needed is called 'rsync' and already used by patcher, there are even API to rsync I think.

And how would you handle the compressed contents then? You'd have to unpack the whole archive and sync it then. Also rsync cannot intelligently handle conflicts or our "hotfix" architecture that easily. Don't get me wrong, I love rsync and use it on a daily base, but I see no reason to introduce yet another dependency here. And creating such a small class handling the file patching is really not that difficult.

And yes, I've already made a terribly ugly heavy package with PAR for a big Perl project on solaris if it was not already enough and that's sucks a lot.

Looks like language with "easy-and-fast" code have to be balanced by "ineffective-poor-bad" executable burden (except on unixes). Which is very sad because doing such features rich in "better-for-delivrey" languages is a lot more work. And I'm not sure C# have yet everything needed for easy and fast coding.

Well, every language has it's specific field in which it is good. Perl's good at efficiently handling/parsing text files on *nix systems, but was never designed to be used for building standalone executables. That's something people hacked in at a later point, and is more of a toy than anything I'd use in production unless I really have to. Also, I'd not say that plain perl fits your "easy-and-fast" description. Writing object-oriented code in perl is nothing anybody out there would like to do for bigger projects without using something like Moose, which in turn adds a ton of overhead, breaks on a regular basis thanks to B.pm and the like and introduces strange bugs that are almost impossible to debug without crawling through Moose itself and it's dependencies. I myself moved away from using perl for bigger projects some time ago because of this issue. I still use it for a lot of sysadmin stuff.

[gurgle528]

And I'm not sure C# have yet everything needed for easy and fast coding.

It probably does, depending on what you mean.

[gurgle528]

And creating such a small class handling the file patching is really not that difficult.

Yeah, I personally suggest having a script run through the mod without the hotfix and then the mod with the hotfix and then automatically generating a file indicating which files are to be updated. This file can then be sent to clients and they can just overwrite the files rather than going through all files on a client's machine. Or, alternatively, you could just package an archive of only the update files and extract the archive and have it overwrite all existing files.

(apologies for double post, I'm used to MyBB post merging)

Edited September 8, 2014 by gurgle528
i dun goofed

[slavik]

To be honest, there are already a bunch of good tools. I wouldn't even worry about the specific tool. There are many packaging tools around to reference. What I would stress is coming up with a good package standard and having either developers of mods or packagers follow this standard.

EDIT: If you are talking about what language should be used, you already lost the packaging game.

[MacTee]

[...]EDIT: If you are talking about what language should be used, you already lost the packaging game.

i like this =)

[slavik]

i like this =)

I used to moderate a programming forum.

[keks]

If you are talking about what language should be used, you already lost the packaging game.

Well, actually we already made a decision here: C#. Simply because of two reasons:

* No extra dependencies needed on the client, because .Net/Mono is already there

* Someone needs to maintain the repo and the tools

As this community is focused around C#, because KSP mods are written in C#, chances to find someone in here willing to help develop and maintain a C# project are by far better than finding a, lets say, Haskell developer.

PS: I did not have time to complete my Prototype yet, as I've been quite busy with my job lately, but I should have a working prototype at the end of the week.

[gurgle528]

Using the KerbalStuff API (which MacTee might be integrating into KSPMA) would probably be better than using GitHub and having maintainers and the like. With KerbalStuff it is already done for us and the modders can update the mods themselves (and to a platform to which most already use for releases).

Edited September 12, 2014 by gurgle528

[keks]

Using the KerbalStuff API would probably be better than using GitHub and having maintainers and the like. With KerbalStuff it is already done for us and the modders can update the mods themselves (and to a platform to which most already use for releases).

As I already posted earlier, I'd love to see integration with KerbalStuff. It does not matter from where the data comes (if it is a human being, or some API), as long as it is complete.

The reason why I, personally, prefer a free repository over some third-party site is, that it can easily be continued by someone else, when the original maintainers disappear for some reason. Think about what would happen if the KerbalStuff-Guys decide to suddenly shutdown their project for some reason. Even with the source-code available, it would take some time to get everything back up and running. Not to mention the data loss...

With a solution based on a GitHub-Repo, it's a simple push of the [fork] button, and you're done. Also it's completely free. Nobody needs to pay for hosting that site, while for KerbalStuff (hosted at digitalocean.com/pricing/) obviously does.

[...] and having maintainers and the like. [...] the modders can update the mods themselves.

Most of the modders host their sources and binary releases on GitHub anyway. All these people would have to do is create a tag, say 'latest' and let our repo automatically update our index to the commit said tag points to. done. That would even be less work than any other solution out there, yet. Including the forums and KerbalStuff.

The only "work" that is to be done, is the initial import or when file structure changes. Then the meta-data has most likely to be updated manually. Simple updates with no fundamental changes to directory structure can be processed automatically. Including bumping the version, as long as they tag their releases properly.

If then we, and the KerbalStuff guys work together, they can easily integrate our repo into their site, if they wish to.

Please be patient and wait for the first prototype. As I already said, I'm quite busy with my job recently, so I don't have that much time to work on the prototype application. Also I have some other things to do as well... damn social life... I might have something to show by Sunday, but I cannot promise that...

[MacTee]

hey kek,

could you link the "spec" and samples in your signiture please. This way it's easy to find and even better its always up to date.

Btw ... good job ... all the fighting agains windmills and don't losing track!

[keks]

could you link the "spec" and samples in your signiture please. This way it's easy to find and even better its always up to date.

Sadly, I did not have the time to write a technical specification.

Looks like my rough outline/summary and the prototype (once it's there) have to suffice until I have some more free time to spend.

There's a link to the summary on my initial post, so it's easier to find.

Btw ... good job ... all the fighting agains windmills and don't losing track!

Well, I think most developers are used to fight against windmills... You don't develop stuff that's great, you develop stuff that gets paid for...

[gurgle528]

Please be patient and wait for the first prototype. As I already said, I'm quite busy with my job recently, so I don't have that much time to work on the prototype application. Also I have some other things to do as well... damn social life... I might have something to show by Sunday, but I cannot promise that...

I'm working on a KerbalStuff prototype just to test it out. There's no reason we have to use one or the other really, we could have both as sources and let the users pick. I will probably have something put together by Sunday as well.

Also, just because there is a chance that KerbalStuff might go dark doesn't mean that it isn't a valid option necessarily.

[TheAlmightyOS]

forgive me, I have not had the time to go though the entire thread. Keks, are you building this prototype by yourself?

[keks]

There's no reason we have to use one or the other really, we could have both as sources and let the users pick.

As I said, as long as their API provides every information we need, we of course can use it to easily feed the repo.

We could for example query their API every X minutes for new uploads, or even better, they push accumulated lists changes to us. We could then easily generate the mod-repo, meta-data, etc and update the index automatically.

But that is something we can talk about once there's some more or less stable codebase to built upon.

Also, just because there is a chance that KerbalStuff might go dark doesn't mean that it isn't a valid option necessarily.

I never said that it's no valid option. In fact I said I'd love to see integration (meaning sharing data in both directions) with them at some point

forgive me, I have not had the time to go though the entire thread. Keks, are you building this prototype by yourself?

I'm currently building a very basic cmd-based prototype in Python à la 'apt-get'.

So far you can download/update your local index, search for mods by name and install/update them.

As mentioned earlier, my job got a bit stressful lately as we managed to acquire two (for us) unusually big projects at a time, hence we're a bit short on manpower right now.

I usually say that I might have something on Sunday, but in fact I have had only a couple of hours over the last few days for working on my reference implementation.

So right now, I really cannot say when I will have something to show to you... And it's past 02:00 again already...

[TheAlmightyOS]

I'm currently building a very basic cmd-based prototype in Python à la 'apt-get'.

So far you can download/update your local index, search for mods by name and install/update them.

As mentioned earlier, my job got a bit stressful lately as we managed to acquire two (for us) unusually big projects at a time, hence we're a bit short on manpower right now.

I usually say that I might have something on Sunday, but in fact I have had only a couple of hours over the last few days for working on my reference implementation.

So right now, I really cannot say when I will have something to show to you... And it's past 02:00 again already...

Don't shoulder it alone. put up what you got already in Github or something, let us take a crack at it while you are working. When you come back, approve or deny the changes and move forward. That is how opensource/linux projects work. That is what we are trying to emulate, right?

We understand your vision. So let us help.

[keks]

Don't shoulder it alone. put up what you got already in Github or something, let us take a crack at it while you are working. When you come back, approve or deny the changes and move forward. That is how opensource/linux projects work. That is what we are trying to emulate, right?

We understand your vision. So let us help.

So here you go: https://github.com/ksprepo-alt/kspmm-prototype

Be careful when using it as it did not undergo any tests at all! Kittens may die when using it!

Also be sure to set config/cache_dir and config/ksp_base_dir according to your environment.

Code is a undocumented mess, but it should be pretty easy to understand.

[Benzschwagel]

wow, this is awesome! I just have to ask though, does curse fit into this in any way? I'm imagining not. I did think it was kind of odd that squad jumped to curse, when there were already several other options lying about...

...or maybe not, I could just not have a clue as to what I'm talking about, that does happen from time to time

Edited September 16, 2014 by Benzschwagel