Calling All Mod/Plugin Developers, I have a proposal, I need your support

[Dani-Sang]

tell you what katateochi (and bluejayek already made a point with that link)

Nobody is going to give you support or whatever, thats not how the stuff works, specially not on the internetz.

We have seen already multiple suggestions like these or even attempts, somebody said that he has seen 20 of these things pass by.

If you truly believe your system will work then I suggest you start with it and release it. If it is indeed good, people will use it, if it won't we will keep continuing with the forum. I said it already on IRC to others;

"You guys can keep discussing as much as you want that a central mod system is needed, but as long there is non that functions properly, I will keep using the forum"

For me this is like saying; "I've got a great idea for a mod and its going to do this and that" - but there is no code of even a plan to pull it off.

Show something, then you get support. And even if you make a good one, you can only hope that 50% of the mod community will use it.

[katateochi]

yes XKCD is right (as always)!! Except in rare cases where the community really pulls together like with Ruby and its centralized-yet-distributed 3rd party lib hosting (the idea which I'm basing this on).

Yes I've seen plenty of ideas about centralised mod websites go by, not one that I've seen is about a distributed, cloud based approach with a focus on dependability and scale.

Show something, then you get support. And even if you make a good one, you can only hope that 50% of the mod community will use it.

That would only fulfil XKCD's prophecy! It would be a waste if no-one used it, but it would actually be worse if only a few did. I was hoping for a discussion about the merits of cloud hosting and raising support for that. I hoped to get people to see git as a means to an end but I wrongly put that aspect over too strongly.

If this was a race;

1st place was everyone already using git and eager to get the cloud benefits of github.

2nd place was people wanting to have the cloud benefits and not mind about having to learn git.

3rd place was people not wanting to use git but coming up with other ideas about cloud based hosting.

[nhnifong]

As a mod user, I would love it if all the mods were on Github. downloading them from this forum is frustrating and ridiculous. Also bug tracking.

[Accelerando]

"If this was a race, my opinions and everybody who agrees with them would be at the top!"

I would like a better mod repository, but frankly I'd prefer if it was integrated with the official KSP site a la Simtropolis's SimCity mod database.

[EndlessWaves]

Another option is to allow mod devs with the "git or whatever its called" mindset (lol) to upload their zips (or whatever their called; archaic file structures) to the web site and have it create and manage repos automatically (cos the github API is just that easy). Why upload more than just the files that have changed is beyond me but ok.

That's the sensible thing to do. Most modders aren't going to care about hosting or version control, they just want to hit an 'upload' button and make their mod available to others. Chances are we'll get a mod manager with self installing packages sooner or later so it'll be a case of manually uploading an 'archaic file structure' for those using online version control (I'm assuming they don't allow you to run custom executables on their servers). Think facebook, not freenode.

It should be easy enough to provide an alternate download link for those who do want to use something like "github" (only a Linux man could come up with such an unusable name).

[Moustachauve]

What we will do for the moment is that when you would "upload" on our website, your file would be uploaded to our guthub repository.

It would be as easy as if you were posting on the forum, but the mods would be easier to find and to browse through

[wizard`]

What we will do for the moment is that when you would "upload" on our website, your file would be uploaded to our guthub repository.

It would be as easy as if you were posting on the forum, but the mods would be easier to find and to browse through

So all you will be doing is indexing mods, in which case why even bother uploading to a git in the first place? It's no more reliable than each mod creator providing a mirror url for the mod files and then you simply make a 'profile' of the mod.

I fail to see the revolutionary idea, if you were to host the files on your own server it would make much more sense (+speed, +consistency, +easier rights management and less legal doohickery, +no need to rely on api's, +did i mention the huge speed difference?, +speed, oh and +speed) and if you choose your host(s) right then uptime wont be an issue.

[Moustachauve]

Yea, but it also mean +bandwidth, and really more bandwidth, which can be translated as Much higher cost for server. It also mean +security leaks possibilities, and +chance of loosing all mod indexed if we stop the service or if we loose all our files.

with Github, it assure :

Security

Low service cost for us

Not loosing any file since they make backup and lots of big industry use it (Not close to stop services)

faster for people away from our server (They use cloud system)

Centralized mod/plugins/... repositories (cloud)

99.69% reliability for Github and 99.98% for the webserver

The cons :

Its a little bit trickier for mod developers (Or we can make this pass transparently, but less support for mods makers)

Maybe a little bit slower (?)

Edited August 29, 2012 by Moustachauve

[katateochi]

It's no more reliable than each mod creator providing a mirror url for the mod files and then you simply make a 'profile' of the mod.

I fail to see the revolutionary idea, if you were to host the files on your own server it would make much more sense (+speed, +consistency, +easier rights management and less legal doohickery, +no need to rely on api's, +did i mention the huge speed difference?, +speed, oh and +speed) and if you choose your host(s) right then uptime wont be an issue.

I think you will find it will be several orders of magnitude more reliable. A single server is a bottleneck, it is also a "single point of failure". Speed and consistency are increased several fold with their infrastructure vs own server.

If one server has to serve 10 users at the same time, they will be queued, hopeful the server can make a number of parrallel requests but it will be limited to how many threads that server can process (therefore requiring a more powerful server as user nos. go up), the same 10 users would all be dealt with COMPLETELY IN PARALLEL by github/any decent cloud architecture. There are literally 1000's of machines behind it, virtual ones, so more just get spawned as needed.

Github will happily serve several 1000's of users concurrently, while a self run server might hope to serve a few 100 concurrently (if its a good one) with a shadow of the bandwidth. If 100 users start downloading from your server it will get slow, not the case with github. The implications of a physical machine failing or being under load becomes less and less significant due to the massive redundancy (ie every machine is cloned over many other machines).

There is no legal doohickery! If you want, all you need do is write "released under the MIT licence" in the readme. (Actually if you wanted to fully protect yourself, you'd have much more legal work in setting up a single server to be a host for a group of 3rd party developers).

"Choose your hosts right to ensure uptime", doesn't that make you ponder? You mean your hosts have significant down time? Github has the occasional glitch in some part of one of its services, they instantly publish the issues (via status page + twitter) and usually the resolution comes 20-30 min's later. The main service stays up; uptime over the last year is 99.69% (https://github.com/features/hosting | https://status.github.com/).

The revolutionary idea is a mindset as much as anything else. Some might understand me if I simply said; the idea is to use a distributed approach with a "bottom-up" design vs a centralised approach which uses "top-down" design. But others may not (and some are prob. still giggling at "bottom-up"). I just don't know what other people know so I'm just not sure how detailed to make my replies! If you don't see right-away why a cloud architecture is more powerful I guess I'd need to explain distributed design some more.

The idea of "top-down" or hierarchical design is to have a leader and a chain of command which goes down from that leader. At every stage in the system you take orders from those above you. We see this design everywhere and it's very logical.

In the 60's they tried to make computers "think" for themselves and used this same design concept. They grumpily announced that the field of Artificial Intelligence was a flop and almost gave up. Then a guy (Valentino Braitenberg) came up with the idea of distributed design which turned the whole problem on its head. Instead of a chain of command telling each part how to function, each part functions more autonomously. This came at a similar time to advances in the study of organic nervous systems which where finding that this same distributed design was found in both simple & complex systems(organisms) in nature.

Without getting into details its pretty hard to explain; An analogy of the two approaches is the Cambridge rowing team vs a flock of birds! The rowing team is a top-down design, it has a leader and must have that leader to maintain a rhythm, choose direction etc. It also requires all its members to function, if one is taken out they can't row straight. The flock on the other hand has no leader, each member follows the same/similar set of simple rules and they act as a cohesive entity which is not effected by one or several members being taken out. If the numbers of the rowing team are changed the whole system has to re-align itself to compensate, where as a flock can merge and split with other flocks, groups and individuals seamlessly.

The rowing team is a group of (most-likely) intelligent humans, that's a lot more neural hardware than in the whole flock of birds put together. But which one will be better at reacting quickly, avoiding obstacles or changing course as a team? Which one can sustain damage and maintain it's purpose/function?

The idea of bottom-up is to not have a leader or special member and consider every member expendable (up to a point). It gets away from having "single-points-of-failure" (ie single thing which when broken takes out everything else) and can continue to scale by adding members without any change to the system as a whole.

Another aspect of distributed design is that all the parts are largely the same; ie a flock of birds, computers in cloud, the neurones in your brain; all make up very complex, very capable systems all built off many very simple, very similar parts acting together in a unified cohesive way. Staggering really.

Bottom-up design is what has led to the notion of cloud computing; a flock of computers with no clear leader, no one valuable member, each machine is expendable with no consequence to the cloud. Data in the cloud is mirrored over several machines and spread in such away to minimise the effect of regional failure. It becomes less and less probable for even a series of failures to impact on any one bit of data.

There is much much more to this concept such as how these systems can self organise etc but that just gets too involved.

I hope that explains why a bottom-up / distributed architecture makes for a far more resilient system which is much much less susceptible to failure and see how that applies in the context of cloud computing.

The idea I'm trying to put over (not doing too well thou!) is not only to benefit from distributed design at the hosting level (ie data is distributed so more resilient etc) but to also apply that thinking to how to rip the Internet off and get free stuff. The greatest thieves have always been the ones who took tiny amounts from 1000's of people, they applied distributed thievery, and that's what I propose. No host is going to let you store GBs of data with high bandwidth requirements for free, that would be like taking too much from one person. However if that data is spread over all the accounts which make up the mod dev community then GitHub will not care that all that data is on them. This idea does not work if all the data is on a single GitHub account, that will exceed the download limits, the data has to be distributed over a number of Github accounts. It won't even be thievery in their eyes, there is nothing in their terms to suggest that this would be a problem.

My idea is to distribute the storage of data over a unified distributed network and gain resilience, speed, scaling etc all completely for free. I really do mean for free, I don't mean free for a couple months then pay, or free while user base is small but when we grow it will hurt, I mean absolutely completely 100% free no strings.

There is one case where my claim might be challenged; If one particular mod becomes so popular and at the same time is very large it could be an issue but you would need something more popular than Twitters Bootstrap or jQuery!!! I really don't see that happening, maybe I'm wrong.