Community Mod Repository and The Majiir Challenge

[Majiir]

I've made some changes based on your comments. Its probably still often too wordy, but can be edited down.

I largely agree with what you've done here. A few more thoughts:

• It's fair to make search a priority, but the requirement needs to be much more specific. You're right that we need a bit more discussion about that here (since tagging vs. full-text search is a core part of that). I've mentioned my thoughts on search elsewhere, but I'd be happy to enumerate in more detail if anyone likes. (I'm personally very much opposed to tags, but I'm interested in hearing a rationale for them. I find them spammy and ineffective.)
  
• Upload timestamps might be insufficient. It works for new releases, but if I want to upload all past Kethane versions, that won't be a useful indicator. An additional optional field for release date would be useful (and it could default to upload date).
  
• For a first round, it may be easier to skip building a permissions system for users, and instead allow administrators to interact with the database through a private back-end interface that's secured through some other mechanism.

We're looking good here. I don't think it represents a super solid consensus, but it gives a good picture for implementers to start working. A lot of the remaining uncertainties may be best settled after the construction of a prototype.

[NathanKell]

I am *so* not a web dev, and I don't have very much constructive from the modder-end to add; I do want to say that I am very seriously excited about this project, and to say thanks to Majiir for offering up the server and bandwidth! I might finally get to advance past dropbox, hurrah!

[Trueborn]

I largely agree with what you've done here. A few more thoughts:

• It's fair to make search a priority, but the requirement needs to be much more specific. You're right that we need a bit more discussion about that here (since tagging vs. full-text search is a core part of that). I've mentioned my thoughts on search elsewhere, but I'd be happy to enumerate in more detail if anyone likes. (I'm personally very much opposed to tags, but I'm interested in hearing a rationale for them. I find them spammy and ineffective.)
  
• Upload timestamps might be insufficient. It works for new releases, but if I want to upload all past Kethane versions, that won't be a useful indicator. An additional optional field for release date would be useful (and it could default to upload date).
  
• For a first round, it may be easier to skip building a permissions system for users, and instead allow administrators to interact with the database through a private back-end interface that's secured through some other mechanism.
  

We're looking good here. I don't think it represents a super solid consensus, but it gives a good picture for implementers to start working. A lot of the remaining uncertainties may be best settled after the construction of a prototype.

Under the assumption that uploading past versions will only happen when an existing mod first hits the system, I would argue that if users just upload the files in order, the problem is solved. And if an author does mess it up, the ability to edit release dates after the fact would fix it. That part is a second line feature, however.

Even if a full permission system isn't implemented for initial release, I think building it into the user prototype still serves a useful purpose (ie, not having to add it later). Also, I don't think we need a "fully featured" permission system. Just two or three levels. Admins and users, with the possibility of either a super-admin or moderator level if the implementation points that way.

Does anyone have anything to add on the search front?

While I certainly understand your feeling that tags can be spammy, in this case a full text search might not capture everything a user would be looking for. I guess it depends on how verbose authors make their descriptions. For example, a short description and a link to the forum thread will clearly offer a lot less material to work with than a two page write up. But I've never done any search stuff, so I'll defer to anyone who has practical experience.

[blizzy78]

Does anyone have anything to add on the search front?

I always cringe when I see comments about the Spaceport and forums search being bad (and they are.) I do have hands-on experience with Lucene and some Solr, and they are really good for searching. And you can always tweak things to your liking, such as "boosting" mod titles over long descriptions, and stuff like that.

[Majiir]

That part is a second line feature, however.

Agreed.

Even if a full permission system isn't implemented for initial release, I think building it into the user prototype still serves a useful purpose (ie, not having to add it later). Also, I don't think we need a "fully featured" permission system. Just two or three levels. Admins and users, with the possibility of either a super-admin or moderator level if the implementation points that way.

When you first praised MongoDB, I was going to say it doesn't really make a difference for a site like thisâ€â€but now that you mention it, schemaless databases do make it a lot easier to add things like that after-the-fact. (Adding fields doesn't require modifying the existing database.)

While I certainly understand your feeling that tags can be spammy, in this case a full text search might not capture everything a user would be looking for. I guess it depends on how verbose authors make their descriptions. For example, a short description and a link to the forum thread will clearly offer a lot less material to work with than a two page write up. But I've never done any search stuff, so I'll defer to anyone who has practical experience.

I wonder how often people search for things that aren't mod names. Searching for "kerbal" will probably fail, and "rocket" likely won't do much better. But something like "mining" or "pipes" will probably work quite well. So long as an existing search library is carefully selected, I doubt we'll have too much trouble. In any case, search is something that can easily be replaced later. (You've gotta rebuild the index, but that's no big deal.)

[regex]

Does anyone have anything to add on the search front?

I don't have a ton of experience with search but wouldn't a simple two field (title and body) keyword search with weighting on downloads (popularity) and/or latest update ("freshness") suffice? I recall seeing something about that earlier in the thread. Like Majiir I am opposed to tags but that's mainly because I keep having to program the damn things and, despite that, I see no real benefit to having them.

Has any developer bitten on this yet?

[avivey]

Here are some things I think are really important to have:

1. RSS feed for updates; Should be very easy to implement, and makes life for everybody who wants to monitor it easy.

Bonus points for filtering the RSS feed, but as a hard-core RSS junky - don't worry about it.

2. Use OAUTH for user management: If you've ever seen Quora's Login page: there are 2 big buttons, "Login with Facebook" and "Login with Google".

The user clicks on one of them, approves the pop-up, and that's the end of the authentication process.

Advantages: User's don't need to pick and remember yet-another username and password.

If the user chooses to, you can get their name, email and photo to use, with no more work from the user.

It's usually considered polite to allow "regular" username-and-password registration (I don't, where I can avoid it).

3. Link to source and license: I think the majority of mods are hosted on GitHub (btw - great OAUTH provider) - links from the mod page to it's github/bitbucket/other code hosting site + actual site (where applicable) would be very nice. Some kind of integration with GH "releases" page would be cool. Hack, they allow serving binaries, don't they?

Plugins that just have the source in a zip file: it should be a separate download by convention than the binaries.

4. A UID for each mod: It will make your life MUCH easier to assign each mod a random, 4- or 5-digit identifier:

- URL for any mod? kerbal.pl/mod/1234 . Simple to remember, simple to type, simple to share.

- APIs (later): you already have each mod.

- "Wiki-style links": whenever the word "mod1234" shows up in any rich-text field, it gets replaced with a link and a name to the right mod.

Why 4- or 5-digits? Humans like patterns. You know that this: 444 4175 is a phone number; So if each mod has a 5-digit code, it's easy to think of any 5-digit code as a mod. But that means no leading-zero, because people find that confusing.

So 4 digits are enough for about 9000 mods, and 5-digits are almost 90000 - probably enough for the foreseeable future.

For reference - this forum uses a running number for thread ID, and this is thread 71133.

Other thoughts:

- I would leave "mod ownership transfer" to version 3 of the site. There's lots that needs to happen first, and there's plenty of time to worry about that later.

- keywords/tags, and tag-cloud, are very useful features both for looking for something specific, and for exploring for new stuff.

[blizzy78]

2. Use OAUTH for user management:

If it also works with any other identity provider, I'd use it. I'd find it rather useless if you can only use predefined providers.

Why 4- or 5-digits? Humans like patterns. You know that this: 444 4175 is a phone number

Nope. 4 44 41 75 is a phone number. Spot the difference?

So 4 digits are enough for about 9000 mods, and 5-digits are almost 90000 - probably enough for the foreseeable future.

Ugh. I'd rather not go the 640 KB route. Make it just a random number or something from 1 to infinity. But definitely not a fixed-digits number.

[Superfluous J]

Is there a particular reason the first mod submitted can't be id "1" and the next one "2"? is the fact that the mods have the same number of digits in their IDs important?

[blizzy78]

Everyone wants to be number one, no?

[acc]

I started some conception about a mod repository a few months ago, but ÃÂTM i have no big enough time window to do some (node.js) prototype programing. Maybe sometimes...

If I remind the conception file URL I'll post it here

Edited March 12, 2014 by acc

[regex]

Is there a particular reason the first mod submitted can't be id "1" and the next one "2"? is the fact that the mods have the same number of digits in their IDs important?

If you're using a document database like MongoDB you'll find sequential numbering to be a slight pain in the rear because it doesn't natively do it like a relational database. We (my work) do it in our task system on a per-project basis but the number of users is pretty small and any concerns are minor. I don't know how much of a concern it would be for this site, but I wouldn't bother and would instead use the native almost-unique 24-digit identifiers.

[KvickFlygarn87]

I would like it to have some "workability" - I'm thinking a way to get a tag of the latest release (in the form of a txt), for autoupdaters.

[Majiir]

RSS feed for updates; Should be very easy to implement, and makes life for everybody who wants to monitor it easy.

With both HTML and JSON listings available, it shouldn't be difficult to add an RSS feed... but it's additional labor, yet another standard to worry about, et cetera. Second- or third-round feature, in my opinion.

Use OAUTH for user management: If you've ever seen Quora's Login page: there are 2 big buttons, "Login with Facebook" and "Login with Google".

The user clicks on one of them, approves the pop-up, and that's the end of the authentication process.

There are frameworks (like Passport) that provide a common structure for implementing all kinds of authentication mechanisms. I think it's well worth using such a framework from the get-go, with simple username/password authentication to start. Supporting additional authentication methods might require some additional work to make sure there's always a sensible username available, for example.

A UID for each mod: It will make your life MUCH easier to assign each mod a random, 4- or 5-digit identifier

There's something to be said for URL slugs, too. Example: http://example.com/mods/kerbal-attachment-system These didn't work so well on Spaceport because convention dictates mod names include the mod version and the compatible KSP version. Accordingly, the KAS slug is "0-18-2-kas-kerbal-attachment-system-v0-1" which probably doesn't do wonders for SEO. A brief note about this ("don't include versions or anything else that might change over time") on the mod submission page might do the trick. Providing other IDs could be useful for URL shortening, but string keys are perfectly sufficient for internal use. In any case, it's important that if there are multiple URLs to a mod (e.g. /kethane and /12345), one URL should have a permanent redirect (HTTP 301) to the other. This indicates to search engines that they're the same entity. In my personal opinion, numeric identifiers are unnecessary here.

I would like it to have some "workability" - I'm thinking a way to get a tag of the latest release (in the form of a txt), for autoupdaters.

This falls into the API category. Querying "/mods/kerbal-attachment-system" (with appropriate Accept headers) might yield something like this:

{
    "id": "kerbal-attachment-system",
    "name": "Kerbal Attachment System (KAS)",
    "owner": {
        "id": "majiir",
        "name": "Majiir"
    },
    "description": "Attach pipes and stuff!",
    "versions": [
        "0.4.5": {
            "released": "2014-03-12T19:19:46.681Z",
            "description": "Fixes some bugs."
        }
    ]
}

It's probably more information than you're looking for at first, but it can serve a wider variety of purposes this way.

Edited March 12, 2014 by Majiir

[Elros]

{
    "id": "kerbal-attachment-system",
    "name": "Kerbal Attachment System (KAS)",
    "owner": {
        "id": "majiir",
        "name": "Majiir"
    },
    "description": "Attach pipes and stuff!",
    "versions": [
        "0.4.5": {
            "released": "2014-03-12T19:19:46.681Z",
            "description": "Fixes some bugs."
        }
    ]
}

It's probably more information than you're looking for at first, but it can serve a wider variety of purposes this way.

So then to download do "/mods/kerbal-attachement-system/version/0.4.5/download" or "/mods/kerbal-attachement-system/version/0.4.5/" for json info?

[Majiir]

So then to download do "/mods/kerbal-attachement-system/version/0.4.5/download" or "/mods/kerbal-attachement-system/version/0.4.5/" for json info?

There's no reason URLs have to match JSON output exactly. A simpler scheme might be "/mods/kas/0.4.5" yields information about a version, and "/mods/kas/0.4.5/download" retrieves the file. Also remember that a server can respond to the same URL differently based on headers, so a web browser will see a well-formatted page while an autoupdater might see JSON (because it sets an "Accept: application/json" header).

You could also run API endpoints on a completely different hostname, e.g. http://api.example.com/json/get-mod-info?id=kerbal-attachment-system ...but I think that's not nearly as clean, myself. The information is retrieved in the same way; it's just a matter of returning it in a different format.

[IronGremlin]

What about site administrative functions?

It seems like there should also be easy ways to do things like:

change a mod name without killing search / statistics for the mod

change a user name without killing the associated content

etc.

Additional functionality there would be largely dependent on what other features ended up getting added, but making sure that the site admin doesn't have to dick around with scripts and or DB entries any time we needed to change something would probably be good to work in early.

[avivey]

Is there a particular reason the first mod submitted can't be id "1" and the next one "2"? is the fact that the mods have the same number of digits in their IDs important?

In addition to what regex said about it being hard, a random fixed number of digits makes all mods "equals", in some sense.

It just makes the thing look more "complete".

There's something to be said for URL slugs, too.

You want to allow mod owners to easily edit these ones, because they typed it wrong or change the mod's name.

So relying on them internally will make quite a mess.

it shouldn't be difficult to add an RSS feed... but it's additional labor, yet another standard to worry about, et cetera.

Pretty please? Otherwise I'll have to poll the site every X time.

[fusty]

Ok, so I was recently alerted that SpacePort download totals are nonsense.

If anyone was unconvinced, check out this album.

Javascript is disabled. View full album

Some counts go up and some counts go down. This is pretty funny, I'd love to know what causes this. Probably just some unforseen sql query goofiness.

Edited March 14, 2014 by fusty

[blizzy78]

Some counts go up and some counts go down. This is pretty funny, I'd love to know what causes this. Probably just some unforseen sql query goofiness.

How much would a sensible person need to goof around to mess it up like this? But anyway, let's go back to discussing a replacement.

[arrdem]

Okay. So it sounds like there is interest in and a design for "ReplacePort". Also it seems like this thread is dying down. Is there some sort of plan for throwing this project up on GitHub or somewhere else so that other people (myself possibly included) can share the load of making this happen? We've got a design document (ish... it'd be nice to see the whole thing in one place) help us help you roll an 0.1.0.

[Greys]

The thing I think a lot of potential 'investers' in this are getting hung up on is that this is one project. The idea is that people are free to try whatever they please, within certain guidelines. Multiple replaceport attempts have been a given from the start, so anybody that wants to start a project should, and should then publicize it here so people who are more interested in joining a project can see what's around.

Github is a good idea, details about test environments should be shared. More specific design docs for each project are a good place to start, TrueBorn's doc is a good generalization of the users desires but it's not much to put on top of a project and yell Go about. Project design docs should talk more about specifically how you intend to solve given problems, software, systems API, what flavor soda will be provided, etc.

Competition breeds excellence might be somewhat untrue, but people working separately run into the same problems from different angles and cooperation between them makes everybody better, plus there's no way everybody's skills and ideas will fit into one project.

I hope we can get some interesting play-by-play going on with people solving the same problems differently soon, I love watching doers do.

[croxis]

I have started putting together a prototype in flask: https://github.com/croxis/kmr Don't get too excited, its not much more than some MongoDB schema and user registration

Right now I'm just viewing this as an opportunity to work on my coding skills more than actually trying to push this as the specific platform. I've also started idling in #kspmodders in the odd event people had ideas/suggestions/accusations for me.

Edited March 16, 2014 by croxis

[regex]

I have started putting together a prototype in flask: https://github.com/croxis/kmr Don't get too excited, its not much more than some MongoDB schema and user registration (and being unable to login, still trying to figure that one out...)

Right now I'm just viewing it as an opportunity to work on my coding skills more than actually trying to push this as the specific platform. I've also started idling in #kspmodders in the odd event people had ideas/suggestions/accusations for me.

This is what I would have used: http://oxformongo.org

I could easily crank out a rough, unstyled prototype of a site with most of the basic features that have been asked for in under a week, but then, I'm probably biased because I use that framework every day at my job. Anyway, it can handle your routing and basic security if you're interested. E: and local file assets as well, and probably a few other things I'm not remembering.

Edited March 16, 2014 by regex

[croxis]

http://forum.kerbalspaceprogram.com/threads/71133-Community-Mod-Repository-and-The-Majiir-Challenge?p=1004462&viewfull=1#post1004462

Granted it is python and not node.js, but at least it isn't django