[WEB API] unofficial api for mod versions [0.5.4-devel]

[Droa]

Hey guys, so i've wanted this feature for along time, to be able to make update tools for the client-side.

i made a crawler, scanning the ksp mod forum, once every day, for updated threads, and grapping version numbers and the forum post.

currenctly its pretty much not worth much, but i am working hard to get it better, and usable

it's a pretty simple crawler, as i am currently figuring out features, and what data to use.

Crawler does the following:

Crawler:

1) loads pages from http://forum.kerbalspaceprogram.com/forums/35-Add-on-Release (/page1 to /pageX (currently 35)

2) Scans the pages threadlist, and graps the thread id (example 29388)

3) Builds the thread url (example http://forum.kerbalspaceprogram.com/threads/29388)

4) Opens all builded urls as forum posts

5) Scans the title of the forum post, for ksp version, and mod version, as it tries to figureout the name of the mod

6) saves the url, as a uniq mod, in the database

7) saves the current context of the forum post (body/title), linked to the uniq mod (inkluding version number, and ksp version number)

the crawler is not very smart, and sometimes doesn't get the meta data right, so i've have implementet a "reviewed" column, in the database, for every new update, to be able to review them manually, if something goes wrong.

now to the api interface:

The api-interface is also under heavy development, so i can use it for my own "update notifier".

it generate xml's from the current information of the crawler database, and cache them up to 10 minutes, from when it was generated, to reduce server-load (only from db -> xml generation, it does not read forum posts every 10 minutes)

list mods with full body and show mods with no version: http://mods.minsk.dk/api/list/showall/fullbody

list 0.25 mods with full body and show mods with no version: http://mods.minsk.dk/api/list/showall/fullbody/0.25

list mods: http://mods.minsk.dk/api/list

list mods and full forum body: http://mods.minsk.dk/api/list/fullbody

list mods and show mods with no version: http://mods.minsk.dk/api/list/showall

list 0.25 mods: http://mods.minsk.dk/api/list/0.25

list 0.25 mods and show mods with no version: http://mods.minsk.dk/api/list/showall/0.25

currently the xml file is very small only (323kb) containing informations about all the mods, however i've removed the post body text, as it was taking up alot of space, making the file (20Megs+) in size

mod\@name name of the mod

mod\@assemblyid is the specific id, for the mod version

mod\@modid is a static id, for the mod (if the url change, so does the modid)

mod\@kspversion is the announced version, the mod runs for

mod\@modversion is the version of the mod

mod\@updated is the last time the crawler, seen an updated version of the forum post

mod\url is the url to the forum post

mod\forum\title the title of the forum post

mod\forum\body (currently disabled) the content of the forum post

hope you guys like it, and are able to use it.

if anyone got any ideas, please feel free to post them

[10. Dec 2014]

Just got my server moved to another rack in the server room, and added SSL encryption to the apache service, meaning i can start to add a better user-crowd controlled edit-system, to the database.

this will open up for more features, i've been working on.

-Usermade list of mods/threads to follow

-Moderation of the Mod list (Changing om mod names, versions, and stuff, if they got read wrong by the crawler)

Edited December 10, 2014 by Droa

[VITAS]

you might want to team up with CKAN. I just made a mysqldb for them today containg mod versions ,links and stuff.

[pjf]

Aye, I was going to say, please use our metadata. It's all under a CC-0 license and has absolutely no restrictions. If you're looking at doing any sort of mod indexing, we already have a wealth of data to get you started. The metadata repo can be found here: https://github.com/KSP-CKAN/CKAN-meta