Combining efforts on proper mod management framework / tools / platform

[keks]

wow, this is awesome! I just have to ask though, does curse fit into this in any way? I'm imagining not.

Information can be fed into the repo from anywhere, as long as there's some kind of API available that provides all the information we need.

Curse does seem to provide a JSON-API (https://github.com/curseforge/api) which theoretically could be used to feed the repo from.

[TeddyDD]

I'm trying to run this. Which git lib did you used?

[keks]

I'm trying to run this. Which git lib did you used?

Python 2.7.8
- GitPython v0.3.2 RC1 (https://pythonhosted.org/GitPython/0.3.2/)
- PyYAML v3.11 (http://pyyaml.org/wiki/PyYAMLDocumentation)

[TeddyDD]

Ok, thank you

So far so good. It works. Now I need to break into this

[Benzschwagel]

so, would this be something like a database that KSP Mod Admin (or other programs) could access to keep mods up to date, something structured and ordered, non-chaotic (as it appears now)? Basically like linux when installing programs and their dependencies (as was mentioned earlier)? If so, that's awesome, finally some sense in the soup of madness!

[keks]

so, would this be something like a database that KSP Mod Admin (or other programs) could access to keep mods up to date, something structured and ordered, non-chaotic (as it appears now)? Basically like linux when installing programs and their dependencies (as was mentioned earlier)? If so, that's awesome, finally some sense in the soup of madness!

Take a look at the first post, there's a link there to a brief summary. If you still have questions after reading the summary, feel free to ask again

[TheAlmightyOS]

EDIT: I just got a lot to say about this project and MacTee's Mod Admin. I see something here and an idea is forming but just not enough to put into words accurately. I will try to write it out when I get a chance.

EDIT2: Ok, trying again.

First off, so we are on the right foot, I 100% support your idea and think all the work you have done has been spectacular. Any criticism is meant to be constructive. So if it comes off as non constructive slap me and let me know so I can reword it.

Now, I want to better understand the goals of the project. In post #102 you redefine the project as being:

• The mod repository
  
• Description files aka meta-data
  
• The client application (developer tools and manager application)
  

It looks as if you are currently working on the manager application? If so, have we got the mod repository and the Description files nailed down? That is, do we know how we want the repo configured and by what rules the file format will follow? If not, we could have someone working on these aspects while you create the client application. Also, populating a repo is a lot of work for one person. Maybe we can start training a select few on how to create the DF's (description files) and populate a development/test repository that we can use in application development (aka, Mod Admin all OS) as well as to develop workflow / automation for the maintainers.

Basically what I am saying is the same thing I said a few posts up. Don't go it alone. The thread name is "Combining efforts", so let's

Edited September 17, 2014 by TheAlmightyOS
figured out how to put what was in my brain on the forum

[keks]

First off, so we are on the right foot, I 100% support your idea and think all the work you have done has been spectacular. Any criticism is meant to be constructive. So if it comes off as non constructive slap me and let me know so I can reword it.

Now, I want to better understand the goals of the project. [...]

The Repository: There really is nothing special about it. What we call "repository" here, actually is just an empty "git repository" pointing to other "git repositories" as submodules. Said submodules must then include a file 'meta.yaml' in their root directory providing all the information about the actual mod. Everything else does not matter to us in terms of the actual repository.

The Description files aka meta-data: We discussed this a bit, but it definitely needs further investigation and tweaking. I'd suggest to start with a minimal set of data and extend it as needed. For a basic prototype all we really need are just the download url's, archive contents and their respective checksums.

The client applications: This is what the prototype I put up earlier represents. Here all the 'magic' happens. The other two points are exclusively data-only.

It looks as if you are currently working on the manager application? If so, have we got the mod repository and the Description files nailed down? That is, do we know how we want the repo configured and by what rules the file format will follow? If not, we could have someone working on these aspects while you create the client application.

Currently I'm not working on this project as I simply do not have spare time to do so. As I said earlier, the meta-data structure needs further work. I'd actually like to dump it and start over again with the following structure:

id: "authorname-modname"
name: "Human readable name of the mod"

release:
  version: "1.0.7.3"

  download:
    - url: "http://some.domain.tld/archive.zip"
      md5: "md5-checksum-of-archive.zip"

  contents:
    extra:
      - name: "some/documentation.pdf"
        md5: "md5-checksum-of-some/documentation.pdf"
      - name: "another/file.doc"
        md5: ...
    data:
      - name: "some/part.cfg"
        md5: "md5-checksum-of-some/part.cfg"
        target: "target/to/install/some/part.cfg"

That's actually all information we need for basic functionality. The 'contents' structure got reorganized to support categories as requested by someone here earlier (too lazy to search the post), so you can chose if you for example do want to install additional data, such as artwork, documentation, alternate textures, etc.

The reason why I use lists instead of dictionaries for the file names is, that some YAML-parsers have problems with keys exceeding a given length in characters.

Also, populating a repo is a lot of work for one person. Maybe we can start training a select few on how to create the DF's (description files) and populate a development/test repository that we can use in application development (aka, Mod Admin all OS) as well as to develop workflow / automation for the maintainers.

Populating the repo at this point is kinda pointless IMHO, as we might need to adjust stuff and possibly break backwards-compatibility at some point. What actually can be done is looking out for mods that need "special treatment". That means, mods that (because of what they do) do not follow the usual pattern of "just dump these files into GameData/ and you're done". KMP and StandaloneMapView are examples for this, as far as I remember.

Basically what I am saying is the same thing I said a few posts up. Don't go it alone. The thread name is "Combining efforts", so let's

Ye, but to combine efforts on something, there actually needs to be something to start with in first place. That's why I wanted to pout out a basic prototype first, which I did. Sources are out there on GitHub, so feel free to mess with it and for example implement the new data-structure I mentioned above.

If you have a GitHub account, feel free to send me pull-requests and I'll happily merge them. I will also create a few tickets and milestones in the issue tracker on GitHub, so we can easily assign tasks to people. For this you'll have to send me your GitHub account name though, so I can give you write access to the repo.

Edited September 19, 2014 by keks

[TheAlmightyOS]

The Repository: There really is nothing special about it. What we call "repository" here, actually is just an empty "git repository" pointing to other "git repositories" as submodules. Said submodules must then include a file 'meta.yaml' in their root directory providing all the information about the actual mod. Everything else does not matter to us in terms of the actual repository.

The Description files aka meta-data: We discussed this a bit, but it definitely needs further investigation and tweaking. I'd suggest to start with a minimal set of data and extend it as needed. For a basic prototype all we really need are just the download url's, archive contents and their respective checksums.

The client applications: This is what the prototype I put up earlier represents. Here all the 'magic' happens. The other two points are exclusively data-only.

Currently I'm not working on this project as I simply do not have spare time to do so. As I said earlier, the meta-data structure needs further work. I'd actually like to dump it and start over again with the following structure:

name: "Human readable name of the mod"

download:
  - url: "http://some.domain.tld/archive.zip"
    md5: "md5-checksum-of-archive.zip"

contents:
  extra:
    - name: "some/documentation.pdf"
      md5: "md5-checksum-of-some/documentation.pdf"
    - name: "another/file.doc"
      md5: ...
  data:
    - name: "some/part.cfg"
      md5: "md5-checksum-of-some/part.cfg"
      target: "target/to/install/some/part.cfg"

That's actually all information we need for basic functionality. The 'contents' structure got reorganized to support categories as requested by someone here earlier (too lazy to search the post), so you can chose if you for example do want to install additional data, such as artwork, documentation, alternate textures, etc.

The reason why I use lists instead of dictionaries for the file names is, that some YAML-parsers have problems with keys exceeding a given length in characters.

Populating the repo at this point is kinda pointless IMHO, as we might need to adjust stuff and possibly break backwards-compatibility at some point. What actually can be done is looking out for mods that need "special treatment". That means, mods that (because of what they do) do not follow the usual pattern of "just dump these files into GameData/ and you're done". KMP and StandaloneMapView are examples for this, as far as I remember.

Ye, but to combine efforts on something, there actually needs to be something to start with in first place. That's why I wanted to pout out a basic prototype first, which I did. Sources are out there on GitHub, so feel free to mess with it and for example implement the new data-structure I mentioned above.

If you have a GitHub account, feel free to send me pull-requests and I'll happily merge them. I will also create a few tickets and milestones in the issue tracker on GitHub, so we can easily assign tasks to people. For this you'll have to send me your GitHub account name though, so I can give you write access to the repo.

My gethub account name is the same as here (and everywhere else): TheAlmightyOS. Nice thing about a unique name is people can find you everywhere. The bad thing about a unique name is people can find you everywhere . Though I am still not sure when I created a GitHub account and for what reason .

There are many different models for development out there so I do not know which one you are currently using, but I am used to the Dev/Test/Production model and that it why I am advocating for creating a DEVELOPMENT working repo. This would not be an open-to-public repo, just one that we use to develop our software against, sort of like a development database. Doing so we can see the pro's and con's of the different processes and file structures, make some changes, wipe it and start over if something isn't working the way we want. I am no coder, just a systems tech that works closely and supports developers. And the ones I work with wipe/restore their dev database almost weekly while they are working on a new piece of software. Now making a TEST/Production repo? yeah, I agree that is madness at this point in the game. However that is just how *I* view things. Actual processes will have to be decided as a whole by everyone who wants to actively participate in the project.

As for finding the "oddball" mods that can not be dropped into GameData, I have a Linux Bash script that did something similar. Will see if I can modify it to identify those mods. Will start with all mods listed here: http://forum.kerbalspaceprogram.com/threads/55401

I agree on the meta-structure. I think a lot of discussion needs to be done on that before anything else as that is the lynch pin that holds everything together.

[TheAlmightyOS]

Ok, so I edited my script to look though a folder of Zip files and print out the contents of each, thus showing us the ones that do not follow the unspoken standard of "gamedata/stuff". Now I just need to download all the (current) mods and run it on them. With some tweaking I think I could edit it to identify version files and readme files. But my script fu is weak as I need sleep. It will have to wait.

Anyone want to help me out, I could use a text file with the direct download URL to every mod you can think of. It could be curse, it could be kerbalstuff, gethub, random cloud... I just need something to feed into my wget command tomorrow so I can start looking for the mods that need special treatment.

[keks]

Anyone want to help me out, I could use a text file with the direct download URL to every mod you can think of. It could be curse, it could be kerbalstuff, gethub, random cloud... I just need something to feed into my wget command tomorrow so I can start looking for the mods that need special treatment.

We certainly do not need every mod out there, just the most common ones and maybe a few uncommon ones as well.

As you are putting hands on a lot of mods, could you also check which license they use? It would be great if you could dump your data into a spreadsheet so we can filter out the information we need, when we need it

Also, it looks like I may have some free time to spend this this weekend I'll go and try getting the prototype fully functional with the specs we currently have.

Edit: I created a few milestones so you guys get an idea how I planned to approach this. Also I'm currently creating a few tickets so you get an overview of what needs to be done.

Edited September 19, 2014 by keks

[TheAlmightyOS]

We certainly do not need every mod out there, just the most common ones and maybe a few uncommon ones as well.

As you are putting hands on a lot of mods, could you also check which license they use? It would be great if you could dump your data into a spreadsheet so we can filter out the information we need, when we need it

Also, it looks like I may have some free time to spend this this weekend I'll go and try getting the prototype fully functional with the specs we currently have.

That is the plan! I want the WIDEST possible sample. What better sample than all of them . So anything on the Community Mods and plugins library that is current, I am downloading. These are my goals:

1. Identify mods with atypical folder structure
   
2. Extract help/readme/instruction files
   
3. Extract or record version
   
4. Extract or record author
   
5. Extract or record download location
   
6. Extract or record License information (by your request)
   

...And more! (linux is awesome)

I am going to see just how much data I can collect from the mods zips themselves without relying on outside sources. I don't know what format that data will be in but I will make it available on the 'hub when I get it. First I got to download it all which (obvious, in hindsight) became quite the chore last night. Kinda forgot why we are making this repo, didn't I Anywho gonna try again tonight to at least get the most popular from curse and kerbalstuff.

Keks, is there any OTHER data we might need down the road? Like you said, I am going to be hands-on a LOT of data. If there is something specific I can set it aside for later use

Edited September 19, 2014 by TheAlmightyOS

[keks]

Keks, is there any OTHER data we might need down the road? Like you said, I am going to be hands-on a LOT of data. If there is something specific I can set it aside for later use

Well, I do not really know. you could add the output of

tree -ahF

for mods that ship non-GameData stuff for easy reference of the directory layout. I really can't think of anything else right now. Just grab all the information you can get and store it in a somehow filterable format. Better to have too much than too few information