Ake74

I'm looking into reviving my old tool "KSP Mod Analyzer" from 2017 and more specific making the CurseForge parser work. The old CurseForge page layout was changed and did not show the supported mod version in the list of mods (https://www.curseforge.com/kerbal/ksp-mods), instead the only way forward (as far as I can see) is to brute force as follows: - Parse each of the 49 pages with the mod names (20 mods / page) - Then, request each individual mod page (e.g. https://www.curseforge.com/kerbal/ksp-mods/mechjeb/files) for the supported KSP version - This approach would be 49 * 20 = 980 individual requests to CurseForge to get all data... I have read something about a new CurseForge API, would that be a way forward? It looks like an involved process for approval and then the question is how to manage a secret API key for an open source project on GitHub... I just need a dump of all CurseForge mods with basic data like supported KSP version, similar to the data provided by the SpaceDock API. Here is a link to the thread from 2017 for reference:

Hi, thanks for your input, I will look into manual parsing of the Curse page, probably with some threading functionality to speed thing up, will post here when I have an update!

Hi @HebaruSan, I noticed that the widget/API has now been updated to handle Curse after the change. I'm looking into some documentation on how to use the API for my application, specifically if it's possible to make a query to get a list of all mod id's, maybe you can help me with some quick hint...? Would it be possible to make a query to get all data for all mods in one call (or something like data for e.g. 100 mods in one call if there is a cap), to avoid calling the API for each mod? For KSP Mod Analyzer I'm looking for "mod name", "supported KSP version" and "URL to mod page" if possible.

Yes, that sounds like the best solution, I will wait for an update on the widget and then adapt my code. It's good that the widget developer is looking into it (see this CKAN issue).

Update: The Curse page layout has changed, the "Supported KSP version" field in no longer available on the list page (https://www.curseforge.com/kerbal/ksp-mods) This complicates things a bit, as I used that info to get the supported KSP version for each mod. Now, it seems I need to parse each individual mod page to the KSP version data, which will take time and also put load on the Curse server. Before, I just needed to parse each list page (42 pages in total); to get the same info now means parsing 42 * 20 = 840 pages... Any ideas on how to proceed...? Maybe if this API/Widget will be fixed I can use that, what do you think? Is there really no official API available for getting the data from Curse?

Thanks for this info, it seems the Curse page has changed and as I'm parsing the Curse page directly (I'm not using the API/Widget) I need to do some updates to make it work again. Stay tuned for an update...

Hi, I'm using the SpaceDock API for my KSP Mod Analyzer application and have noticed that sometimes the API gives inconsistent results. I'm using the API to get data for all mods, and the API may return +/- one mod compared to the previous run. To make this reproducible and easy to test, I created a small Python application found here: GitHub Gist The script uses the same functionality as in KSP Mod Analyzer but just writes the list of returned mods to a text file. It can be configured to run e.g. 3 times, creating three different output files. Comparing these shows that sometimes there is difference between the runs, e.g. one mod may be missing in the first run. Note that which mod is affected seems random. Just like to highlight this, I hope my test script may be helpful in reproducing the problem finding the root cause. Please let me know if I can do some more troubleshooting from my side. Note that KSP Mod Analyzer has a cool down period of 60s between each run (to limit the traffic to the server), but this does not seem to solve the problem.

KSP Mod Analyzer 1.1.1 is released, CSV export added as discussed above, please enjoy. The stand alone version can be downloaded here.

@linuxgurugamer OK, I have a working CSV exporter now (with user configurable delimiter set to "," in this example), but I would like to get some advice on the "escape character" and how to handle double quotes in field values. Here are some examples (first row is the header), note that the exporter takes the current view (e.g. "All mods", "Mods only on curse" etc) automatically. 1) Double quotes around all fields, and an escape character "\" preceding the double quotes in the URL:s "Mod","SpaceDock","Curse","CKAN","Source","Forum" "- Rocket Factory -","","<a href=\"https://mods.curse.com/ksp-mods/kerbal/243373-rocket-factory\">1.2-pre</a>","","","" "10km Omni Antenna for Kerbals on EVA for Remotetech","<a href=\"https://spacedock.info/mod/1218\">1.3.0</a>","","1.3.0","","" 2) No double quotes around fields, same "\" preceding the double quotes in the URL:s Mod,SpaceDock,Curse,CKAN,Source,Forum - Rocket Factory -,,<a href=\"https://mods.curse.com/ksp-mods/kerbal/243373-rocket-factory\">1.2-pre</a>,,, 10km Omni Antenna for Kerbals on EVA for Remotetech,<a href=\"https://spacedock.info/mod/1218\">1.3.0</a>,,1.3.0,, 3) Using one additional double quote for "double quotes in field values" as follows: "Mod","SpaceDock","Curse","CKAN","Source","Forum" "- Rocket Factory -","","<a href=""https://mods.curse.com/ksp-mods/kerbal/243373-rocket-factory"">1.2-pre</a>","","","" "10km Omni Antenna for Kerbals on EVA for Remotetech","<a href=""https://spacedock.info/mod/1218"">1.3.0</a>","","1.3.0","","" Not sure which option is considered best practice, please let me know your view. Note that the escape character in 1) and 2) can be changed to any character. Here is the Python code for reference, settings are changed in the parameters to "writer": def export_csv(self): """Exports the current view to a CSV file.""" suggested_filename = "mod_export" filename, _ = QtWidgets.QFileDialog.getSaveFileName(self, "Save File", QtCore.QDir.homePath() + "/" + suggested_filename + ".csv", "CSV Files (*.csv)") if filename: # Get rows and columns from data model rows = self.model.rowCount() columns = self.model.columnCount() with open(filename, 'w', newline='', encoding='utf-8') as csvfile: writer = csv.writer(csvfile, delimiter=',', escapechar='\\', doublequote=False, quoting=csv.QUOTE_ALL) #writer = csv.writer(csvfile, delimiter=',', doublequote=True, quoting=csv.QUOTE_ALL) # Write the header header = [self.model.headerData(column, QtCore.Qt.Horizontal, QtCore.Qt.DisplayRole) for column in range(columns)] writer.writerow(header) # Write the data records for row in range(rows): fields = [self.model.data(self.model.index(row, column), QtCore.Qt.DisplayRole) for column in range(columns)] writer.writerow(fields)

I'm using PyInstaller, it works really well and is in active development, here is the GitHub repo.

Sure, let me look into that, should be easy to implement. My idea is an "Export to CSV" button that exports the current view. Please let me know if you have any other improvement ideas as well.

Hi, maybe a bit off topic, but as you mentioned on the readme that you have plans for a future update with a QT user interface, I just like to mention that I have built my application KSP Mod Analyser in Python 3 + PyQt 5. It just analyzes all available mods on SpaceDock, Curse and CKAN and aggregates the meta data for a clean presentation (no mod manager features), but maybe some ideas can be reused. You can find my project on GitHub, and here is the forum thread. I used "PyInstaller" for packaging the stand-alone version which works without having Python installed.

Hi, that would be very helpful!

Hi, I just release a new version of KSP Mod Analyzer, see details on first page. Enjoy and please let me know if you have any improvement ideas or any other feedback. Regards, Ake74

@Thomas P. Would it be possible to generate a ZIP with all the JSON data and allow me to download that with my application? That would probably be more efficient that requesting all sub pages. Maybe the ZIP could be generated automatically on given intervals, like once 1h, once a day. Just an idea...

Hi, yes that is probably a good idea, here is how my application works: 1. Get the first page from SpaceDock (https://spacedock.info/api/browse?count=100). Note that I'm using "100 mods per page", please let me know if any other value is more preferred. 2. Store the JSON data and get the total number of pages (based on 100 mods in this example). Currently 11 sub pages. 3. Loop and generate a new URL to fetch the next page, store the JSON data for each page. This is repeated for each page, in total 10 more calls besides the initial URL. Which limitation would you prefer I implement, maybe a timer that allows to run the whole data collection (all steps) only once every minute (or other value)? Or would you recommend a delay in the loop for the sub-pages? I would prefer a delay for the whole function, but I'm listening to recommendations from the SpaceDock admin

I see, that is indeed a very good idea! Let me look into that as well, if I'm not mistaken, links to the project/source for each is available using the SpaceDock API so that should be easy to implement. Not sure for Curse, as there is no API available and I'm parsing the HTML to extract the mod names, I will investigate that. Maybe another option could be to use CKAN data as I believe there is a link to the project/source there as well?

Yes, GitHub is nice, there is a link to my project in the OP Or did you mean CKAN? I planning to add CKAN support in a future release.

KSP Mod Analyzer 1.1.1 has been released, with the following new exciting features: Support for SpaceDock, Curse and CKAN repositories Clickable links to mod page, source code repository and forum if available Filter on mod name Many UI improvements 60s cool down period after updating SpaceDock and Curse, to limit network traffic Possiblity to do analysis on mods, e.g. "Mods only available on CKAN", with aggredated metadata CSV export Note! Due to recent changes to the Curse page, "Update Curse" is broken, I'm looking into the problem. In summary, this application gives you a clear overview of all all KSP mods with relevant metadata in one single application. Note that is is not a mod manager like CKAN, but I hope you find it useful anyway :-) Please find download details below the screenshot. Technical details Written in Python 3 using Qt 5 for the UI Source code is available on GitHub Latest release, packaged as a stand-alone application, is available on the release page (Python is not required) Let me know what you think about this application, feedback and improvement ideas are welcome.

Updated the analysis module to calculate statistics, files are available on GitHub. Here is a summary: All mods on SpaceDock: 594 All mods on Curse: 641 Mods only available on Curse: 540 Mods only available on SpaceDock: 494 All mods on SpaceDock All mods on Curse Mods only available on Curse

Updated original post with GitHub link.

SpaceDock and Curse analyzer Hi, I would like to share a Python project I have been working on, "SpaceDock and Curse analyzer". I started working on this mostly for learning Python, and now I have a version up and running. The idea behind the project is "How to get a list of all mods that are hosted only on Curse but not on SpaceDock?" My preferred mod hosting site is SpaceDock, but I suspected that some mods are only available on Curse. How to get this analyzed in a nice way, and learning Python at the same time? Long story short, here is how it works, and a description of each Python file: get_curse.py - Parses the Curse page and retrieves all mods and store them on disk as a list. I did not find any API for Curse, so it's done with BeautifulSoup and a bit of reg-exp hacking. get_spacedock.py - Using the SpaceDock API to get a list of all mods and storing the list on disk. disk_helper.py - Helper functions to serialize data for storing on disk as well as reading and de-serializing the data from disk (used in the files above). curse_helper.py - Helper functions for parsing Curse spacedock_helper.py - Helper functions for parsing SpaceDock analyse.py - Reads the mod lists from disk for further analysis (e.g. to check which mods are only on Curse). This file is still a work in progress, currently it just list the mods. The files are available on GitHub The serializing to disk is for offloading the web servers, as you can play around with "analyse.py" without causing unnecessary traffic. Please let me know your comments on this project. There is probably a lot of Python code that can be optimized, but bear with me as it's my fist project