Ake74

I'm looking into reviving my old tool "KSP Mod Analyzer" from 2017 and more specific making the CurseForge parser work. The old CurseForge page layout was changed and did not show the supported mod version in the list of mods (https://www.curseforge.com/kerbal/ksp-mods), instead the only way forward (as far as I can see) is to brute force as follows: - Parse each of the 49 pages with the mod names (20 mods / page) - Then, request each individual mod page (e.g. https://www.curseforge.com/kerbal/ksp-mods/mechjeb/files) for the supported KSP version - This approach would be 49 * 20 = 980 individual requests to CurseForge to get all data... I have read something about a new CurseForge API, would that be a way forward? It looks like an involved process for approval and then the question is how to manage a secret API key for an open source project on GitHub... I just need a dump of all CurseForge mods with basic data like supported KSP version, similar to the data provided by the SpaceDock API. Here is a link to the thread from 2017 for reference:

Hi, thanks for your input, I will look into manual parsing of the Curse page, probably with some threading functionality to speed thing up, will post here when I have an update!

Hi @HebaruSan, I noticed that the widget/API has now been updated to handle Curse after the change. I'm looking into some documentation on how to use the API for my application, specifically if it's possible to make a query to get a list of all mod id's, maybe you can help me with some quick hint...? Would it be possible to make a query to get all data for all mods in one call (or something like data for e.g. 100 mods in one call if there is a cap), to avoid calling the API for each mod? For KSP Mod Analyzer I'm looking for "mod name", "supported KSP version" and "URL to mod page" if possible.

Yes, that sounds like the best solution, I will wait for an update on the widget and then adapt my code. It's good that the widget developer is looking into it (see this CKAN issue).

Update: The Curse page layout has changed, the "Supported KSP version" field in no longer available on the list page (https://www.curseforge.com/kerbal/ksp-mods) This complicates things a bit, as I used that info to get the supported KSP version for each mod. Now, it seems I need to parse each individual mod page to the KSP version data, which will take time and also put load on the Curse server. Before, I just needed to parse each list page (42 pages in total); to get the same info now means parsing 42 * 20 = 840 pages... Any ideas on how to proceed...? Maybe if this API/Widget will be fixed I can use that, what do you think? Is there really no official API available for getting the data from Curse?

Thanks for this info, it seems the Curse page has changed and as I'm parsing the Curse page directly (I'm not using the API/Widget) I need to do some updates to make it work again. Stay tuned for an update...

Hi, I'm using the SpaceDock API for my KSP Mod Analyzer application and have noticed that sometimes the API gives inconsistent results. I'm using the API to get data for all mods, and the API may return +/- one mod compared to the previous run. To make this reproducible and easy to test, I created a small Python application found here: GitHub Gist The script uses the same functionality as in KSP Mod Analyzer but just writes the list of returned mods to a text file. It can be configured to run e.g. 3 times, creating three different output files. Comparing these shows that sometimes there is difference between the runs, e.g. one mod may be missing in the first run. Note that which mod is affected seems random. Just like to highlight this, I hope my test script may be helpful in reproducing the problem finding the root cause. Please let me know if I can do some more troubleshooting from my side. Note that KSP Mod Analyzer has a cool down period of 60s between each run (to limit the traffic to the server), but this does not seem to solve the problem.

KSP Mod Analyzer 1.1.1 is released, CSV export added as discussed above, please enjoy. The stand alone version can be downloaded here.

@linuxgurugamer OK, I have a working CSV exporter now (with user configurable delimiter set to "," in this example), but I would like to get some advice on the "escape character" and how to handle double quotes in field values. Here are some examples (first row is the header), note that the exporter takes the current view (e.g. "All mods", "Mods only on curse" etc) automatically. 1) Double quotes around all fields, and an escape character "\" preceding the double quotes in the URL:s "Mod","SpaceDock","Curse","CKAN","Source","Forum" "- Rocket Factory -","","<a href=\"https://mods.curse.com/ksp-mods/kerbal/243373-rocket-factory\">1.2-pre</a>","","","" "10km Omni Antenna for Kerbals on EVA for Remotetech","<a href=\"https://spacedock.info/mod/1218\">1.3.0</a>","","1.3.0","","" 2) No double quotes around fields, same "\" preceding the double quotes in the URL:s Mod,SpaceDock,Curse,CKAN,Source,Forum - Rocket Factory -,,<a href=\"https://mods.curse.com/ksp-mods/kerbal/243373-rocket-factory\">1.2-pre</a>,,, 10km Omni Antenna for Kerbals on EVA for Remotetech,<a href=\"https://spacedock.info/mod/1218\">1.3.0</a>,,1.3.0,, 3) Using one additional double quote for "double quotes in field values" as follows: "Mod","SpaceDock","Curse","CKAN","Source","Forum" "- Rocket Factory -","","<a href=""https://mods.curse.com/ksp-mods/kerbal/243373-rocket-factory"">1.2-pre</a>","","","" "10km Omni Antenna for Kerbals on EVA for Remotetech","<a href=""https://spacedock.info/mod/1218"">1.3.0</a>","","1.3.0","","" Not sure which option is considered best practice, please let me know your view. Note that the escape character in 1) and 2) can be changed to any character. Here is the Python code for reference, settings are changed in the parameters to "writer": def export_csv(self): """Exports the current view to a CSV file.""" suggested_filename = "mod_export" filename, _ = QtWidgets.QFileDialog.getSaveFileName(self, "Save File", QtCore.QDir.homePath() + "/" + suggested_filename + ".csv", "CSV Files (*.csv)") if filename: # Get rows and columns from data model rows = self.model.rowCount() columns = self.model.columnCount() with open(filename, 'w', newline='', encoding='utf-8') as csvfile: writer = csv.writer(csvfile, delimiter=',', escapechar='\\', doublequote=False, quoting=csv.QUOTE_ALL) #writer = csv.writer(csvfile, delimiter=',', doublequote=True, quoting=csv.QUOTE_ALL) # Write the header header = [self.model.headerData(column, QtCore.Qt.Horizontal, QtCore.Qt.DisplayRole) for column in range(columns)] writer.writerow(header) # Write the data records for row in range(rows): fields = [self.model.data(self.model.index(row, column), QtCore.Qt.DisplayRole) for column in range(columns)] writer.writerow(fields)

I'm using PyInstaller, it works really well and is in active development, here is the GitHub repo.

Sure, let me look into that, should be easy to implement. My idea is an "Export to CSV" button that exports the current view. Please let me know if you have any other improvement ideas as well.

Hi, maybe a bit off topic, but as you mentioned on the readme that you have plans for a future update with a QT user interface, I just like to mention that I have built my application KSP Mod Analyser in Python 3 + PyQt 5. It just analyzes all available mods on SpaceDock, Curse and CKAN and aggregates the meta data for a clean presentation (no mod manager features), but maybe some ideas can be reused. You can find my project on GitHub, and here is the forum thread. I used "PyInstaller" for packaging the stand-alone version which works without having Python installed.

Hi, that would be very helpful!

Hi, I just release a new version of KSP Mod Analyzer, see details on first page. Enjoy and please let me know if you have any improvement ideas or any other feedback. Regards, Ake74

@Thomas P. Would it be possible to generate a ZIP with all the JSON data and allow me to download that with my application? That would probably be more efficient that requesting all sub pages. Maybe the ZIP could be generated automatically on given intervals, like once 1h, once a day. Just an idea...