Jump to content

KSP Mod Analyzer 1.1.1

Ake74

By Ake74
in KSP1 Mods Discussions

 Share

Recommended Posts

Ake74

Ake74

Posted
Posted (edited)

KSP Mod Analyzer 1.1.1 has been released, with the following new exciting features:

  • Support for SpaceDock, Curse and CKAN repositories
  • Clickable links to mod page, source code repository and forum if available
  • Filter on mod name
  • Many UI improvements
  • 60s cool down period after updating SpaceDock and Curse, to limit network traffic
  • Possiblity to do analysis on mods, e.g. "Mods only available on CKAN", with aggredated metadata
  • CSV export
  • Note! Due to recent changes to the Curse page, "Update Curse" is broken, I'm looking into the problem. 

In summary, this application gives you a clear overview of all all KSP mods with relevant metadata in one single application. Note that is is not a mod manager like CKAN, but I hope you find it useful anyway :-)

Please find download details below the screenshot.

0Bbu64V.png

Technical details

  • Written in Python 3 using Qt 5 for the UI
  • Source code is available on GitHub
  • Latest release, packaged as a stand-alone application, is available on the release page (Python is not required)

Let me know what you think about this application, feedback and improvement ideas are welcome.

Edited by Ake74
Problem updating curse
Link to comment
Share on other sites
HebaruSan

HebaruSan

Posted
Posted
17 minutes ago, Ake74 said:

Yes, GitHub is nice, there is a link to my project in the OP :cool: Or did you mean CKAN? I planning to add CKAN support in a future release.

I think he meant, what about a third column for Github inside of your application, to reflect whether each mod is available on Github.

Link to comment
Share on other sites
Ake74

Ake74

Posted
  • Author
Posted
Just now, HebaruSan said:

I think he meant, what about a third column for Github inside of your application, to reflect whether each mod is available on Github.

I see, that is indeed a very good idea! Let me look into that as well, if I'm not mistaken, links to the project/source for each is available using the SpaceDock API so that should be easy to implement. Not sure for Curse, as there is no API available and I'm parsing the HTML to extract the mod names, I will investigate that.

Maybe another option could be to use CKAN data as I believe there is a link to the project/source there as well?

Link to comment
Share on other sites
Thomas P.

Thomas P.

Posted
Posted

Please note, that spacedocks server will issue a temporary IP ban if users are doing too many requests in a short time intervall, to avoid a huge slowdown of the main site, and other services who are running on that server.

@Ake74 Maybe some kind of cooldown - built into the app directly - would be a good idea?

Link to comment
Share on other sites
Ake74

Ake74

Posted
  • Author
Posted (edited)
1 hour ago, Thomas P. said:

Please note, that spacedocks server will issue a temporary IP ban if users are doing too many requests in a short time intervall, to avoid a huge slowdown of the main site, and other services who are running on that server.

@Ake74 Maybe some kind of cooldown - built into the app directly - would be a good idea?

Hi, yes that is probably a good idea, here is how my application works:

1. Get the first page from SpaceDock (https://spacedock.info/api/browse?count=100). Note that I'm using "100 mods per page", please let me know if any other value is more preferred.

2. Store the JSON data and get the total number of pages (based on 100 mods in this example). Currently 11 sub pages.

3. Loop and generate a new URL to fetch the next page, store the JSON data for each page. This is repeated for each page, in total 10 more calls besides the initial URL.

Which limitation would you prefer I implement, maybe a timer that allows to run the whole data collection (all steps) only once every minute (or other value)? Or would you recommend a delay in the loop for the sub-pages? I would prefer a delay for the whole function, but I'm listening to recommendations from the SpaceDock admin :wink:

 

Edited by Ake74
Link to comment
Share on other sites
Ake74

Ake74

Posted
  • Author
Posted

@Thomas P. Would it be possible to generate a ZIP with all the JSON data and allow me to download that with my application? That would probably be more efficient that requesting all sub pages. Maybe the ZIP could be generated automatically on given intervals, like once 1h, once a day. Just an idea...

Link to comment
Share on other sites
  • 3 months later...
HebaruSan

HebaruSan

Posted
Posted

It was a slight hassle getting this to run on Linux, no fault of yours. First I was missing several dependencies, then I wasn't sure which file to run, then it turned out my 'python' command is Python 2 instead of Python 3. It works quite well now though.

Would you be interested in a README.md pull request with instructions?

Link to comment
Share on other sites
Ake74

Ake74

Posted
  • Author
Posted
5 minutes ago, HebaruSan said:

It was a slight hassle getting this to run on Linux, no fault of yours. First I was missing several dependencies, then I wasn't sure which file to run, then it turned out my 'python' command is Python 2 instead of Python 3. It works quite well now though.

Would you be interested in a README.md pull request with instructions?

Hi, that would be very helpful!

Link to comment
Share on other sites
  • 4 months later...
Ake74

Ake74

Posted
  • Author
Posted
5 hours ago, linuxgurugamer said:

Would it be possible to get this to dump it's data to a CSV?

Sure, let me look into that, should be easy to implement. My idea is an "Export to CSV" button that exports the current view. Please let me know if you have any other improvement ideas as well.

Link to comment
Share on other sites
Ake74

Ake74

Posted
  • Author
Posted

@linuxgurugamer

OK, I have a working CSV exporter now (with user configurable delimiter set to "," in this example), but I would like to get some advice on the "escape character" and how to handle double quotes in field values.

Here are some examples (first row is the header), note that the exporter takes the current view (e.g. "All mods", "Mods only on curse" etc) automatically. 

1) Double quotes around all fields, and an escape character "\" preceding the double quotes in the URL:s 

"Mod","SpaceDock","Curse","CKAN","Source","Forum"
"- Rocket Factory  -","","<a href=\"https://mods.curse.com/ksp-mods/kerbal/243373-rocket-factory\">1.2-pre</a>","","",""
"10km Omni Antenna for Kerbals on EVA for Remotetech","<a href=\"https://spacedock.info/mod/1218\">1.3.0</a>","","1.3.0","",""

2) No double quotes around fields, same "\" preceding the double quotes in the URL:s 

Mod,SpaceDock,Curse,CKAN,Source,Forum
- Rocket Factory  -,,<a href=\"https://mods.curse.com/ksp-mods/kerbal/243373-rocket-factory\">1.2-pre</a>,,,
10km Omni Antenna for Kerbals on EVA for Remotetech,<a href=\"https://spacedock.info/mod/1218\">1.3.0</a>,,1.3.0,,

3) Using one additional double quote for "double quotes in field values" as follows: 

"Mod","SpaceDock","Curse","CKAN","Source","Forum"
"- Rocket Factory  -","","<a href=""https://mods.curse.com/ksp-mods/kerbal/243373-rocket-factory"">1.2-pre</a>","","",""
"10km Omni Antenna for Kerbals on EVA for Remotetech","<a href=""https://spacedock.info/mod/1218"">1.3.0</a>","","1.3.0","",""

Not sure which option is considered best practice, please let me know your view. Note that the escape character in 1) and 2) can be changed to any character.

Here is the Python code for reference, settings are changed in the parameters to "writer": 

    def export_csv(self):
        """Exports the current view to a CSV file."""

        suggested_filename = "mod_export"
        filename, _ = QtWidgets.QFileDialog.getSaveFileName(self, "Save File",
                                                            QtCore.QDir.homePath() + "/" + suggested_filename + ".csv",
                                                            "CSV Files (*.csv)")
        if filename:
            # Get rows and columns from data model
            rows = self.model.rowCount()
            columns = self.model.columnCount()

            with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
                writer = csv.writer(csvfile, delimiter=',', escapechar='\\', doublequote=False, quoting=csv.QUOTE_ALL)
                #writer = csv.writer(csvfile, delimiter=',', doublequote=True, quoting=csv.QUOTE_ALL)
                
                # Write the header
                header = [self.model.headerData(column, QtCore.Qt.Horizontal, QtCore.Qt.DisplayRole) for column in range(columns)]
                writer.writerow(header)
                
                # Write the data records
                for row in range(rows):
                    fields = [self.model.data(self.model.index(row, column), QtCore.Qt.DisplayRole) for column in range(columns)]
                    writer.writerow(fields)

 

Link to comment
Share on other sites
  • 2 weeks later...
Ake74

Ake74

Posted
  • Author
Posted
On 11/9/2017 at 7:43 PM, HebaruSan said:

Is the Curse API working for you? CKAN is having problems with it, so I thought I would see how your app was handling it, and it gives me a divide by zero when I click Update Curse:

OwpfRyn.png

Thanks for this info, it seems the Curse page has changed and as I'm parsing the Curse page directly (I'm not using the API/Widget) I need to do some updates to make it work again. Stay tuned for an update...  

Link to comment
Share on other sites
Ake74

Ake74

Posted
  • Author
Posted

Update: The Curse page layout has changed, the "Supported KSP version" field in no longer available on the list page (https://www.curseforge.com/kerbal/ksp-mods) :/ This complicates things a bit, as I used that info to get the supported KSP version for each mod. Now, it seems I need to parse each individual mod page to the KSP version data, which will take time and also put load on the Curse server. Before, I just needed to parse each list page (42 pages in total); to get the same info now means parsing 42 * 20 = 840 pages...

Any ideas on how to proceed...? Maybe if this API/Widget will be fixed I can use that, what do you think? Is there really no official API available for getting the data from Curse?

Link to comment
Share on other sites
HebaruSan

HebaruSan

Posted
Posted
4 hours ago, Ake74 said:

Update: The Curse page layout has changed, the "Supported KSP version" field in no longer available on the list page (https://www.curseforge.com/kerbal/ksp-mods) :/ This complicates things a bit, as I used that info to get the supported KSP version for each mod. Now, it seems I need to parse each individual mod page to the KSP version data, which will take time and also put load on the Curse server. Before, I just needed to parse each list page (42 pages in total); to get the same info now means parsing 42 * 20 = 840 pages...

Yikes, that sounds awful. It might be worth contacting the KSP-Curse liaison to ask about this. If this was just an innocent site redesign rather than a deliberate effort to suppress third party parsing, maybe they'd be amenable to making the format easier to handle. I'll see if I can find their forum username...

4 hours ago, Ake74 said:

Any ideas on how to proceed...? Maybe if this API/Widget will be fixed I can use that, what do you think? Is there really no official API available for getting the data from Curse?

The widget has a bug report to track this:  https://github.com/citricsquid/widget/issues/44

They do seem to be planning to fix it, but it sounds difficult for the reasons you described. I suppose the cleanest solution would be to adapt your app to use the widget and then help the widget folks figure out how to fix it (it's in PHP), but that's your call.

As for an official API, I just assumed there was one when this all started, but there must not be if everyone (you, CKAN, and the Minecraft folks that maintain the widget) relies on parsing their HTML instead.

Link to comment
Share on other sites
HebaruSan

HebaruSan

Posted
Posted

Hey @Jadedcat and @Kaelten (please let me know if you're no longer the KSP-Curse liaison team), we noticed that the Curse[forge] web site was redesigned recently. This has caused some difficulties for third party tools that attempt to monitor the site for changes so interested users can be directed to mod pages when there are updates. @Ake74 posted a description above if you need to forward the details to someone, and there's also a team of Minecraft folks who maintain a forum widget for much the same purpose.

Would you be open to modifying to the site to facilitate this monitoring? Having Curse pages linked from third party tools should help to drive a few more page views.

(I have no idea whether either of those users are active enough to receive a notification, but maybe it's worth a try.)

Link to comment
Share on other sites
This thread is quite old. Please consider starting a new thread rather than reviving this one.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...