ModStatistics 1.0.3 - Anonymous mod usage statistics - Now for public distribution!

[Majiir]

Awesome. Perhaps just let authors see the crash data as they are the only ones who really need it?

I'd rather not restrict it like that. It helps users to see which mods are unreliable. On that note, I also have an obligation to make sure the analysis is not misleading. What's up on the site right now is just exploratory, and every listing has some flaws. (If anyone has a stronger statistics and machine learning background, I'd certainly appreciate help!)

[codepoet]

It would be interesting to see the numbers of a particular version of a mod in the wild so a mod author can track how many folks have not yet upgraded.

[blizzy78]

[EDIT 2] And here's average total loading time: http://stats.majiir.net/loading

I'm not sure what this tells us. Is that something like, "the average loading time of users who have plugin X installed is ~Y seconds"? As the list is right now, it is not very useful, because it leads to the wrong conclusions. For example, I can't think of a reason why PartHighlighter should increase loading times by a large amount.

So it turns out computing statistics is hard.

Personally, I'd use Splunk for analysis and graphing things. I'm sure there are other neat tools as well.

Splunk example:

Edited June 5, 2014 by blizzy78

[Justin Kerbice]

@Majiir: previous reports from yesterday have not been sent (maybe due to the exception below).

I got:

[ModStatistics] report-0.json sent successfully

(Filename: C:/BuildAgent/work/ea95e74f6e5f192d/Runtime/ExportGenerated/StandalonePlayer/UnityEngineDebug.cpp Line: 54)

[ModStatistics] Could not upload report-1.json:

System.Net.WebException: The remote server returned an error: (403) Forbidden.

at System.Net.HttpWebRequest.CheckFinalStatus (System.Net.WebAsyncResult result) [0x00000] in <filename unknown>:0

at System.Net.HttpWebRequest.SetResponseData (System.Net.WebConnectionData data) [0x00000] in <filename unknown>:0

[Majiir]

It would be interesting to see the numbers of a particular version of a mod in the wild so a mod author can track how many folks have not yet upgraded.

It would, yes. The plugin collections assembly file and informational versions, but unfortunately, it's hard to tell which of these is correctâ€â€if either.

I'm not sure what this tells us. Is that something like, "the average loading time of users who have plugin X installed is ~Y seconds"? As the list is right now, it is not very useful, because it leads to the wrong conclusions. For example, I can't think of a reason why PartHighlighter should increase loading times by a large amount.

Yes, it's the average loading time of installations which include that mod. There are a few things going on with the current analysis. There's not enough data, so there are going to be irregularities here for a while; crashes aren't filtered out, so one person who let the loading screen sit there frozen for 22 minutes really skewed things; and most importantly, the analysis is flawed to begin with. It's not wrong, it's just not a particularly useful stat yet. With large volumes of data, we might see that certain mods stand out, but a proper analysis is more complicated.

To expand on that: Ideally, we'd like to know how long each mod takes to load on average hardware. We have access to a list of installed mods and the total loading time. In principle, it's possible to work this out, but only if you have a lot of data. You can't very well use data that spans multiple users because users have different hardware. In effect, each user has a different load time multiplier. You could try to find users who have identical mod lists and compute the factor between them, but there will be few such coincidences. What you really have is a more complex system: lots of users which all overlap, and lots of runs for each user with potentially different mod combinations. The part that I don't know how to solve is the simultaneous over- and under-specification. The system is under-specified because some mods are never used alone, or not used in enough combinations to pin down their contribution to load time. It's over-specified because popular mods will show up a lot, and you need to be able to average those results. I may just be blanking on something I should know, but this seems like a difficult problem. Perhaps I can think of it as fitting a line to data in n-dimensional space where n is the number of mods, but then I'm not sure how different users play into it. (Maybe it should be n+1 dimensions? But then users need to be properly distributed along their dimension, and that's part of the challenge.)

Personally, I'd use Splunk for analysis and graphing things. I'm sure there are other neat tools as well.

I'm not sure an analytics package is going to help here. Simple queries are easy, and harder ones are difficult to even conceptualize. If you can point me to specific capabilities, I'll give it a look.

Right now I'm using MongoDB's aggregation pipeline, and it works nicely if you know what you're trying to compute. I could certainly use something else, but working with MongoDB directly keeps the whole thing simpler.

@Majiir: previous reports from yesterday have not been sent (maybe due to the exception below).

Which version are you using? The server is only accepting reports from 0.0.3.

[Justin Kerbice]

It would, yes. The plugin collections assembly file and informational versions, but unfortunately, it's hard to tell which of these is correctâ€â€if either.

It may be even impossible if author didn't take time to add version to the library . (md5/other ? + reference like pluginchecksumA = 1.0.0.1, pluginchecksumB = 1.0.2 sounds like a real pain somewhere to maintain)

Which version are you using? The server is only accepting reports from 0.0.3.

0.0.3, this particular report (report-1.json) has been removed, so I guess it has been uploaded, confirmed by:

[ModStatistics] report-1.json sent successfully

And it's do the same again for second report:

[ModStatistics] Could not upload report-2.json:

System.Net.WebException: The remote server returned an error: (403) Forbidden.

at System.Net.HttpWebRequest.CheckFinalStatus (System.Net.WebAsyncResult result) [0x00000] in <filename unknown>:0

at System.Net.HttpWebRequest.SetResponseData (System.Net.WebConnectionData data) [0x00000] in <filename unknown>:0

Looks like your plug-in is some kind of psycho-spaceball , first time it crash, second time, it is ok.

[blizzy78]

If you can point me to specific capabilities, I'll give it a look.

Mind you, I'm not a Splunk guru. Even though we could use it at work, we don't, so the only experience is what I got from private use on my server. It has an awful lot of searching (filtering) and aggregation commands (see http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Top, then expand "Search Command Reference" on the left), and it can display all sorts of reports and diagrams. Personally I don't think I'm using even 20% of what it can do.

So in that regard, it would be better to ask what info you would want to see, and then try to figure out how to do that with Splunk. (For example, a graph over time that shows the average KSP loading time of users that have plugin X and Y and Z installed. Such a graph would be more on the easier side, I think.)

Oh, and you could correlate things with crash reports, even if crash reports and regular statistics are transmitted separately.

Edited June 5, 2014 by blizzy78

[Majiir]

Looks like your plug-in is some kind of psycho-spaceball , first time it crash, second time, it is ok.

Do you have multiple ModStatistics.dll plugins present? It's possible that version negotiation is failing. Also: are you behind some kind of proxy or filter? There are some requests with a blanked-out User-Agent header, which are also rejected.

[Majiir]

ModStatistics 1.0.0 has been released. The download link in the first post has been updated.

This is the first release suitable for redistribution. Details and best practices have been added to the first post. Please read that information thoroughly before including ModStatistics with your download.

Additionally, the site has been revamped a bit, although there is now only one listing. I'll be expanding it over time. As before, if there is a particular stat you'd like to see, let me know.

[EDIT] One more thing: A big thanks to everyone who tested the beta! Your feedback was immensely useful. The stats you contributed over the last three days helped a great deal as well.

Changes in this version:

• Assembly title is now reported. This is a name contained by each plugin and not bound to the DLL filename, which will improve mod identification. (This should fix the ModuleManager categorization issues.)
  
• Reports no longer freeze the game while sending.
  
• Reports are now sent only when the game starts. This provides consistency between crash and non-crash reports and eliminates the delay when closing the game.
  
• On startup, the plugin installs the latest version of itself into the GameData/ModStatistics folder. This ensures that the plugin version is not bound to the mod distributing it.
  
• Removed the blank settings file from the download package and improved the instructions in the file that the plugin generates on first run.
  
• Reduced debug messages to avoid spamming the output log.

Edited June 7, 2014 by Majiir

[Justin Kerbice]

The plugin detects other copies of itself and only runs the latest version it finds. It also manages its own directory structure, so you should not place it in GameData/ModStatistics within your download. Instead, place it in your own mod's folder (e.g. GameData/YourMod/ModStatistics/ModStatistics.dll). This ensures your mod is properly detected, even if it is a pluginless mod.

I know you've work a lot I guess to avoid conflicts or weird issues of many kind but should it not be better to have:

Gamedata/ModStatistics/ModStatistics.dll

which is a "global oversight"

and:

GameData/modA/ModStatistics.cfg

GameData/modB/

GameData/modC/ModStatistics.cfg

modB is not surveyed, A and C are, cfg file may contains specific items to each mod (include crash data or not, monitor parts if such option is added later, or anything which could be useful to mod author)

As you have made it for now, it is a kind of dll invasion, and

they will negotiate amongst themselves, and only the latest version will execute.

is not very clear, how much dll are loaded ? Does one load after another then disabled itself ?

What if 3 latest versions are there ? All 3 will be executed ? Or only one (which one ?) of the 3 ?

[Majiir]

I know you've work a lot I guess to avoid conflicts or weird issues of many kind but should it not be better to have:

Gamedata/ModStatistics/ModStatistics.dll

The issue here is this can cause the ModStatistics directory structure (or individual files within) to be overwritten when another mod is installed. Additionally, users are used to erasing such directories before installing mods; that can cause issues when a mod is distributed with an out-of-date version. Finally, pluginless mods cannot currently be detected unless a plugin (like ModStatistics) is in their directory structure.

If the packaging instructions are followed, the user is unlikely to notice that multiple files are present. It would also be acceptable, for example, to put the ModStatistics DLLs in GameData/YourMod/Plugins, where other plugins are already distributed. ModStatistics is very small, so I don't think this is an issue.

GameData/modA/ModStatistics.cfg

GameData/modB/

GameData/modC/ModStatistics.cfg

modB is not surveyed, A and C are, cfg file may contains specific items to each mod (include crash data or not, monitor parts if such option is added later, or anything which could be useful to mod author)

I will be adding configuration options in a future version, but for now there's nothing to put in those files.

is not very clear, how much dll are loaded ? Does one load after another then disabled itself ?

What if 3 latest versions are there ? All 3 will be executed ? Or only one (which one ?) of the 3 ?

Only one instance of the latest version of ModStatistics will run.

Note that I'm looking into ways to have ModStatistics erase duplicate versions, but that's a technical challenge. (Assemblies cannot be unloaded without unloading the entire AppDomain, so I'd have to do something like create a new AppDomain which runs a cleanup task, or maybe even spawn a separate process.)

[Justin Kerbice]

ok thanks.

Note that I'm looking into ways to have ModStatistics erase duplicate versions, but that's a technical challenge. (Assemblies cannot be unloaded without unloading the entire AppDomain, so I'd have to do something like create a new AppDomain which runs a cleanup task, or maybe even spawn a separate process.)

IMHO, this is VERY BAD ! It's kind of Microsoft programming style (="the average user is too stupid to understand" + doing thing behind the back of people on their behalf).

The best way I see is like standard "shared" plug-in, ex FireSpitter, many mods use it and there is (shoud be) only one single file.

I know it's not the same usage of course, so the config file style given before may be preferred, with maybe a global finding/parsing of configs like MM does.

[Majiir]

IMHO, this is VERY BAD ! It's kind of Microsoft programming style (="the average user is too stupid to understand" + doing thing behind the back of people on their behalf).

I'm not sure what's so bad about it, but it doesn't matter anyway. Deleting them just cleans up the plugins that aren't running to begin with.

The best way I see is like standard "shared" plug-in, ex FireSpitter, many mods use it and there is (shoud be) only one single file.

I know it's not the same usage of course, so the config file style given before may be preferred, with maybe a global finding/parsing of configs like MM does.

If every modder puts the file in the same path as you suggest, modders need to constantly stay up-to-date with ModStatistics or they'll end up overwriting new versions with old ones. This is easier on modders because they can package it once and not worry about ModStatistics updating. (Another mod will probably come along with a more up-to-date version.) It's easier on users because they don't have to figure out how to merge the ModStatistics folder. (This would inevitably cause problems when users merge incorrectly or delete the old folder; or they'd delete the new one, and thus not receive an update. Directory merging has been a constant source of confusion.)

I know that this is a little unusual, but as you say, it's a different use case. This is somewhat similar to ModuleManager and Toolbar, and the system I've designed solves many of the problems those mods have experienced. If you have specific concerns, let me know, but I think this will work quite well for all involved.

[willow]

Just a generic comment...

I think it would be highly undesirable if users would inadvertantly install this plugin via a third party mod and have it upload data to servers without their knowledge. No matter how anonymous the information is. Let me put it another way. In Europe information gathering like this by law should be an opt-in scenario. If it happens behind my back, you're breaking the law.

Admittedly almost every smartphone out there is laden with software who in some shape or form does this behind our backs. But this doesn't change the law or ethics.

You described the process as opt-out... I think it should be opt-in.. Upon startup of KSP a dialog may explain to me exactly when and what kind of information is going to be sent to the servers so I may make an informed decision. But I as an end user should clearly give my consent. And then if you'd honestly tell me the benefits I'd be giving to the mod makers I'll probably say yes.

However, curse gaming already keeps pretty good statistics about mod downloads, which is a close indication of usage. Meaning that this plugin is superfluous.

Consider alternatives that you can offer as services to the user while they achieve your goals at the same time.

You could change your plugin so it does automatic update checks for the user. Which gives your server the same information and adds benefits for the users. You'd also need to offer a repository for mods on your server though. Proxying to curse wouldn't be the same. You could add in game error reporting where users can upload bugs to your servers which benefits both the users and the mod makers, and gives you usage statistics. You could for every bug send a mod list to the server. Because that's relevant information when reporting a bug. You could also consider ingame feature requesting to this system which adds the same benefits. You could offer a list of update information so the user is informed of version information for every mod in their game.

In fact a plugin which finds updates to my mods (Or as yet uninstalled mods) and offers to install them for me in game would definately be on my wish list. That you learn about the mods I have installed in the process isn't that big a thing to me.

[codepoet]

You could change your plugin so it does automatic update checks for the user.

I like this idea - you know what version the user has and you can check if other users have a higher version based on the reports of other users, so you can figure out that a user is out of date without even having to worry about what is on curse or any other download site.

No comment on the legal stuff.

[Faark]

I'm not sure what's so bad about it, but it doesn't matter anyway. Deleting them just cleans up the plugins that aren't running to begin with.

Well there are minor effects like they at least having some memory impact and the current version adds ~10 types, so all code that does reflection will be slowed down a tiny bit. That's ofc not significant unless this mod grows in complexity or is very frequently updated. I though about that for the error reporting stuff i'll probably never have the time to realize anyway, and one approach might be to manually load assemblies. KSP won't touch them as long as they are named *.notdll, allowing the mod to load or delete whatever dependencies it likes...

But imo the more serious problem is trust. Automatically downloading and running code can end very bad for the user.... very very bad. Who cares about privacy of uploaded data when you can push code that uploads all my password files without me noticing. I certainly wouldn't install anything like that from some community member that isn't that well respected as you, but it still doesn't feel great.

[DMagic]

However, curse gaming already keeps pretty good statistics about mod downloads, which is a close indication of usage. Meaning that this plugin is superfluous.

I believe the whole point of all this is that simple download numbers aren't very valuable. They tell you very little about actual usage.

Does this generate a file with the info being sent? I think that might help assuage some of people's worries about data privacy.

Edited June 7, 2014 by DMagic

[Justin Kerbice]

Regarding downloads, I, myself usually download some mods but not install them right away, sometimes left one, two, ... n versions even before I paid them some attention, like KAX for example. Then I install a single mod on a test install for a few days/weeks/months, and then and only then, I install he mod on my main game.

So, KAX can have a few more downloads without any more use. That's make such counter unreliable at all, without mentioning people redownloading a mod because they erase it by accident, or any other reason.

[Majiir]

I think it would be highly undesirable if users would inadvertantly install this plugin via a third party mod and have it upload data to servers without their knowledge.

This is why I notify users in a prominent place in this thread, and I instruct mod authors to do the same. This isn't the first plugin to contact a remote server; some are quite popular and don't even tell users about it. If you see a mod you want to play and it has ModStatistics, and you don't want to contribute, then run the game once, let ModStatistics generate the settings.cfg file, and change the disabled flag. No data is sent on the first run.

You could change your plugin so it does automatic update checks for the user. Which gives your server the same information and adds benefits for the users. You'd also need to offer a repository for mods on your server though. Proxying to curse wouldn't be the same. You could add in game error reporting where users can upload bugs to your servers which benefits both the users and the mod makers, and gives you usage statistics. You could for every bug send a mod list to the server. Because that's relevant information when reporting a bug. You could also consider ingame feature requesting to this system which adds the same benefits. You could offer a list of update information so the user is informed of version information for every mod in their game.

I have offered my server resources for hosting a community-run mod repository. The coder of one of the candidate sites has already remarked that there's potential for integration between ModStatistics and their repository. I have a lot of "meta-modding" projects, and this is one of them. I think they're all useful on their own, but clearly there's a possibility they'll converge. It's far too early in any project to give it much serious thought, though.

Well there are minor effects like they at least having some memory impact and the current version adds ~10 types, so all code that does reflection will be slowed down a tiny bit. That's ofc not significant unless this mod grows in complexity or is very frequently updated. I though about that for the error reporting stuff i'll probably never have the time to realize anyway, and one approach might be to manually load assemblies. KSP won't touch them as long as they are named *.notdll, allowing the mod to load or delete whatever dependencies it likes...

The performance impact will be truly negligible. The *.notdll idea is interesting, but there needs to be a plugin to load those in the first place. If I figure out a good way to clean up DLLs, I'll raise the discussion again and we'll see what works best.

But imo the more serious problem is trust. Automatically downloading and running code can end very bad for the user.... very very bad. Who cares about privacy of uploaded data when you can push code that uploads all my password files without me noticing. I certainly wouldn't install anything like that from some community member that isn't that well respected as you, but it still doesn't feel great.

While I recognize this guarantees nothing, I'll say: If I wanted to do something malicious, I'd have already used Kethane or KAS as delivery mechanisms; or I'd have more aggressively pursued Replaceport with my own code to harvest passwords; or, years ago when I discovered a public-facing database with thousands of plaintext passwords, I'd have saved a copy instead of typing the drop-column command as fast as possible.

Updating and reporting can both be disabled. ("update=false" in settings.cfg will prevent it from checking for new versions; I need to add that property to the default config file.) Honestly, lack of SSL is a far bigger problem than I ever will be. I did try a few times to add it, but Mono requires you to implement certificate validation logic yourself, and I'm loathe to include something with the appearance of security unless it's really secure. (Of course, if someone is MITM'ing your anonymous usage logs, you have bigger problems.)

Does this generate a file with the info being sent? I think that might help assuage some of people's worries about data privacy.

Yes. As of the latest version, files are generated only when the game closes, and they are sent only when the game starts. So you can run a session, close KSP, and inspect the file to see what will be sent.

[TaranisElsu]

As of the latest version, files are generated only when the game closes, and they are sent only when the game starts. So you can run a session, close KSP, and inspect the file to see what will be sent.

I really like that. Mainly so that I can make sure my mods are being reported correctly (making sure that I am filling out the Assembly information encoded in the DLL properly).

[JewelShisen]

HL Mods will be including this in the next update.

[codepoet]

I am consulting other modders on including this in the next release of ConnectedLivingSpace.

[willow]

This is why I notify users in a prominent place in this thread, and I instruct mod authors to do the same..

Perhaps the place should be on first ksp startup? It seems to me that most users will not have read this thread if somewhere in the future they load up a mod that includes your software.

It should be possible for you to cause a popup right? And that way you've solved all ethical issues.

You say a lot of mods upload data without telling us about it.. Could you list them so I can uninstall them?

[Majiir]

JewelShisen and codepoet: Cheers. I'll appreciate you letting me know when you do end up releasing it so I can figure that into my query design.

It seems to me that most users will not have read this thread if somewhere in the future they load up a mod that includes your software.

I ask modders to include a notice in their thread in a prominent place. This was done both as a courtesy to users and also to leave no doubt about compliance with the forum rules. I'm considering a popup, but I want to test a few different things first.

You say a lot of mods upload data without telling us about it.. Could you list them so I can uninstall them?

I said some mods contact a remote server. (The very act of connecting transmits some information, but that's distinct from uploading application data.) Toolbar, for instance, contacts an update server. It can be disabled, but there aren't warnings plastered everywhere as with ModStatistics.

--------

On another note: I've discovered an odd issue with report submission as of 1.0.0. The server logs show that a large majority of clients that are able to contact the server are not submitting reports. At first, I thought it was one or two users manually deleting the reports every time they were generated, but there are too many instances of the issue now for that to be the case. (Only one user has been able to submit 1.0.0 reports.) I'm working on a patch that will shed some light on the issue. In the meantime, if you're using 1.0.0, please give your GameData/ModStatistics folder a look and tell me what you see. In particular, I'm interested in whether checkpoint.json exists while KSP is running, and whether report-#.json files are stacked up. I suspect there's an issue with file write permissions that's preventing reports from being saved for transmission at all. (To be clear: I think this is an issue affecting all versions released thus far, but only 1.0.0 made it discoverable on my end.)

[codepoet]

Surely the place to notify users is in the license. "If you want to use this software then you must agree to this ...." No-one can reasonably argue that it is not reasonable to read the license.

Perhaps a standard text could be offered that modders can add to their license if they want to ship with ModStatistics.