KSP Forums Archival Options.

[Minmus Taster]

I felt this needed it's own thread separate from the update and discussion thread, I'm not sure where this belongs so feel free to either move or remove/merge it.

What are we going to do as a community without the forums? Speaking both about the wider community and how this site acts as a hub and to the crowd that spends most of their time here.  I know there's a subreddit (Jeb help them) which we could migrate to in the event of a shutdown.  Also concerned about the modding scene, I know that most of the major mods and developers that are popular in the community are based here. And when it comes down to it I don't want this community which I've come to love to be erased from the internet, I hope we can come up with a survival plan.

[Scarecrow71]

The unfortunate answer, should the forums get shut down, is to use Discord.  I don't like that very much either, but unless someone wants to pony up for a new forum site, that is the only answer.

[PDCWolf]

1 hour ago, Scarecrow71 said:

The unfortunate answer, should the forums get shut down, is to use Discord.  I don't like that very much either, but unless someone wants to pony up for a new forum site, that is the only answer.

*points at the subreddit.*

Yes, I know, it's not a forum and you can't host threads for mods in the same way it works here, but it is infinitely better than discord for hosting links. Nothing more annoying than stuff not being google-able because people put it inside discord servers. If you wanna kill the modding scene, make it all unsearchable by hiding it behind a discord.

[Scarecrow71]

18 minutes ago, PDCWolf said:

*points at the subreddit.*

Yes, I know, it's not a forum and you can't host threads for mods in the same way it works here, but it is infinitely better than discord for hosting links. Nothing more annoying than stuff not being google-able because people put it inside discord servers. If you wanna kill the modding scene, make it all unsearchable by hiding it behind a discord.

Yeah, I forgot Reddit.  Might be because a lot of the people there are kids who can't use the search function before asking if they should buy KSP2.  But you are correct that it exists and could be used.

[RayneCloud]

Hey all!

As many of you know, I'm the former KSP 1 CM (many of you knew me by the moniker of Damion Rayne back in the day.) As the first CM (next to my old talented friend CptSkunky) I still feel responsible for this community, this game, this wonderful franchise that gave me my start in the industry all those years ago.  So, that being that...  I just wanted to take a moment to discuss our options for Archival, if there really are any. (Which I think there are.)

Firstly, I was the person who was directly responsible for choosing VBulletin5 Connect as the forum software solution and upgrade from the original forums. I was not, however, involved in the move to Invision Community which currently powers the forums as that happened well after my departure from the project. That being said, IC is in its 4.x.x versioning. This solution is a PHP/MySQL based forum solution, which means it's written in PHP and for database storage uses MySQL  This is just some base level information I thought we should all know about. 

Now, in short, what's this mean for me? Well,  I have no admin level experience with this forum software, nor do I know how its database backup options work. Not that we'd get the database backup regardless....

Now, that's all well and good but what we are actually looking at here?

206,000+ topics across 4.3 million total posts with user data for 226,873 unique users. As I said earlier getting ahold of a database backup is, as far as I know, pretty much impossible without access to the admin back end and direct access to the database itself. That, and the backups are no doubt quite large. 

So, that leaves us, as a community, with one option..

Pulling the sites data directly from the server while it's still operational using several tool options. Now at a concern of possibly talking about stuff the mod team wouldn't want me talking about, I've reached out to some people in private to let them know what these tools and options are. None of these things are illegal, as they're mostly used to pull down private viewing backups to your local machine. 

Now, if the mod team allows it, I'll go over the tools and options here later in the thread. Also, if anyone wants to, please feel free to mirror this to reddit, as we might be able to pull some talented peeps from that part of the community to help out. (I'm LadyRaineCloud over there as a note, I just don't post all that often.)

Anyway, thoughts? Options? Anyone want to sit down and talk this out? Let me know!

Edited July 8 by RayneCloud

[RayneCloud]

Okay, so, let's start with some tools and options that might work..

https://www.httrack.com/

https://www.cyotek.com/cyotek-webcopy

https://wiki.tcl-lang.org/page/Getleft

There's also potentially using a Cygwin wget command - https://cygwin.com/packages/summary/wget.html Cygwin is a posix emulator for linux, if memory serves, and has some options we can use to pull down or clone a local copy of the forums. 

As a note, this would potentially cause increased traffic and load on the server while the pulldown / mirroring is happening and that might potentially crash the server. 

Edited July 8 by RayneCloud

[Superfluous J]

6 hours ago, Minmus Taster said:

What are we going to do as a community without the forums?

I can't speak for the community, but as one of its members I'll speak for me.

Without this Forum I'll stop interacting with you all. I won't frequent Discord, Reddit, or any other site. I don't play KSP or KSP2 anymore and I don't generally visit sites or channels for games I don't play.

KSP is actually a huge exception from that rule because I've barely played for years yet am still here.

Edited July 8 by Superfluous J

[Lisias]

There's the tool the Internet Archive uses:

https://archive.org/developers/internetarchive/index.html

Somewhat more complicated than httrack (what I had used for a decade, by the way), but it's how these guys do things.

It also worths to mention that I found this too:

https://archive.org/details/forum.kerbalspaceprogram.com_202305

Updated at May 2023 with this tool.

I'm reading that documentation to see if we could use the 2023.5 dump as a baseline to save Forum's bandwidth.

Additionally, I think it's a very good idea to have this dump shared between many people, again to save Forum's bandwidth - there's no need to every single user here to do the job by themself, we can collaborate.

[RayneCloud]

Much agreed Lisias.

[Gargamel]

Moved to the network. 

[Gargamel]

I have merged these two threads,  I ended up swapping thread Title and OP in the process, as it better describes the situation.  There might be a little confusion to the merge, but it’s early enough we’ll get over it.  

[xD-FireStriker]

6 hours ago, Superfluous J said:

I can't speak for the community, but as one of its members I'll speak for me.

Without this Forum I'll stop interacting with you all. I won't frequent Discord, Reddit, or any other site. I don't play KSP or KSP2 anymore and I don't generally visit sites or channels for games I don't play.

KSP is actually a huge exception from that rule because I've barely played for years yet am still here.

Same here, I do still play KSP from time to time but KSP is the only game community I haven't peaced out from. 

I'm in the KSP Reddit discord but its a mess, the IG discord is here to stay but it never felt as homely as the fourm. 

2 hours ago, Lisias said:

I'm reading that documentation to see if we could use the 2023.5 dump as a baseline to save Forum's bandwidth.

Is that a complete backup? 

[Scarecrow71]

Is there talk in here somewhere about starting a new forum if this one gets shut down?  I see talk above about backing this one up, but nothing about a new forum.

I would be highly interested in having a conversation about starting up a new forum should it come to that.  Moderators, cost, DBA work, front end scripting...all the pieces.

[Lisias]

1 hour ago, xD-FireStriker said:

Is that a complete backup? 

I will check this by night (GMT-3). Keep in mind that that is a dump from the Browser point of view, we just don't have access to the Forum inner guts, so no private messages or anything that demands being logged on to see.

[Bellabong]

Well I hope that stays that way because I'd be very concerned if that kind of data made it to a different entity, no matter how honourable. 

[PDCWolf]

2 hours ago, Gargamel said:

Moved to the network. 

I know Network is definitely the right place, not questioning the move. What I want to suggest is that given the perceived urgency of this topic, and it's massive importance to the modding community and the community in general to safeguard the presence of said modding scene (which is also the biggest thing keeping the KSP1 alive). That this should not be hidden so far down the forum even if the section is the correct one it should be in.

Where it is right now, no one that isn't subscribed to the topic or has posted in it will see it.

[Gargamel]

38 minutes ago, PDCWolf said:

I know Network is definitely the right place, not questioning the move. What I want to suggest is that given the perceived urgency of this topic, and it's massive importance to the modding community and the community in general to safeguard the presence of said modding scene (which is also the biggest thing keeping the KSP1 alive). That this should not be hidden so far down the forum even if the section is the correct one it should be in.

Where it is right now, no one that isn't subscribed to the topic or has posted in it will see it.

There’s a redirect from the announcement page.   All other discussion about this will be directed here.    There won’t be a need to move this anywhere else at this time. 

Edited July 8 by Gargamel

[Poodmund]

Essentially this has already been discussed elsewhere, and movement started on this a while back with a current estimation (of public facing content) that a static snapshot of the forum as of around 3 days ago totals to roughly 8.5GB in size. Its not "exactly" a static snapshot as bandwidth is being cranked right down using a single connection as to not put any considerable load onto the site, and therefore there may be some desync in posts but as its front facing content using a drill-down approach its good enough.

If this matures and is workable then, at the very worst, we would have likely a monthly fall back to an archive of sorts. I'm speaking entirely on behalf of those working on this so they can continue in quiet without being hassled.

[PDCWolf]

5 hours ago, Gargamel said:

There’s a redirect from the announcement page.   All other discussion about this will be directed here.    There won’t be a need to move this anywhere else at this time. 

And with that original thread locked it slowly fades down the order, which is obviously telling you something you already know... which is why I made that suggestion, but hey, at least now it's publicly clear it might be intended behavior for the topic to be forgotten fast.

2 hours ago, Poodmund said:

Essentially this has already been discussed elsewhere

This kind of mystery and subterfuge is exactly why people wanted to stay away from the KSP2 modding scene. All the more reason to have more than one backup and alternative.

[Poodmund]

Subterfuge? Ooooooh, so we have deceit and nefarious meetings now do we? I would say that 'people' rather 'wanted to stay away from the KSP2 modding scene' because the game state was a mess.

Hopefully this whole thing is completely irrelevant and the forum will stay in operation. That's the number 1 hope here.

[Lisias]

12 minutes ago, Poodmund said:

Subterfuge? Ooooooh, so we have deceit and nefarious meetings now do we? I would say that 'people' rather 'wanted to stay away from the KSP2 modding scene' because the game state was a mess.

The best way to earn trust is doing everything you can to prevent people to have to trust you.

People here are somewhat traumatized, and trust is becoming a pretty scarce commodity nowadays.

 

13 minutes ago, Poodmund said:

Hopefully this whole thing is completely irrelevant and the forum will stay in operation. That's the number 1 hope here.

I completely agree with you here. Hope for the best, but expect the worst.

The hard part will be to expect the worst without causing it - like a self-fulfilling prophecy.

[Superfluous J]

4 hours ago, PDCWolf said:

at least now it's publicly clear it might be intended behavior for the topic to be forgotten fast.

I'm so glad I wasn't drinking when I read this. I'd be cleaning my monitor now.

Hey you people doing this because you love the game and want to donate your time and work to the cause, admit it you are trying to sabotage efforts to preserve it!

1 hour ago, Lisias said:

The best way to earn trust is doing everything you can to prevent people to have to trust you.

If something as innocuous as putting a thread where it belongs causes conspiracy theories that even Mulder from X-Files would laugh at...

[Lisias]

9 minutes ago, Superfluous J said:

If something as innocuous as putting a thread where it belongs causes conspiracy theories that even Mulder from X-Files would laugh at...

Once bitten, twice shy.

Trust needs to be earnt - and once lost, it's twice as hard to earn it back. It's up to the one that lost that trust to balance the gains and losses of trying to earn trust back.

The first step is, well, exercising empathy.

[PDCWolf]

2 hours ago, Poodmund said:

Subterfuge? Ooooooh, so we have deceit and nefarious meetings now do we? I would say that 'people' rather 'wanted to stay away from the KSP2 modding scene' because the game state was a mess.

Of course, talking about sekrit club discords and saying you all have a backup of something similar but flexing that "you can't reveal anything" (so why say anything in the first place?) is either kids spy games or actual subterfuge. And yes, no one wants to mod a game that was on flimsy legs and couldn't be played properly, but also a lot of people didn't want to mod a game that requires you to join a discord and submit to arbitrary "standards" or be judged and shunned for not doing it.

Quote

Hopefully this whole thing is completely irrelevant and the forum will stay in operation. That's the number 1 hope here.

We agree.

48 minutes ago, Superfluous J said:

Hey you people doing this because you love the game and want to donate your time and work to the cause, admit it you are trying to sabotage efforts to preserve it!

If something as innocuous as putting a thread where it belongs causes conspiracy theories that even Mulder from X-Files would laugh at...

Considering how aggressively the discourse was "shaped" during the early life of the game, how even now the game is dead and there's no boss around and there's a looming threat of the forum just disappearing the discourse is still aggressively "shaped" towards one side, and then considering my own experiences with this forum on top... No, I don't think there's attempts at sabotage from anyone, on the contrary, there's a lot of attempts at forcing a view like it was before Dakota pulled on the reins after a couple debacles.  And no, before you hyperbole me again, I don't think every thread should just be hate or insults or criticism, even if at some point they became irrefutably well deserved and the community still needs to vent a lot.

The forum is barely hanging on between 502s and being completely headless, but hey, the poll that would've put discussions like these at the top got magically ignored, and now this thread made by a mod in one section gets kicked down by another mod, like they haven't worked together for ~10 years. It's funny, and worrying.

Trust being at an all time low isn't just for the devs and T2. It's for the whole machine because it has failed us multiple times and I don't mean the 502s with that.

Edited July 9 by PDCWolf

[Vanamonde]

Moving something to a sub farther down the page only "hides" it from people too lazy to look down the page. Now please get back to the actual subject of the thread.