So, we had some kind of technical problem.

[Royale37]

22 minutes ago, PDCWolf said:

That parenthesis, that's what I believe should've been part of the first statement in this post: it's not that you don't know, It's not that the tech guy magically worked the issue and disappeared, but rather something happened and you can't reveal it or it can't be communicated to you. The "little outrage" was meant to point at what happened in this thread, not at the handling of the forum failure, I really have no comment of that because there's no way to gauge how it was handled, so I'm just hoping you all did your bests.

Shouldn't have taken 2 pages of asking to get that parenthesis (which you never closed by the way), that's my criticism.

Look; 

They don't know what happened, as they've already said multiple times.  We should be thankful that we have mods and a forum, considering what happened to KSP2. I understand you guys are frustrated that we don't know what happened, because I am too, but we need to stop laying the blame on the mods.

[Fizzlebop Smith]

The death of any relationship is resentment. It really doesn't matter which party fosters the resentment first.

One of the quickest ways for resentment to take root is when Appreciation is replaced with Expectation.

This is often most significantly felt regarding the time we (freely) gove another. Time is the one commodity unlike others... if i am running low on pork bellies, inmerely need to buy more.

add *unrealistic* on top of that expectation which really should be appreciation.. and the death throes are nie.

I work too many hours for too little pay. I do not have much *extra* time.

There are a couple things I do not mind doing...

I leave the house an hour early for D&D twice a month so that I can pick up a player without a vehicle. This is 4 hours per month I do without much thought...

Until I started to have to wait increasingly longer and longer for person to be ready... they stopped saying thank you for the ride.

I appreciate the effort made to reach out on reddit. I think a couple days of waiting while emails were exchanged is not unrealistic, and was super happy to see some word from Vanamode within short order.

 

Thank you (all the mods still chugging along) for all the hundreds of hours you have freely giving.

 

Edited October 31, 2024 by Fizzlebop Smith

[Royale37]

14 minutes ago, Fizzlebop Smith said:

Thank you (all the mods still chugging along) for all the hundreds of hours you have freely giving.

Agreed 

[PDCWolf]

1 hour ago, Royale37 said:

Look; 

They don't know what happened, as they've already said multiple times.  We should be thankful that we have mods and a forum, considering what happened to KSP2. I understand you guys are frustrated that we don't know what happened, because I am too, but we need to stop laying the blame on the mods.

That's fine, and if you read my messages, it was never my problem. I think the reason why they don't know  was paramount to the announcement, and support being under NDA to tell the moderators of the forum why their forum isn't working is pretty vital information for the users to know, specially since it's just more to their recurring argument that they're being left in the dark as much as we.

If anyone should be thankful here is T2, for there people out there that are still bothering to check this place, and also others who volunteer to keep it moderated.

At a personal level, I think communication was of course lacking that one crucial detail (and I'll definitely draw my own conclusions from that they now revealed), and the subreddit should really be the go-to place once the forum fails. I stated so, and it got solved already so I don't see why I keep being quoted about it, but hey, at least I'll take the chance to clarify that I'm not -blaming- the mods as you seem to point at me for doing, I'm asking them to be as clear and transparent as possible. They should know much better than being secretive is not the way, as bad communication is pretty much a big part of what doomed KSP2.

[glibbo]

Hello everybody, I've missed you....

[Gargamel]

3 hours ago, PDCWolf said:

(which you never closed by the way)

Fixed

[StrandedonEarth]

16 hours ago, Bej Kerman said:

Imgur? There's a community there?

Of a sorts, through people’s posts and the resulting commentary. Some imgurians post fascinating stuff

[kerbiloid]

Google.

It helped me to ask "kerbal forums down" and be led to the webpages with the forums state discussion, including the subreddit with more details.

[Kaa253]

I am so glad the forum is back. 
Sincerely, thank you to all and everyone who got it sorted.

[TomKerbal]

I am very concerned about the forum. There are constant problems, now 2 weeks of total failure !!! There are many years of valuable contributions and information here that must not be lost. What happens if T2 pulls the plug? Is there a contingency plan ? Guys, this is important!

[The Aziz]

Whatever the fix did, it didn't help with neverending 502 errors.

[para 9]

On 10/30/2024 at 6:22 AM, Lisias said:

 

"Scared people do crappy things. Really scared people do really crappy things..."

 

I guess that explains why some people were spreading misspelled misinformation like wildfire on Reddit. 

Edited November 2, 2024 by para 9

[The Laythean]

I was just coming to terms with the possibility that the forums may be gone forever. I know we'll all be mourning this forum's death one day, but I'm just glad it's not this day. Everyone enjoy the forums while you can. It's possible the next outage could be the last.

[Timbo]

14 hours ago, The Laythean said:

I was just coming to terms with the possibility that the forums may be gone forever. I know we'll all be mourning this forum's death one day, but I'm just glad it's not this day. Everyone enjoy the forums while you can. It's possible the next outage could be the last.

Yes. This outage, in the light of the abandonment of KSP2, shows there is no commitment at all to maintaining it indefinitely. Action should be taken immediately to preserve it in perpetuity.

[Bobbejans]

how many gigabytes/terabytes does all of the forum use? if you can say that

(clarification: by you i mean the mods of the forum)

[Gargamel]

As for efforts to backup the forums, there are threads in the Network sub related to this.   Discussion on that topic should be continued there for continuity’s sake.  

[Timbo]

There is a Forum Preservation Project happening on Reddit

https://www.reddit.com/r/KerbalSpaceProgram/comments/1g97amg/ksps_forum_preservation_project_was_update/

 

[Kimera Industries]

I aced my math test, KSA is looking like the most promising spaceflight sandbox game yet, and the KSP forums are back. Things are looking up  

[RedDwarfIV]

Oh! I'm afraid the servers will be quite operational when your friends arrive.

Edited November 12, 2024 by RedDwarfIV

[Codraroll]

On 11/12/2024 at 11:39 PM, RedDwarfIV said:

Oh! I'm afraid the servers will be quite operational when your friends arrive.

Building onto the most recent post in this thread, which seems to be the closest thing I've found to a "Forum technical issues" thread (EDIT: I just found the Network subforum. Silly me):

The servers don't quite seem to be operational. For the past couple of months, the forum has had whiteouts, 502 errors, Cloudflare errors, and downtime out the wazoo. It seems I cannot browse for five minutes without some major hiccup. The forums went down completely during the latest Starship test flight. Everything seems to be slow and prone to breaking down all the freaking time. Sure, it's usually back up and working within a few minutes, but after the last incident when the forums were out for weeks, it's hard to shake the feeling that they suddenly won't be back up anymore.

What the heck is happening? Is the new server host massively lacking in capacity, or is it located somewhere with terrible bandwidth or frequent electrical brown-outs? Why is there service at all, but with such inconcistency?

EDIT: Even as I pressed the button to submit this post, the forums threw a "bad gateway" error at me. What is going on?

Edited December 7, 2024 by Codraroll

[Vanamonde]

We don't know the specifics but have informed our contact and we're told they're working on it. 

[Lisias]

6 hours ago, Codraroll said:

What the heck is happening? Is the new server host massively lacking in capacity, or is it located somewhere with terrible bandwidth or frequent electrical brown-outs? Why is there service at all, but with such inconcistency?

I have a theory. As you are probably aware, I'm monitoring the Forum's health (as a side effect of another activity currently on the works). And this is the report for the last week (Zulu time):

The gap is me reworking my infra and not monitoring. You see, we have http 502, 503, 525... This is essentially a showroom for http errors, a car dealer would not do better - and it's not impossible that part of these came from my own infra: I'm monitoring the stack as a whole, after all, and not individual hops.

However... This week I realized that counting the failures is just part of the job, I was failing to monitor the successes! How are the http 200 responses behaving? Well... not that good, this is the a sample of the worst response times of the requests that worked - on a minutely basis:

I had a request taking 320 seconds to be fulfilled! But, interestingly enough, it was an anomaly - the average is between 0.5 to 2 seconds, and the worst cases on each minute are around 40 seconds. Again, this chart samples the response time by minute, it's not an average.

This doesn't appears to be related to the Forum's services (as database) as I was thinking initially. As a matter of fact, apparently there's no directly correlation between these worst response times and the HTTP events from the previous graphic, suggesting Forum may be facing more than one problem at the same time that are piling up on my reports.

So I decided to upload a monitor (just a bash script doing curl into a tee in a loop ) on a server I have access on another continent and monitored both at the same time on the ssh console side by side - something that, frankly, I should had done before, because I noticed that the borkage is not homogeneous over the landscape! While one monitor was getting http errors, the other was OK and vice versa (or somewhat rarely, both OK or both borking at the same time). The borkage runs on batchs, and "moves" across different networks kinda of sequentially.

This may be a possible explanation due some people being, apparently, more affected than others.

Since Forum is under Cloudflare, this means that different CF subnets are involved, and it's almost sure that each subnet is independent from each other, with their owns caches and timeouts.

So, and now I'm guessing: what would induce CF to bork on some subnets and not on others at the same time?

These are my current working theories:

1. There're multiple servers serving Forum running on different machines with different capabilities, and all of them are overloaded. More powerful rigs take a bit more to bork than weaker ones, leading to a chaotic borkage pattern. If CF tries to leverage QoS by switching between them, we could have found a possible explanation for the behaviour I described.
2. There're multiple containers (VMs, LXC, Docker, Kubernetes, whatever) serving Forum on a round robin scheduling over multiple endpoints, each endpoint kinda of dedicated to a CF subnet. Some configuration error, or perhaps resource exhaustion or even a bug on the Container's code is causing random but recurrent crashes over the pool of containers, affecting different CF subnets as the crashes happens and the Containers are restarting the services.
3. There're multiple containers being fired up and spun down under demand, and something (perhaps a configuration error?) is inducing the Container to do that continuously: a container is fired up, do some work but then it's killed and another one replaces it - and them the stack takes time to notice that the endpoint it was talking to was killed and that it needs to find another one.
4. All of the them. None of that are mutually exclusive.

Again, these are guesses. They appear to fit the behaviour I describe above, but they are still guesses.

 

 

 

 

Edited December 8, 2024 by Lisias
Tyops, as usulla...

[Codraroll]

10 hours ago, Vanamonde said:

We don't know the specifics but have informed our contact and we're told they're working on it. 

Thanks. As @Lisias said, it has been an issue for months now, but recently it has gone from "bad at times" to "borderline unworkable". I encountered three whiteouts and spent ten minutes just on my way from opening the forum, seeing the "your message was quoted" notification, and opening this thread.  Every step of the process has taken several minutes. And this is not the first nor only time the forums have been like that recently. Every other time I try to open them, some error message or whiteout pops up instead. The rest of the time, it works, but slowly and service is frequently lost. It's a sad sight.

[Scarecrow71]

11 hours ago, Vanamonde said:

We don't know the specifics but have informed our contact and we're told they're working on it. 

Nothing against you Vanamonde, and we know that the lot of you admins/mods here on the site don't have any control over what is happening.  But I don't think your contact or their team give a damn about the site or the problems.  It took them, if memory serves, more than a week to fix it when it went down completely in spite of all of us giving the actual error number and links to fixes we found.  And this issue with the gateway errors has been going on for months. MONTHS.  They either don't care about the site, or they aren't qualified to be trying to fix it.

Again, this isn't against you.  It's just hard to believe that they are actually trying to fix it when the reality is that they simply haven't.

[CatastrophicFailure]

16 hours ago, Scarecrow71 said:

Nothing against you Vanamonde, and we know that the lot of you admins/mods here on the site don't have any control over what is happening.  But I don't think your contact or their team give a damn about the site or the problems.  It took them, if memory serves, more than a week to fix it when it went down completely in spite of all of us giving the actual error number and links to fixes we found.  And this issue with the gateway errors has been going on for months. MONTHS.  They either don't care about the site, or they aren't qualified to be trying to fix it.

Again, this isn't against you.  It's just hard to believe that they are actually trying to fix it when the reality is that they simply haven't.

As the parent company is effectively defunct, said contact is probably a now-former employee doing this in their spare time and quite likely with their spare budget as well. I think perhaps the forum has devolved to some rough equivalent of highly-limited low-propriety “free trial.” So basically, we are lucky to have this forum at all and it may or may not wink out of existence at literally any moment not unlike the universe itself. 

Then let us eat, Kerbal, and be merry, for tomorrow we may diet. 
 

502 error incoming in 3… 2…