Lisias

NEWS FROM THE FRONT Still working on scraping Forum's contents that missed the 202407 release, as well new content. In the process, I'm now eye balling all the logs trying to find out ways to minimize hits on Forum. Found some: At least until last week Forum used to beg for mercy using HTTP 429 Too Many Requests . But scrapy not only ignores it, but just doesn't knows about it, logging "Unknown Status" what ended up preventing me from connecting the dots and missing an excellent opportunity to pause scraping when Forum is being overloaded. I wrote a new middleware to handle it, as well doing the same for any 5xx received. My reports for the 5xx occurrences will be affected, as I will lose the ability to keep a close eye on these occurrences, but I will still log something and, so, a still useful report can be issued - only way less detailed. The same log analysis revealed what's now a pretty evident shortcut on scraping: For each page of a thread, there're something between 1 and 25 requests that returns a HTTP 301 Moved Permanently related to direct links to a specific post on that thread. Since I'm intending to mimic Internet Archive and provide a navigable Archive, I need to scrap these HTTP 301 - but only once. Once I had scraped it one time, this will never, ever change again and so I can add it to a black list to never be scraped again, preventing hitting the Forum Same thing for abbreviated URLs. When addressing a thread, Forum really cares only about the number after the / and before the -. For example, the URL https://forum.kerbalspaceprogram.com/index.php?/forum/53-x/ generates a HTTP 301 to https://forum.kerbalspaceprogram.com/forum/53-general/, and from that point, I don't need to ever hit Forum again using https://forum.kerbalspaceprogram.com/index.php?/forum/53-x/. Rebuilding the ignore-list with the Redirects revealed to be cumbersome, as they are all merged with the Forum's user contents and I had to parse everything - about 90 Gigabytes until 202407. So I wrote a small utility to split the redirections from content on the current dataset, archiving them on separated files. This allows me to quickly build an updated ignore-list before a scraping session. I hope this will help to prevent some hits in the future. Threads that change the title regularly, as Add'Ons' release threads, are currently the focus on my attention - some of these threads are really huge, and re-scraping everything every time the thread changes the Title sounds like a waste for me. But not doing it may cause confusion as the new pages will not properly link into the older pages as scraping continues. I scraped into a dedicate WARC file some legal pages that doesn't belongs to Forum, but affects the terms in which we use it. It will be added to the IA's Torrent this weekend (as soon as I sanitize it and double check the contents to avoid republishing it again) Complementing my "sanitizer" tool, I'm finishing coding another one that will be responsible for splitting the scraped data from a monolithically WARCball into the specialized WARC files - not only for legal reasons, but also from practical ones as the compression hugely improves when we avoid contaminating the highly compressible stream with already somewhat entropic data. Since I ended up writing this tool, I will simplify the current pywb architecture making way easier to setup the rig. I'm going to use that tool on the WARC files anyway, no need to segregated them at scrapingtime anymore - so, why bother? I already reached the limit about what can be done on this (now archaic) architecture, as it forces me to know beforehand the result of the request, what's plain impossible on a HTTP Redirect. Since I was forced to write this splitter tool anyway, there's no reason to not further use it on the whole segregation process taking advantage of the Response already on the WARC file. For the curious people, follows the simplified report of the splitting tool in action: Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-00000.warc Found 245820 redirect records found. 390886 * records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-00001.warc Found 81682 redirect records found. 162248 * records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-202407.warc Found 342338 redirect records found. 236470 * records found. 49 files records found. 5 movies records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-20240811131631900139.warc Found 23 redirect records found. 70613 * records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-20240812082950967614.warc Found 97 redirect records found. 19029 * records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-20240812191456172206.warc Found 50 redirect records found. 6901 * records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-20240813010523547341.warc Found 16470 redirect records found. 24966 * records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-20240813205303128187.warc Found 23478 redirect records found. 29387 * records found. 5 images records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-20240815130028563811.warc Found 48682 redirect records found. 51656 * records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-20240817001954848148.warc Found 30844 redirect records found. 26028 * records found. Processing ../forum.kerbalspaceprogram.com/collections/forum-kerbalspaceprogram-com/archive/forum.kerbalspaceprogram.com-20240817164016690427.warc Found 7202 redirect records found. 7958 * records found. Note: "* records" are anything that is not a redirect, a movie, an image, a file or a style related asset - hopefully, user content. Each scraped item necessarily have two records, the Request and the Response, both are archived. The Redirect/Ratio is significant, sometimes near 1... Well, I'm currently revising (again) the 202407 release and recompressing the data. I will update the torrent on Internet Archives (the repository is being kept up to date as soon the artifacts are working) and advise here. === == = POST EDIT = == === Finally I managed to put my sheets together and properly handled the http 429 thingy. 2024-08-18 15:51:30 [forum.kerbalspaceprogram.com] INFO: Site complained about "502". Retrying <GET https://forum.kerbalspaceprogram.com/topic/160140-122-kolyphemus-system-the-kerbalized-polyhemus/?do=findComment&comment=3107347> in 900 seconds. 2024-08-18 16:11:41 [forum.kerbalspaceprogram.com] INFO: Site complained about "502". Retrying <GET https://forum.kerbalspaceprogram.com/profile/178446-findingclock4/content/?type=core_statuses_status&sortby=status_content&sortdirection=asc> in 900 seconds. <....> 2024-08-18 20:51:21 [forum.kerbalspaceprogram.com] INFO: Site complained about "429". Retrying <GET https://forum.kerbalspaceprogram.com/tags/camera%20focus/?_nodeSelectName=cms_records1_node&_noJs=1> in 60 seconds. 2024-08-18 20:52:31 [forum.kerbalspaceprogram.com] INFO: Site complained about "429". Retrying <GET https://forum.kerbalspaceprogram.com/tags/focus%20bug/?_nodeSelectName=cms_records4_node&_noJs=1> in 60 seconds. 2024-08-18 20:53:32 [forum.kerbalspaceprogram.com] INFO: Site complained about "429". Retrying <GET https://forum.kerbalspaceprogram.com/tags/focus%20bug/?_nodeSelectName=cms_records1_node&_noJs=1> in 60 seconds. I lost the ability to closely monitor the http 5xx responses, so any report will not be minutely accurate as the previous ones - but at least I managed to get myself out of chain of events and ceased to be a possible contributing factor: as soon as the site cries for mercy, now I hear it and leave it alone by the time it tells me to do it (usually 60 seconds). Nasty 5xx errors triggers a 900 seconds (15 minutes) embargo on the spot, no questions asked. Day Job© duties prevented me from finishing the revised 202407 artifacts. I will update this post as soon as I manage to update the IA's torrent. === == = POST² EDIT = == === Torrent on Internet Archive updated. https://archive.org/details/KSP-Forum-Preservation-Project CHANGE LOG: Project: https://github.com/net-lisias-ksp/KSP-Forum-Preservation-Project

Tim Peters.

I found something on eBay that I should, but couldn't let it pass. So I compromised with myself by making a shameful offer that, frankly, I would not accept if I would be in the seller's shoes - this way I would keep my inner child quiet by "trying", but also my adult as I would not had to use money on shipping and taxes and exchange fees on my country. That dude accepted the offer. There're so many different "people" right now screaming on my head, for many different reasons...

Are you meaning Unity's PBR and KSP's implementation for PQS (procedural quadtree sphere)? I originally envisioned an add'ons acronyms database, but something more generic would be useful too - no doubt. What do you think on this thread? https://forum.kerbalspaceprogram.com/topic/225775-useful-acronymns-on-ksp/

I recently noted that there's a need for some technical acronyms used on KSP. This is my attempt to do for general KSP what I did for the Add'Ons. Taking the hint from Grenartia, let's start! PAW (Part Action Window) The window full of part settings that pops up when you right click a part PBR (Physically Based Rendering) A Rendering technique to create physically accurate images. To learn more: https://pbrt.org/ PQS (Procedural Quadtree Sphere) A technique for implementing LOD (Level Of Detail) for complex meshes, saving CPU/GPU juice by using simplifications of he original mesh on parts of the mesh (partitions) when distance would render them unobservable or negligible. To learn more: https://gamedev.stackexchange.com/questions/149762/quad-sphere-subdivision-algorithm Found a nice video about here.

I think you may like to watch this: Me? I would try to reach the Mars colonists pretending I'm a Martian that got lost on Earth and it's now returning home!

And in the case someone missed the memo, Floor Jansen is a hell of a singer! Oblivion, M83 - Cover by Floor Jansen.

Take a 3M super car, speed it to 250 MPH on the KSC - Kennedy Space Center , crash it, totally it and pay less than 720 USD for repairing the facilities. It made me remember the following thread so badly... Found my video on that contest!!

Perhaps we can salvage the discussion arguing that we are hyping KSP2 reborn under OSI and Kickstarter?

I do. I'm a bitter and harsh criticizer of KSP2 not because I don't see any value on it, but because I see the value it could have. Anyway... I completely agree with you. And I say more: by the time the franchise could have any chance of resurrection, the niche will be taken by the competition, that are not going to sit on their hands. Their less worst (because "best" is not applicable anymore to PD) option is to go Open Source and try to gather OSI developers around the idea. They will earn way less that they would like, but they would earn infinitely more than now - anything is infinitely bigger than Zero. I agree with reserves. But, yet, the chances of success on trying a Kickstarter are infinitely better than what we have now. And there are cases of success (hell of game!) on Kickstart! With a ratio way better than PD scored, I must say. There's one good thing on a Kickstarter - whoever will dare to try it, will put their skin in the game (pun not intended) - reputation matters, and people will eventually learn it sooner or later.

I would not bet on this horse, besides not being impossible. I doubt TTWO would sell the Franchise alone - they want to get rid of the whole PD thingy, portfolio included, but assuming they would, they footed something between 60 and 100M USD on it. Elite Dangerous managed to raise 350M USD in 2012, and it got 2.5M pledgers. Correcting it to 2024 money, something like 479.5 M USD. Or about 191,8 USD per pledger. KSP2 have something between 250K and 660K owners, says SteamDB. Assuming all of them would be willing to "buy" KSP2 again (or ever, I heard some speculation about 50% return ration on Steam), it's something between 47.95 M and 126.588M USD (also assuming they would be willing to foot, in average, the same 191.8 USD on it). It's a hell of an assumption, I can say - I really doubt that someone that bought this game for 60USD will be willing to foot another 192USD on this thing. And for the same 60 bucks, the funding would be something between 15M and 39M USD. And I doubt they would foot (again) even that. But... It's not impossible. It's almost impossible, but I can't say it's impossible. IMHO, we would have more chances (anything is bigger than zero, to tell you the true) with KSP¹ Source Code (see my signature). That's the problem - they can write some of that money as loss on their IRS. They are keeping PD alive for a reason, and my bet is Fiscal. The word I heard is that they want to sell the whole Private Division, batteries included. And the whole thing must be valuated to really, really, REALLY more money... "Sonhar não paga imposto." as we say around here... And I agree with you.I don't see a light a the end of the tunnel (that doesn't sounds like PEEEWEEEI) without Open Source or similar.

The CPU is way better than mine, you will not see too much problems on performance. And you have plenty CPU memory too. Your weak point is the GPU. Your Quadro have only 2GB of VRAM (I don't know of any notebook using it with more) and, frankly, nowadays 6GB is the bare minimum to a good KSP experience nowadays. DLSS and FSR will not help you at all - KSP will clog the CPU way faster than the GPU (what KSP really needs is VRAM). You may want to consider using a older KSP version, as 1.9.1 or even 1.7.3. These older KSP versions have a way smaller GPU VRAM footprint. Additionally, I want to quote myself from similar thread:

FYI Report for the http 5xxx errors from last Sunday to today (until 16:20 Zulu). This doesn't reports all the ocurrences, only the ones I was monitoring. I had salvaged all the logs this time, I will further expand the reports with my own hits. I noticed some changes on CloudFlare yesterday and before. It caused some changes on the numbers, but didn't changed the time the main batch of problems starts: more or less at 11:00 GMT-3 (14:00 Zulu). I think some other changes happened again early today or late yesterday, as the last 12 hours of the graph below (from 0:00 to 13:20 GMT-2) shows what appears to be an improvement. Source: https://github.com/net-lisias-ksp/KSP-Forum-Preservation-Project/tree/master/torrent/reports

"Ladies and gentlemen, this is your captain speaking. We have just started our atmospheric reentry and we expect to land at the Olympus Spaceport in about 45 minutes. The weather in Olympus is clear with a temperature of -45 degrees Celsius on this beautiful martian summer. We’ve just been informed that a non-routine inspection will be carried out in our spacecraft by local authorities. We kindly ask you to keep seated with your passports and boarding cards at hand. Please comply and be polite, Martians are short in stature and temper - and somewhat trigger happy with their deatomizer guns"

What model is your notebook? I can play KSP 1.12.5 on a i7 mobile, but with a lot of compromises - but I can play KSP 1.4.3 beautifully on it You may want to try older KSP versions to see if they fits you.

The CPU is decent, but the memory is lacking. But you still can add some mods on it - prefer mods without new textures, or with lower res textures. SXT, KAX, Hooligan Airlabs, even AirplanePlus should work fine (if you like airplanes, of course). As a rule of thumb, try to avoid downloading mods with more than 100 megabytes and you will be able to shove a dozen at least. Since you have a decent CPU, you will go fine with mods what adds features as automatic pilots, etc. You may also want to have a look on Distant Objects.

Strongly suggesting the problem is something both of them use, as a database. My current working theory is the contracted database's hourly quota is burning up pretty quickly. I think someone needs to reconfigure CloudFlare to limit the allowed number of requests by minute. I have word that it's currently 1500 hits per minute, and this appears to be too much. I would try setting it to 500, and if still failing, 250 and later 125. I would not lower this value below it as a lot of people shares an IP due Carrier Graded NAT (or VPNs). From this point, I think some other measure would be needed. @Deddly, @Vanamonde, @Gargamel, there's any report you can access from the Forum Console where the accesses are logged? Of course, you can't share it to me due privacy concerns, but if you can provide me with a sample entry (I can hit forum with my company server's IP with CURL once, and then you send me that exactly log entry) , I can cook a small python or perl script and then you can run the logs on it and detect the offenders. An idea just occurred to me. I detected that wiki.kerbalspaceprogram.com resolves to the same IP for me. I will monitor wiki too, at the same time I will monitor (somehow) the forum. This will help on zeroing into the root cause. === == = ADDENDUM = == == AS 13:13 GMT-3, I'm not experiencing any http 5xx anymore. === == = ADDENDUM² = == == It's 13?37 GMT-3, and everything is screwed again.

And one this (GMT-3) Morning, starting from 08:43 (GMT-3) and still ongoing. I detected a pretty small improvement at 9:00 sharp but it last less then 5 minutes. I noticed another small improvement at 09:59 but it lasted only 2 minutes. So I stopped all my activities at 11:20 GMT-3 fearing I could be responsible for the problem somehow, so I don't have any more information from that point. Unfortunately (as if I would be involved on the mess, there would be something I could do) nothing changed. Given the time stamps in which I noticed that small improvement, I'm guessing one of the triggers for the problem is some quota being exhausted, and my timings above strongly suggests this quota is renewed hourly. I will publish a small report later, but it will not provide us with better information that I already summarize above.

De Havilland DH.89 Dragon Rapide. Weird, but an effective way to keep the CoM low!

I know Boeing is going to excrements, but... really? https://www.businessinsider.com/united-boeing-777-turned-around-after-toilet-overflowed-report-2024-4

I need the KSP.log in order to see what's wrong! Please publish your KSP.log Thx! I added some comments.

I suggest to create a new sandbox savegame, copy the craft file into it and cheat it into Kerbin Low Orbit. Then go back to the VAB, launch a crew capsule (with crew! ), cheat it to rendezvous with the problematic craft and start doing tests from there, preserving your main savegame from mishaps. on this test savegame, you can destroy the craft and cheat new versions of it into orbit unchecked. This will make way easier to diagnose the problem!

You are smashing the engine on the ground. You need more clearance under the craft, ie, reposition the landing gears to make the craft higher. === == = POST EDIT = == === Alternatively, reconfigure the hears from tricycle to dragging tail. Put the main gears in the front, and the steering gear in the back, like the WW1 era airplanes. This way, the gear will protect the engine on landing.... === == = POST² EDIT = == === Never mind. The craft became terribly unstable on taking off with the main gears ahead... === == = POST³ EDIT = == === I managed to "fix' it by moving the main gears way behind, almost near the engine. Taking off became a pain, consider deploying the canards as taking off flaps (set them to "deploy" on an action key, like "0").

Currently, only archive. I remember a discussion about hosting a Forum (part on the early pages of this thread, part on another one IIRC), but my take on it is that footing money on hosting a Forum now would not only be wasteful, but probably deleterious for the Scene. We would not manage to get sponsorship, because Companies do sponsorship to get visibility, and a backup Forum idling while waiting for the main one to die is not a good way to get such visibility. But yet someone would have to foot the bills - and by then, for what? Whoever pay the bills usually are the one calling the shots, and we need to consider this if we ever face the need to replace this Forum: who do you want calling the shots on (at this time) hypothetical Forum replacement? Anyway, a "backup Forum" will not be able to republish this one's content as this would create a "derivative", way beyound the scope of Fair Use, Fair Dealing, or whatever legal or juridical exempt a Country allows for using copyrighted material. It will really have to start from scratch - but it can index the Internet Archive's data and, if by any reason that one falls, a replacement Archive made with the material I'm building.

Now that you mentioned it... I missed an important detail... You are using Canards in your plane, and when the thing gets underwater, it ends up working as an Hydrofoil by accident, and should not allow the nose to go so deep. Can you redo your initial video, but with F-12 hit (showing the Force Vectors)? This will help to see how the lifting surfaces are behaving, probably giving us a hint of the problem...