kfsone

Thanks for the link! "Learning project" wouldn't be entirely fair, fence-over-peeking perhaps? I'm really not a UI person, more of a back-end/cli (https://bitbucket.org/kfsone/tradedangerous was an actual learning exercise, and at the end what I learned was I shouldn't have done something in python if I thought I was going to use parallelism )

Thanks -- if it turns out there isn't another of these out there already, I'll figure out how to build an installer for it.

While I'm sure better already exists, I felt like writing code in a new language (to me), so I knocked up a regedit-style SFS (save game) viewer/editor to help me find & fix a problem with one of my saves. License: MIT Release Release v0.6.0 · kfsone/sfsed (github.com) I've not done any optimization work or serious polish. I had ported it to .net core but switching from WinForms to a cross-platform ui was leaving my immediate sphere of need.

It seems to be Mobile Drill Base PART #49 and PART #58 specifically where the problem occurs: I'd tried spacing them out but all that did was make a visual mess: So I changed the temperature of the gangway (part #58) to 5500, which was enough to destroy it and separate the vessels finally.

KSP: 1.9.1 Windows 32bit Problem: Can't undock rocket from mining base Mods installed: [x] Science! Continued (xScienceContinued 5.26) All Aboard! (AllAboard 1.0.1.1) All Y'All (AllYAll 1:0.11.18.3) AlphaMensae's Modular Launch Pads (ModularLaunchPads 2.0.8) Astrogator (Astrogator v0.10.0) AT Utils (AT-Utils v1.9.0) B9 Part Switch (B9PartSwitch v2.16.0) Better Science Labs Continued (BetterScienceLabsContinued 0.2.0) ClickThrough Blocker (ClickThroughBlocker 0.1.9.5) CommNet Antennas Extension (CommNetAntennasExtension 2.1.1) CommNet Antennas Info (CommNetAntennasInfo 2.4.0) Community Category Kit (CommunityCategoryKit 5.0.0.0) Community Parts Titles (CommunityPartsTitles 0.6.0) Community Parts Titles Extras: Categories (CommunityPartsTitlesExtrasCategory 0.6.0) Community Parts Titles Extras: CCK - No Duplicates (CommunityPartsTitlesExtrasNoCCKDup 0.6.0) Community Resource Pack (CommunityResourcePack 1.3.0.0) Community Tech Tree (CommunityTechTree 1:3.4.1) Configurable Containers Core (ConfigurableContainers-Core 2.4.8) Contract Configurator (ContractConfigurator 1.28.0) Contract Pack: Bases and Stations Reborn (ContractConfigurator-KerbinSpaceStation 2:3.7.2.2) Contract Pack: Exploration Plus (ContractConfigurator-ExplorationPlus 1.0.2) Contract Pack: Kerbal Academy (ContractConfigurator-KerbalAcademy 1.1.10) Contract Pack: Tourism Plus (ContractConfigurator-Tourism 1.5.2) Craft Manager (CraftManager 1.2.0) Cryo Tanks (CryoTanks 1.4.2) Cryo Tanks Core (CryoTanks-Core 1.4.2) Deployable Engines Plugin (DeployableEngines 1.2.1) Distant Object Enhancement Continued (DistantObject v2.0.0.2) Distant Object Enhancement Continued default config (DistantObject-default v2.0.0.2) DMagic Orbital Science (DMagicOrbitalScience 1.4.3) Docking Port Descriptions (DockingPortDescriptions 1.0.1.1) Dr. Jet's Chop Shop (ChopShop 1:0.11.4.5) Duna Direct (DunaDirect 1.8.1.1.1) Dynamic Battery Storage (DynamicBatteryStorage 2:2.1.5.0) EVA Enhancements Continued (EVAEnhancementsContinued 0.1.15.2) EVA Handrails Continued (EVAHandrailsPackContinued 0.3.0.3) Experiment Tracker (ExperimentTracker 1:v1.3.5) Firespitter Core (FirespitterCore v7.15) Global Construction (GroundConstruction 2.6.0) Global Construction Core (GroundConstruction-Core 2.6.0) Hangar (Hangar 3.5.0) HeapPadder (HeapPadder 0.0.2) Impact! (Impact v1.8.0) Internal RCS (InternalRCS 1.2) KEI (KEI 1.2.10.2) Kerbal Alarm Clock (KerbalAlarmClock v3.12.0.0) Kerbal Atomics (KerbalAtomics 1:1.1.2) Kerbal Atomics - NFE Integration (KerbalAtomics-NFECompatibility 1.1.2) Kerbal Atomics - Other Mod Support (KerbalAtomics-NTRModSupport 1.1.2) Kerbal Attachment System (KAS 1.5) Kerbal Dust Experiment (KDEX v1.11.3) Kerbal Engineer Redux (KerbalEngineerRedux 1.1.7.1) Kerbal Improved Save System (KerbalImprovedSaveSystem v2.4.1) Kerbal Inventory System (KIS 1.24) Kerbal Inventory System - No Fun (KerbalInventorySystemNoFun v1.4.2) Kerbal Planetary Base Systems (KerbalPlanetaryBaseSystems v1.6.11) Konstruction (Konstruction 1.3.0.0) KRASH - Kerbal Ramification Artifical Simulation Hub (simulation mod for KSP) (KRASH 0.5.32) KXAPI (KXAPI 1.2.0) Malemute Rover (MalemuteRover 1.3.0.0) MechJeb 2 (MechJeb2 2.9.2.0) Mk2 Stockalike Expansion (Mk2Expansion 2:1.8.7) Mkerb Inc. Science Instruments (MkerbIncScienceInstruments 1.1) Module Manager (ModuleManager 4.1.3) MSP-3000 Material Science Pod (MSP3000 v1.1) Near Future Construction (NearFutureConstruction 1.2.2) Near Future Electrical (NearFutureElectrical 1.1.1) Near Future Electrical - Decaying RTGs (NearFutureElectrical-DecayingRTGs 1.1.1) Near Future Electrical Core (NearFutureElectrical-Core 1.1.1) Near Future IVA Props (NearFutureProps 1:0.6.2) Near Future Launch Vehicles (NearFutureLaunchVehicles 1.3.0) Near Future Propulsion (NearFuturePropulsion 1.2.1) Near Future Propulsion - Xenon Hall Effect Thrusters (NearFuturePropulsion-XenonHETs 1.2.1) Near Future Solar (NearFutureSolar 1.2.1) Near Future Solar Core (NearFutureSolar-Core 1.2.1) Near Future Spacecraft (NearFutureSpacecraft 1.3.1) Orbital Survey Plus (OrbitalSurveyPlus 2.3.6) Patch Manager (PatchManager 0.0.17.1) PicoPort (PicoPort 0.1.6) PicoPort Shielded (PicoPortShielded 1.0.1.1) Procedural Fairings (ProceduralFairings 1:v1.8.1) Procedural Parts (ProceduralParts v2.0.2) RCS Build Aid Continued (RCSBuildAidCont 1:0.10.0) Real Plume (RealPlume 2:v13.2.0) Real Plume - Stock Configs (RealPlume-StockConfigs v3.1.0) Real Solar System Textures - 8192 x 4096 (RSSTextures8192 v18.1) RemoteTech Redev Antennas (RemoteTechRedevAntennas 0.1.1) SCANsat (SCANsat v18.14) ScienceAlert ReAlerted (ScienceAlert 1.9.8.6) Ship Effects Continued (ShipEffectsContinued 1.0.11) Smart Parts (SmartParts 1.9.16) SmokeScreen - Extended FX Plugin (SmokeScreen 2.8.13.0) Solar Science (SolarScience 1:v1.2.0.0) SpacetuxSA (SpacetuxSA 0.3.13.1) StageRecovery (StageRecovery 1.9.2.2) Station Science Continued (StationScienceContinued v2.6.0) Stockalike Station Parts Expansion Redux (StationPartsExpansionRedux 1.3.4) Stork Delivery System [SDS] (StorkDeliverySystem 1:0.0.5.0) TAC Fuel Balancer (TacFuelBalancer v2.21.5.1) Toolbar (Toolbar 1:1.8.0.5) Toolbar Controller (ToolbarController 1:0.1.9.4) TriggerAu Flags (TriggerAu-Flags v2.9.3.0) USI Tools (USITools 1.3.0.0) Zero MiniAVC (ZeroMiniAVC 1:1.1.0.1) Reproduction steps: Load the linked save game, Right click the rocket's docking port facing the mining base, Select undock, --> Fails Right click the docking port to which it is attached (K&K), Change to primary docking node, Select undock, --> Crash You can undock the other ports towards the end of the vehicle, but ultimately you cannot undock the rocket from the K&K docking port? Log: https://www.kfs.org/oliver/ksp/Player.log Save Game: https://www.kfs.org/oliver/ksp/quicksave.sfs https://www.kfs.org/oliver/ksp/quicksave.loadmeta The problem seems to be caused by physics settling at warp transitions that have caused the nodes attaching the docking port to the rocket to become physically separated.

Holy shame-on-me, batman. I have Konstruction installed but hadn't noticed the ports. See my issue about finding parts, lol Right - so each extension I add tends to have one waiting on the end for the next unit, but there also seems to be a much bigger hit unpacking a vehicle related to how many connected parts you have. I've also found that my game gets progressively slower the more multiple-docked-component vehicles I visit. I can hop vab/launchpad, tracking/probe a lot of times, but hop to a 3+ docked units vehicle and every subsequent transition is significantly slower (multiple seconds per hop) And eventually I get a stack mark crash. I tried running a totally clean install on a different OS partition without mods, and the same proved true for me.

Couple of my biggest problems with KSP could probably be addressed with a couple of simple tweaks: - Is there a mod that makes finding parts ... sensible? Better filtering (only show me lf/ox engines; don't see a way in advanced to do that) and inline stats (I want to see dv, twr, isp, all at the same time or at the very least on the immediate mouse-over for easier comparison). Or an option to sort by dv, twr, isp and/or more than one at a time (that feels like too lazy for me, I'd be good enough looking thru a refined list rather than basically having it tell me "use that engine") - Is there a mod that lets you eliminate docking ports during on-orbit assembly somehow? I find myself caught between two strategies: A reasonable, fully-modular launch strategy: One launch for guidance, one launch for power, one launch for fuel tanks, one or more 'payload' launches (e.g one for isru, another for cooling). The catch is each sub-unit frequently requires ~2 docking ports, and docked parts seem to have the absolute worst impact on unpacking and physics performance, plus you're pushing redundant mass around. The other is stupidly-overbuild: It's so easy to think "I'll just add this part" to a payload to try and avoid another launch/pair of ports, and for some reason going from a pair of fl-t800s to achieving 3000dv to a single fl-txi00 seems to cross you over some kind of dv valley of doom. If you don't catch that, now you've gone thru all the intermediates and replaced the fl-txi00 with an s3-144 and a mainsail. You're down to 1200dv and have no reaction wheels, adding mp-rcs drops your dv to 800; adding lf/ox-rcs doesn't drop your dv from the vab but it does in-flight. So you take that "one more part" off the top stage and you can get the whole rig to your minmus station *and* recover the first stage using just 1.25m parts? I get part of that is the real difficulty of launching to space, but the VAB just converts it from challenging to miserable.

Did I misunderstand that the decoupler is at the top of the tank? When I marked the top-most tank as having a decoupler, I was expecting the behavior on the left, but I was seeing the behavior on the right.

@linuxgurugamer Minor but annoying, StageRecovery seems to get confused by this when visualizing safe stages and possibly during actual determination of recoverability. I'd guess that SR has a short-circuit/early-out for decoupler nodes, but if you go into VAB, make a vehicle of two small tanks, put two chutes on the top one, make the top one a decoupler, and then click the 'SR' button to color-code recoverability ... the top tank will be green and the lower one red. Replication for me: . Add tank, . Set Decoupler, . Left click on tank to pick it up, . Place tank again -> setting lost, . Add tank, . Set Decoupler, . Alt-Left click on tank to pick up a copy, . Place copy, -> setting lost, . Add tank, . Set Decoupler, . Launch, . Revert to vehicle hangar, -> setting lost I've not exhaustively tested for this, but they've been consistent enough to either be deterministic or an uninitialized value/undefined behavior

It caught me by surprise early on, when I moved from BCPL to C, took me weeks to figure out the bug the first time. SAS C wasn't terribly generous with errors/warnings but I was also writing a compiler generator so I had several layers to try and figure out.

Only C-like languages, the majority actually use '=' in both cases or give them distinctive behaviors (void operator=, bool operator==) Lua 5.2.4 Copyright (C) 1994-2015 Lua.org, PUC-Rio > x = 1 > y = 2 > if x = y then print("hello", x) end stdin:1: 'then' expected near '=' > if x == y then print("hello", x) end > Python 2.7.17 (default, Nov 7 2019, 10:07:09) [GCC 9.2.1 20191008] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> x, y = 0, 2 >>> if x = y: print("Hello", x) File "<stdin>", line 1 if x = y: print("Hello", x) ^ SyntaxError: invalid syntax oliver@dogfish:~$ perl x = 0; y = 2; if (x = y) { print("Hello", x) } Can't modify constant item in scalar assignment at - line 1, near "0;" Transliteration replacement not terminated at - line 1. oliver@dogfish:~$ rustc test.rs -o test.exe error[E0308]: mismatched types --> test.rs:4:8 | 4 | if x = y { | ^^^^^ | | | expected bool, found () | help: try comparing for equality: `x == y` | = note: expected type `bool` found type `()` error: aborting due to previous error For more information about this error, try `rustc --explain E0308`. oliver@dogfish:~$ go run test.go # command-line-arguments ./test.go:7:14: syntax error: assignment x = y used as value But head C-ward oliver@dogfish:~$ node > x = 0; y = 2; 2 > if (x = y) { console.writeln("Hello", x); } TypeError: console.writeln is not a function > if (x = y) { console.write("Hello", x); } TypeError: console.write is not a function > if (x = y) { console.log("Hello", x); } Hello 2 undefined But even C-derivatives have moved away from this behavior because it's bad: oliver@dogfish:~$ tail +9 test.cs if (x = y) { Console.WriteLine("Hello"); } } } oliver@dogfish:~$ mono-csc test.cs test.cs(9,7): error CS0029: Cannot implicitly convert type `int' to `bool' Compilation failed: 1 error(s), 0 warnings That depends on what planets.Eeloo is. It could be an enum, it could be a constant of value 0. If Eeloo *is* an object, you could also easily determine whether you really ever want people assigning planets to each other and simply disable it, require an rvalue, etc.

Only in more C-like languages, the majority of languages disallow assignment in a condition like that, often because they hated this when they were writing their language in C In C++ `if (planets.Eeloo.operator==(planet))` could do something very different than `if (eeloo.operator==(planets.Eeloo))` for a host of reasons, especially since == is an operator and not guaranteed to be commutative. There's a lot better ways than reverting to yoda-conditionals. I make judicious use of const, nullptr, enums and constexpr, and I generally hold myself to -Wall -Wextra -Wpedantic. C's definition of the if, while, etc conditionals is in terms of `expression`, and `expression` is any statement that resolve to a value (I'm going to avoid C++'s rvalue mayhem and stick to C). Thus `int x = 0, y = 2; if (x = y) { printf("Hello, %d\n", x); }` will print Hello, 2 (https://ideone.com/Lr3qR0) Why? Because the '=' operator evaluates to the value assigned, so you can do `x = y = z = a = b = c` (b = c; a = (b = c); z = (a = (b = c)); ...` which sets the value of c to all the variables. So `if (x = y)` assigns y to x, then evaluates the value it assigned, which was non-zero, so true. [edit: using -Wpedantic: http://coliru.stacked-crooked.com/a/47e2e78a3ca22333]

English is my first language, but American is my second, and I love creating a file called "localization.h" and having it open a namespace called "localisation" and then spelling it with a 'z' in enums and an 's' in functions/classes Gotta have a pattern, or you're trolling yourself as well as your co-workers!

Sure - golang turned 130,000 lines+ of program code into an executable program in just over 1/8th of a second. Because the C++ language doesn't give a hoot how difficult it is to write a parser for, it's obnoxiously slow. The cutting edge compilers for C++ are amazing, but given a file with one line of code and 130,000 empty lines, they take as long to do the first half of their job as golang took to read all the code, deal with all the external stuff, and spit out a runnable program. In both cases, each line of code probably invokes tens or hundreds of functions. So, I demonstrated how long it'd take *just* to call that many functions. Calling a function is one of the most expensive operations in Python, so unless you can compile your 130k line program with a single function, Python wouldn't be able to do it. If you designed your language with that in mind, you could probably just about pull it off. Python itself is a program that parses python source codes and then does Python stuff. But that program is written in something else, usually C. If we time how long Python takes to load an 130,000 line program, we can see how efficient the C-based python-parser is at parsing python: [G:\tmp] > python -c "open('130k.py', 'w').write('a = False if True else False\n' * 130000)" [G:\tmp] > measure-command { python .\130k.py } | select TotalMilliseconds TotalMilliseconds ----------------- 1166.6474 (To be fair: this was a typical result, one of the 100 times it only took 144ms) I've given Python an advantage here, too: the Go program was 130,000 lines of non-trivial code plus all the external stuff it references. Here, Python just had to process the same line 130,000 times over, huge advantage on modern hardware. And still ~10x slower than the go parser at parsing go. The lion's share of that speed comes from the design of the language heavily factoring the cost of those decisions on parsing the language on modern hardware. I'm expecting people to take certain things for granted here, so I acknowledge that I'm not providing end-to-end concrete proof that all other possibilities aren't in play. If you're actually interested, there's plenty of good stuff on language design and parsing out there Was this what you were asking or were you referring back earlier to my example of XML vs CFG vs JSON? It's simply a fact that when most people parse XML they are using one of the core xml libraries - sax, libxml2, tinyxml, etree, ... The only good way to parse XML on modern hardware is to use a trivial schema and generate a schema-specific FSM statically or at runtime. At which point, you might better use something easier to parse in the first place. As a result, XML parsing has only gotten slower (relative to overall perf) over the decades, and I don't think I've seen any major developments in XML parsing since xml2 or etree. It's not the worst. I really enjoyed embedding it into one project, I wrote a couple of fun WoW mods, but some stuff that seems like it should be easy takes a lot more effort; which is fair enough, it was originally intended as a config language with some programmability. I've found designers and artists can manage their way around it reasonably efficiently. I think the preference for words over symbols for flow (`if .. then ... end`, vs `if (...) { ... }`) makes it seem less alien to them. You can expose all the innards of your application automatically through most scripting languages, Lua included, but that's generally a bad idea because that requires a hell of a lot more exposure to internal documentation. So instead - as with any language - you would probably want to present a relatively clean and lightweight set of interfaces. So, for example, you might have a "planet" module that exposes planet stuff, and encourage people not to write "if planet == 'Eeloo'" but "if planet == planets.Eeloo". That avoids localization issues, etc. Take a squint at the WoW Lua documentation. Because we have C# for the actual mods in KSP, I don't know that an all-out Lua implementation would be the right thing; you probably still want people to write their mods in C#/C/C++ if possible, its just the configurations that you want to give flexibility to, and giving them the ability to be programmatic would be a huge boon.

No, but most people use the same backend libraries and when they don't the obscure one they pull out of the internet's nether or write themselves has a nice api and parses sample documents beautifully, and performs atrociously in real-world scenarios. Parsing a balanced, arbitrarily tokenized tree out of a text document is just very costly on modern architecture. Cf CMake: Used to require balancing conditionals `if (thing) ... endif (thing)` but relaxed to optional when shown how much slower it was than `if (thing) ... endif()`. And someone pointed out earlier, you really need some kind of tool to work with it rationally. What's the point of a text format that requires a GUI tool? If you're going to require a manipulation tool, store the data in an application/exchange beneficial format. I used to (94-04) swear by an XML-based server-side scripting language, RXML; built this site using RXML and XML/XSLT. I get the beauty of XML in the right context, but there are good reasons we are using CSS and not XSLT in today's web, despite the fact that for a long time you could do so much more with XSLT/XPath in browser than CSS, and because it was XML-based, it was vastly less buggy. Bandwidth and performance being two of them. When the Golang team designed their language, parsing speed was a big factor. You can *compile* a 13,000 line golang application to an executable in the same time it takes gcc or clang to parse a 13,000 line file that contains "#include <cstdint.h>" + "\n"*13000, or the same time it takes msvc's cl.exe to start, or the same time it takes python on the same machine to invoke the same, empty function, 1,300,000 times -- i.e. milliseconds (it's not that python is slow, it's that calling a function in cpython has a massive overhead) [C:\Users\oliver\go\src\github.com\stretchr\testify] > dir -r *.go | cat | measure -line | select -property Lines Lines -------- 16822 [C:\Users\oliver\go\src\github.com\stretchr\testify] > go clean ; Measure-Command { go build . } | select -property TotalMilliseconds TotalMilliseconds ----------------- 124.992

One important factor of parsing is text volume - byte count. XML was designed as - its in the name - a Markup Language, and this manifests in the sheer amount of repetition and bloat present in XML documents. json: {"data": {"hello": {"world"}}} XML: <?xml version="1.0" language="en" encoding="utf-8"> <!DOCTYPE data SYSTEM "data.dtd"> <data> <key name="hello"> <string>world</string> </key> </data> (plus add the dtd definition) Terse XML: <data><key name="hello"><string>world</string></key></data> KSP CFG: data { hello = world } Or: data { hello = world } Lets review: {"data":{"hello": {"world"}}} <data><key name="hello"><string>world</string></key></data> data { hello = world } XML parsing generally tends to be either loose-and-slow or restrictive-and-slightly-less-slow. One of the big overheads of XML processing, especially for modern CPU architectures, is the unpredictable ordering of nodes coupled with the expectation of structure and the need to match tokens to a dynamic stack of them. You don't get this with json because there are no identifiers; in language parsing, you don't care at *parsing* what the identifiers are, it's an identifier or not. But in XML you care about matching </key> vs </data> at the right point. (Obviously, you *could* do it differently, but is generally how the parsers that people use operate, whether they are sax- or dom-like) The height of XMLs popularity coincided with a surge in Java development that emphasized XML, and the performance overhead of XML was generally amortized by assorted factors (jvm implementation/overheads, prevalent design patterns, what people were interoping with) so that things like SOAP didn't seem hideously bloated and obnoxious if you were working from inside Eclipse or JetBrains products. Microsoft picked it up in C# as it also caught on in the C++ world, where it encountered bare-metal performance observations which was its undoing. Either I can parse the document agonizingly slowly and have it presented as an AST, or I can write a cache-antagonistic event model that can deliver me nodes blazingly fast but require me to implement an fsm for almost every document I want to work with, which turns out to be a super incredibly slow way to consume configuration files, message packets, etc when you are measuring performance in cpu cycles not milliseconds. edit: indeed, the beauty of json is that it is [mostly] meaningful, or close to, in several languages, such that you can reuse the language's own parser to interpret it, increasing cache coherency.

EBIDIC Base64 encoded with LHA compression, pgp signed and packaged for SOAP exchange over UUCP with FidoNet headers. Text that has been English->French->German->English translated, maybe. Again, if you're going to go with text, at least make it programmatic like Lua or Python. # Py altitudes = { Planets.Kerbin: (60000, 70000, 60000), Planets.Gilly: (6000, 7000, 6000), Planets.Eve: (85000, 91000, 88000), # ... } RESOURCE = 'ore' for planet, (lowest, highest, best) in altitudes: register_abundance(RESOURCE, lowest, highest, best) -- Lua scan_altitudes = { [ksp.Planets.Kerbin] = {60000, 70000, 60000}, [ksp.Planets.Gilly] = {6000, 7000, 6000}, [ksp.Planets.Eve] = {85000, 91000, 88000}, -- ... } RESOURCE = "ore" for planet, alts in pairs(scan_altitudes) do register_abundance(RESOURCE, table.unpack(alts)) end Then imagine each of the above tables fleshed out for all the resources scansat handles. Right now all of that is unpacked *into* the cfg file itself rather than only exposing to kerbal the parts that are actually important. Python: I have a hate/kick-in-the-balls relationship with Python, but it helps pay my bills. CPython and Jython are slow, Pypy is JITed and faster but lacks the breadth of packages that makes cpython so popular. Lua: Super-lightweight C bindings make it lightning fast, although like any language there are ways you can make it cripplingly slow (why are you creating a table of 5000 elements every iteration of a 10,000 iteration loop?), it's popular among modding communities and easy to embed, but the inventor was Portuguese so as a language it's about what you'd expect... if ... then .. end; while .. do .. end .. etc. <not-seriously> Or we could roll back time and use uLPC/Pike! https://pike.lysator.liu.se/ </not-seriously>

If you're going to go with a textual layout, at the very least use an existing one or go with a config/scripting language like lua or one of the .net equivalents. Alternatively, provide a robust interface to the data. Using something like thrift/protobufs lets you declare the structure so that you can access the representation through widely available tools to work with the data in human form, and you can use human-form during development. Modders/hackers can mess with the data from a variety of languages: node, py, c#, mono, powershell across platforms, javascript, webasm, rust, go, c++, ruby, ... I love me some text files - I've written MUD languages and parsers are a hobby of mine, but - and ksp 1 demonstrates handily - they can hamstring engines by fooling you into dealing with object instantiation in certain limited and non-performant ways. That said, as others have mentioned - there's the middle ground of text ingestion and binary retention with a simple timestamp+size check on the source files.

Agreed, sqlite is heavy handed unless you're actively going to use it as a datastore. If all you want to do is load the data, it's a sub-optimal approach because it's not in memory-ready format. Better, where possible, to try and compile the data into memory-ready binary representation, even if that winds up wasting some storage/etc. If you can just mmap the files into memory and know they have a usable binary layout, the size stops mattering so much (https://github.com/google/flatbuffers, https://github.com/capnproto/capnproto, https://grpc.io/docs/guides/)

Facebook used ZooKeeper to distribute configs fleet-wide(*) and the configs were pretty massive. They used python to allow variable-complexity config generation which they then serialized as json and fed to zk to distribute, proved to be hella-fast and relatively easy to maintain, but it mean't the overhead of two human-readable language formats to contend with. I think you'd be better off with something like thrift or protobufs for ksp: there are great, highly optimized parsers; the human format is relatively readable; versioning included; designed for the wire and in-memory use, which is pretty important because that's the other half of loading: allocating for and populating the structures the data represents. Several of the teams I've interacted with have hard-coded pool allocators with speculative initial sizes that are basically always undersized because they're based on the last art/asset built the programmers have pulled and not the current art/asset dev branch "Yeah, but we don't know how many total meshes or bones there are going to be". Right, because you've got 429k .mesh files to read and you're doing upto 50 allocations per line of text you read. The technical CS term for this is "derp"

I got back into the bad habbit while working on a port, and found with vscode, wsl, docker, etc, it makes a really nice window manager for stuff. "PowerShell Core" - the open source/cross-platform version - had just come out, and I jokingly used that to solve one of the CI issues we were having, and was like "wait, what, this is posix shell with objects? so like a repl for the OS?". s/35 years of ed, sed, awk, grep, ksh, csh, bash, zsh/pwsh/ftw. All my home vms, mac mini, bsd boxes have it as either a second or default shell. $s = new-pssession vm1, vm2, vm3, win1, win2, mac1 invoke-command $s { upgrade-os-packages ; upgrade-pip-packages ; update-go-packages } Mmmm

`stat`ing files is much cheaper than opening them, and if you choose your directory crawl carefully you get that info with your list of files and without any further effort. ext4 and ntfs anticipate and optimize for this pattern. If they want a text format, they should really consider using an existing one optimized for performance, such as json or https://cuelang.org/. If they choose yaml, I'm gonna necrorez this thread and beg everyone to kick me in the head, though. Other options, especially since they're talking multi-player, would be using something like thrift/protobuf/grpc and serializing the message formats. Yes, as a programmer, I love that I can open the files in vim/emacs/vscode/textedit ... And yes, I hate the guy who says "if we did it in formatX we could write an editor", but the other option is to look at scripting choices like Jupyter Notebooks where you can use a textual entry/modification method that translates to a binary representation. Going to something like sqlite would reap the benefits of a single file that can be mmap/createfilemapping'd, definite win there. It can be a little more overhead than you bargain for, tho, and it means you have to start being a dba along with everything else. I've used sqlite for a few things, including https://pypi.org/project/tradedangerous/, but I supplemented it with textual ingestion and transfer formats so users still have access to text files if/when they want them, but general usage isn't bothered by them, and startup times are significantly less painful than they might have been. Alternatively, aim for a more centralized approach and try to encourage collation of files to reduce the overall file count. You can try this for yourself, this is how long it takes *just* to read all the part/craft files, never mind all the texture files etc. No parsing, just reading the files, and this is immediately after I loaded the game on a 64GB machine with a high-end plextor 2tb nvme: C:/KSPPath/> $files = dir -r *.cfg,*.part,*.craft C:/KSPPath/> $files | measure Count : 3461 ... C:/KSPPath/> measure-command { get-content $files >$null } Days : 0 Hours : 0 Minutes : 0 Seconds : 42 Milliseconds : 751 Ticks : 427514498 TotalDays : 0.000494808446759259 TotalHours : 0.0118754027222222 TotalMinutes : 0.712524163333333 TotalSeconds : 42.7514498 TotalMilliseconds : 42751.4498 You'll have to take my word that these are representative samples rather than one offs But... [G:\Steam\steamapps\common\Kerbal Space Program\GameData]> wsl oliver@Spud:/mnt/g/Steam/steamapps/common/Kerbal Space Program/GameData$ find . -name \*.cfg -print0 | xargs -0 perl -i -pe 's/^[ \t]+//; s/[ \t]+([\r\n]+)/$1/; s/\s+\/\/.*?([\r\n]+)/$1/' oliver@Spud:/mnt/g/Steam/steamapps/common/Kerbal Space Program/GameData$ exit logout [G:\Steam\steamapps\common\Kerbal Space Program\GameData] > measure-command { get-content $files >$null } Get-Content: An object at the specified path G:\Steam\steamapps\common\Kerbal Space Program\GameData\[x] Science!\PluginData\[x] Science!\settings.cfg does not exist, or has been filtered by the -Include or -Exclude parameter. Get-Content: An object at the specified path G:\Steam\steamapps\common\Kerbal Space Program\GameData\[x] Science!\PluginData\[x] Science!\settings.cfg does not exist, or has been filtered by the -Include or -Exclude parameter. Days : 0 Hours : 0 Minutes : 0 Seconds : 18 Milliseconds : 806 Ticks : 188060925 TotalDays : 0.000217663107638889 TotalHours : 0.00522391458333333 TotalMinutes : 0.313434875 TotalSeconds : 18.8060925 TotalMilliseconds : 18806.0925 So - 20 seconds, big whoop. I tripple dare the KSP team to add metrics to their build and launch system so that they track every dev-build startup and see just how much they are paying their staff to watch loading screens Open a simple tcp connection to something like a flask/sinatra server, and log the connection time, write the build info and pid over the socket, and leave the socket open. Any termination of the process will close the socket; if the machine hard-locks, the tcp session will get closed soon after. Otherwise, you can have the client send breadcrumbs during execution and shutdown that you can use to monitor general health of your dev builds. At my last game gig, their asset pipeline spent 15 minutes processing text files, which I whipped into 50 seconds with a minor change to their language spec and some changes to the parser. I actually got it down to ~800ms with a quick parser written in C++ but the tooling was all in Python so it wasn't worth the codebase overhead. -Oliver

I recently had to try and help a team of devs understand why our Windows port of an app went from a 15s startup on Mac and Mobile to 2-5 minutes: The current Windows OS is based on Windows NT which is in turn based on the VMS operating system. So it starts with a fantastic permissions system, but it comes with a perf cost. Then, over the years, it's acquired various things in the kernel that Linux, for example, did in user space. The upshot is that just opening files on Windows is anywhere upto 80 times slower than opening a file under another OS. This isn't "Windows is slow" or "Windows IO is slow", it's just opening files. Open a few large files and you'll not notice it. Open a lot of small files, and you'll add seconds or minutes. The project I was helping port had 429,000+ assets it was accessing on startup. If you want to see this for yourself: https://github.com/kfsone/filebench (uses C++17) The other significant factor is the amount of whitespace in the KSP1 files. I have 76MB of .cfg files in my fairly lightweight Windows folder, and about 16MB of that is [edit: beginning-of-line] whitespace... On startup, you spend quite a lot of time skipping over this. I was able to shave a chunk of my startup time by running this under wsl on my ksp .cfg files: # s://.*?([\r\n]+):$1: - removes comments but leaves end-of-line as is # s:^[ \t]+:: - removes leading whitespaces # s:[ \t]+([\r\n]+):$1: - removes trailing whitespaces but leaves end-of-line as is find . -name \*.cfg -print0 | xargs -0 perl -i -pe 's://.*?([\r\n]+):$1:; s:^[ \t]+::; s:[ \t]+([\r\n]+):$1:' Fair enough - the file format has been a great boon to the modding community, I get that, and I get how handy it is as a dev not to have to put files thru the asset pipeline to see a change. Could you maybe at least convert the outputs to a binary representation and cache those in N large files? Allocate a bunch of padding between source files so that relatively small changes don't require rebuilding the entire file. The other thing you might consider is parallelizing tokenization vs loading. On a fast filesystem, the tokenization will be unlikely to starve the alloc-heavy instantiator, on a slow filesystem it may fall behind but it will still be significantly faster than before.

"Duh", I thought; "Try your other PC", I thought. So I tried installing KSP on my Surface. "Beautiful!" I thought, until I looked more closely, and sure enough the dithering/banding is there too. It's most noticeable when you have adjacent, similar colors, so Kerbin's ocean, or the galactic skybox. "Well", my wife said; "Does it run under Linux?", she stunned me with. So the following are screenshots from the same machine but KSP running under an Ubuntu 19.04 install: https://steamcommunity.com/sharedfiles/filedetails/?id=2023830668 https://steamcommunity.com/sharedfiles/filedetails/?id=2023830617 https://steamcommunity.com/sharedfiles/filedetails/?id=2023830693 https://steamcommunity.com/sharedfiles/filedetails/?id=2023830652 That's a fresh 19.04 default install + steam + ksp. No config, no mods, no tweaks, and then I switched to full screen mode with full textures etc for the last one.