Where is Nate?

[Skorj]

Well, it's certainly easier to understand how we got what we got if the team were much smaller (or had a lot of do-nothing overhead jobs)!  I don't see why there would have been a re-start, though, unless work product was lost, or the reason ST ended was that T2 really didn't like what they made.  Imagine me ranting for an hour here about ancient dev practices like waiting until all pre-prod work is done before starting any prod work (benefit of being retired is no longer having to care).  But for KSP2 in particular there's a lot the team could start on right away: you don't need rounds of concept art to know what a Kerbal looks like, you know the first coding task is to fix the physics, and so on. 

Come to think of it, didn't Nate did say early on that they started with fixing the "big problem" of KSP1 physics, and completed that to prove the idea viable?  One reason I suspect code was lost with the studio switch is, well, it sure doesn't look like we got a fixed core physics sim. 

[K^2]

4 hours ago, Skorj said:

I don't see why there would have been a re-start, though, unless work product was lost, or the reason ST ended was that T2 really didn't like what they made.

I think ST and Nate personally didn't like what they made, but not for quality reasons.

So this is largely speculation. I got some second hand confirmation for parts of it, but for the most part, I'm going with what we've seen starting with E3 presentation and up until first serious WIP footage started emerging from Intercept, and filling in blanks with my personal experience. Lets start with pre-KSP2 Star Theory. The two games they shipped in 2016 and 2017 were Unity games for PS VR (Wayward Sky and Dino Frontier) and the credits for both games list Uber Entertainment (then name of ST) as 18 people who are directly involved in building the game, including a 1 (one) engineer. 2017 is also when the talks with Take Two would have started. I'm sure ST hired additional people to work on the contract for KSP2, but even most generously, you don't look at that team and say, "Yeah, they're going to build a AAA game from scratch in under 3 years." As far as I can tell, what we know as KSP2 was not what Star Theory was contracted to make.

The biggest guessing game for me is narrowing it down to the rough specs of what ST was meant to deliver in 2020. I think the trailer was meant to represent that game, but potentially with a lot of artistic license. I'm picturing a game that's entirely based on KSP1 codebase, with basically two large DLCs strapped to it. Colony building - including parts and some basic resource management letting you actually build stuff there, and interstellar - one new star system and parts to get you there. Plus some visual improvements for an aging game. I think Star Theory of late 2017 was capable of that, and with a few hires probably on schedule for 2020 release.

Somewhere along the way, the dream got bigger. The E3 trailer was likely created with two goals in mind. 1) Stay within bounds of what's already on the roadmap, but 2) Present it as dialed to 11 under an excuse of the NOT ACTUAL GAMEPLAY disclaimer to gauge the reactions. I don't think it was meant to mislead, but rather represent the game ST wanted to make. And I think the pitch for that larger and bigger KSP2 was already in the air, and pending approval based on public reaction to the trailer. Reaction was good, lights were lit, and Nate is giving us a pitch for a bigger game pretty much from the first interviews. Multiplayer, colony management and transportation network are brought up, as well as plans for more star systems, realistic scale of space in between, physics overhaul, etc.

It's hard to say how quickly the team realized that this is going to be a lot more work than anticipated. We know that by November 2019, T2's earnings report had KSP2 pushed back to 2021, but it's not clear if that was still a realistic expectation, or just, "Push it back a year until we figure it out." What we do know is that Star Theory was renegotiating the work contract based on larger team and a later deadline. We also know that the negotiations didn't go well for ST, and about a dozen of developers jumped ship to start Intercept Games. Later, more of the 20-30 developers ST had joined Intercept as ST got dissolved in early 2020. From there it's fairly straight forward. New road map got made, the team started hiring people, and while the exact scope is still hard to pin down, it sounds like they were planning for late 2022 to early 2023 release, which suggests that either they had harder time growing the team or underestimated the problem or both.

But rolling back to that transition from Star Theory to Intercept, what can we say about the state of the game? We did see a preview of gameplay at Gamescon 2019. What do we see? New KSC, but still mostly blocked out with old terrain system. Rocket that does have shinier parts, but seems to be mostly KSP1 assets with an update to the materials. Very flimsy physics - suggesting they started experimenting with the physics overhaul, but got as far as removing autostruts. Abysmal framerate. (Clearly a priority on that last one, since they got it into the game so early. Sorry, couldn't resist.) What we don't see, but was probably heavily in the works are new parts for interstellar and colonies, and a bunch of planet work. Plus, probably tutorials. That last bit might seem like it's not a priority, but ST started out with good 2D art team, so I think they were trying to make the best use of the people they had.

The problems with all of this is that the overhaul made a lot of that work place-holder quality at best. The new material system meant that even the parts that would have been made at quality, needed an overhaul pass. Planets likely had to be scrapped almost entirely, because a completely new system was built. Code, likewise, mostly thrown away. The parts that would have survived are concepts, UI work, tutorials, and whatever assets were good enough to be used as a starting point. Everything else had to be built from scratch, which, if it sounds like, "Rest of the game," yeah, I agree.

[The Aziz]

4 hours ago, K^2 said:

I'm sure ST hired additional people to work on the contract for KSP2, but even most generously, you don't look at that team and say, "Yeah, they're going to build a AAA game from scratch in under 3 years."

I think that was (is?) the biggest problem in this whole endaevour. KSP1 was an indie game in pretty much every way. Independent studio, small team, niche genre, not much of a budget, a novelty on the market, but loved by those who were interested.

Then T2 took over the IP, came up with an idea for a sequel that would make money, and... still treated it like an indie title. Despite the fact that the scope proposed by our very Creative Director could beat some blockbuster AAA games. But it still got a small team, *some* budget and a short time to develop all those ideas.

 

In any case, we will probably know what happens next in 22 hours or so.

Edited May 15 by The Aziz

[cocoscacao]

8 hours ago, The Aziz said:

we will probably know what happens next in 22 hours or so.

End of May more likely 

[KerikBalm]

On 5/13/2024 at 7:56 PM, Scarecrow71 said:

Well, what we can gather from the Roadmap is that the scope was:

• A ground-up re-envisioning of the core systems of KSP1
  • Whether or not popular mods were to be added or integrated remains to be seen. We were literally told that DLC from KSP1, such as robotics, weren't going to make it into the game prior to 1.0, so we can probably infer that mods might not have ever made it

I was very excited about this ground up redo of the core systems, supposedly allowing vessels with much higher part counts than in KSP1.

It is my understanding that part count is even more limiting than it was in KSP1

Also, regarding mods, I did not mean litterally using the same mod, but rather incorporating similar functionality to functionalities provided by KSP1 mods

On 5/13/2024 at 7:56 PM, Scarecrow71 said:

• A new science system
  • Which turned out to be the old system just dumbed down a whole lot. [...]

Soo.... they win no points with me here

On 5/13/2024 at 7:56 PM, Scarecrow71 said:

• Colonies

Planned, not yet present after ~7 years of development. Also, KSP1 mods give this functionality.

On 5/13/2024 at 7:56 PM, Scarecrow71 said:

• Interstellar Travel

Planned, not yet present after ~7 years of development. Also, KSP1 mods give this functionality. I had hoped that KSP2 would be able ot handle this significantly better with the thrust on rails mechanic, allowing for long low TWR burns, with brachistichrone trajectories, etc. I had hoped for being able to plan maneuvers like with the MPD thrusters in CoDE, but instead it seems there's the same node that plans burns as if they are an impulse trajectory, and equires you to keep the vessel in active focus while thrusting. It is an improvment, but not what was really needed for more proper interstellar travel.

On 5/13/2024 at 7:56 PM, Scarecrow71 said:

• Resource Management
  • Which was kind of already in KSP1 with ISRU
  • And which we had a couple of mods for, like Extra-Planetary Launchpads, to deal with converting ore to iron (or something along those lines)

Planned, not yet present after ~7 years of development. Also, KSP1 mods give this functionality... Soo.... they win no points with me here

On 5/13/2024 at 7:56 PM, Scarecrow71 said:

• Multiplayer

Planned, not yet present after ~7 years of development. Also there's no indication they even have a clear idea how it would work

On 5/13/2024 at 7:56 PM, Scarecrow71 said:

That's a pretty large scope.  Just go through that and type out a simple problem statement and its resultant potential high-level solution for each one of those bullets, and you're already over 1 page in length.  That's a hell of a scope for any game. 

Here we need to distinguish between planned scope, and delivered scope. The scope they have delivered so far is underwhelming, and their planned scope is actually still within the limits of KSP's engine, as shown by mods. Maybe they planned for KSP2's engine to handle this scope better, but given the bugs and the performance limits restricting vessel part count, I see no external evidence that this has been achieved, or that good progress has been made.sadsad

On 5/13/2024 at 8:44 PM, K^2 said:

I don't know where to even star with how far off you are. Data-driven KSC, vehicles as part of the overall world data hierarchy, deeper hierarchy on vehicles and spaces allowing for more data-driven components,

I really don't understand what you mean here.

On 5/13/2024 at 8:44 PM, K^2 said:

procedural support for basically everything, with several procedural parts in the game,

KSP1's engine allows for procedural parts, as shown by a variety of mods for procedural tanks, wings, etc.

On 5/13/2024 at 8:44 PM, K^2 said:

just absolutely everything about the celestial bodies, but especially the procedural generation and how the environments are authored.

The celestial bodies are a high point of KSP2. I don't disagree here. To the point that I would even be interested in exporting their height and color maps, and backporting them to KSP1 

though I don't know what level of proc gen KSP2 uses for fine scale detail, I am not ignoring this point:

On 5/13/2024 at 8:44 PM, K^2 said:

And your throwaway "graphics overhaul" includes scattering, atmospherics, weather, entirely new materials system, etc. The "graphics overhaul" alone is a huge scope expansion requiring a team of artists and tech artists.

Scattering- they litterally hired the modder from KSP1 who did the KSP1 mod Scatterer. The weather is just cosmetic, no (ie, clouds)? And yes, the art team did a good job. I give KSP2's team credit for that. Its more of the core gameplay elements that I have reservations about.

On 5/13/2024 at 8:44 PM, K^2 said:

This is just stuff delivered at the start of EA. Not even discussing all the things that were still being worked on, but just stuff that was in the game at EA, which is only three years after Intercept started working on it, hiring up to full strength only, what, a year, year-and-a-half into it?

Oh, yeah, right. You think Intercept has been working on KSP2 for 7 years, having been founded 4 years ago. 

I did not say that. Please don't put words in my mouth. Work has been done on KSP2 for years, it is not clear when Star Theory started on it. Nate has been working on it from the beginning.  IIRC, about 1/3 of ST's team left to continue working at IG. There is certainly some level of continuity, and a core team that has been working on KSP2 since before it was announced in 2019.

On 5/13/2024 at 8:44 PM, K^2 said:

 - "I have experience actually doing this. This is wrong."
 - "I guess we'll have to disagree."

Neither of those are quotes from me.

On 5/13/2024 at 8:44 PM, K^2 said:

"I disagree" has meaning when it comes with an argument, experience, citation of evidence, or absolutely anything that actually addresses points being made. Otherwise, it's not even a good disengagement tactic.

See my points above, and also the following:

When speaking of scope, we need to distinguish between the scope they have delivered and what they has planned. What they have delivered after years of work (with the team varying in number and composition over time, but still with significant continuity), and access to KSP1's source code, does not seem to exceed KSP1's scope. 

1) I see that the visuals and the detail of the celestial bodies have been improved significantly.

Beyond that, (2) I don't see evidence of a more stable physics simulation (more bugs). (3) I don't see evidence of a more optimized physics simulation (has a lower part count limit before performance becomes unacceptable).

Points 2 and 3 above are absolutely critical to me. I would have been happy with KSP2 if it was just KSP1 with 1, 2, and 3, and the rest of the planned scope was left to mods.

Their apparent failure on points 2 and 3  (as far as I can tell) is the reason for my negativity. My opinion on if they did well or not depends heavily on whether they addressed points 2 and 3. They were supposed to rebuild the engine from the ground up to make it able to do all these things better than KSP1 - huge high part count vessels and colonies. They even stated that they would "slay the kraken"

As far as I can tell, they did great work on the art, but left the core engine in a dire state, and the state of the physics engine is why I cannot say they did reasonably well. IMO, that should have been their top priority. I'm left with the impression they were just doing artwork and daydreaming about what the engine would do, and didn't put in the work to make sure the engine could do it.

[Skorj]

16 hours ago, K^2 said:

I think the trailer was meant to represent that game, but potentially with a lot of artistic license. I'm picturing a game that's entirely based on KSP1 codebase, with basically two large DLCs strapped to it. Colony building - including parts and some basic resource management letting you actually build stuff there, and interstellar - one new star system and parts to get you there. Plus some visual improvements for an aging game. I think Star Theory of late 2017 was capable of that, and with a few hires probably on schedule for 2020 release.

Somewhere along the way, the dream got bigger. The E3 trailer was likely created with two goals in mind. 1) Stay within bounds of what's already on the roadmap, but 2) Present it as dialed to 11 under an excuse of the NOT ACTUAL GAMEPLAY disclaimer to gauge the reactions. I don't think it was meant to mislead, but rather represent the game ST wanted to make. And I think the pitch for that larger and bigger KSP2 was already in the air, and pending approval based on public reaction to the trailer. Reaction was good, lights were lit, and Nate is giving us a pitch for a bigger game pretty much from the first interviews. Multiplayer, colony management and transportation network are brought up, as well as plans for more star systems, realistic scale of space in between, physics overhaul, etc. UI work, tutorials, and whatever assets were good enough to be used as a starting point. Everything else had to be built from scratch, which, if it sounds like, "Rest of the game," yeah, I agree.

That sounds entirely believable.  Damn shame too, as a cleaned-up KSP1 codebase with some core fixes to let the physics scale better, and graphics the equivalent of the better mods would have won my heart.  Other than multiplayer, adding the rest as effectively big mods seems the safe way to go.  I'd think multiplayer would have to be plumbed in to the base game early.  Unless the team had some hands-on experience with multiplayer development, I don't think that would have been practical, 

I think a lot of us would have loved an EA release that was KSP1 with fewer bugs and updated graphics, with new features coming over time.  Even if multiplayer was off the table, I certainly would have been happy with colonies / logistics / interstellar coming in every quarter or two if it started with a better KSP1.

K^2, slightly off-topic, but how hard is combining multi-threading physics and multi-player these days?  I've done a ton of distributed systems dev, but the big performance gains required accepting that the system was not quite deterministic.  As I understand it, that's unworkable for multi-player, unless someone's gotten clever in the past few years.

[PopinFRESH]

9 hours ago, KerikBalm said:

....I did not say that. Please don't put words in my mouth. Work has been done on KSP2 for years, it is not clear when Star Theory started on it. Nate has been working on it from the beginning.  IIRC, about 1/3 of ST's team left to continue working at IG. There is certainly some level of continuity, and a core team that has been working on KSP2 since before it was announced in 2019...

In regards to that timeline; Nate's story regarding how he learned they'd be working on KSP2 and coupled with when his title changed at Uber Entertainment would point toward the beginning of development in Q4'17. Regarding the two referenced games and their credits listing roughly 18 developers and 1 engineer; Uber Entertainment still had multiple teams working different projects as they were in the process of spinning off Planetary Annihilation Inc. to continue their work on PA Titans and future support separately from Uber Entertainment. The head count at Uber Entertainment when they started KSP2 was reported to have been approximately 30 developers.

The Uber Entertainment rebranding to StarTheory in 2019 was due to losing/settling with Uber the ride sharing service in regards to the Uber trademark. When StarTheory failed to deliver the game within the contracted timeframe they began negotiations to extend the contract. To which Take Two attempted to purchase the studio and the resultant events unfolded with the contract renewal being denied and Intercept Games being formed. They poached roughly half of the development staff (around 15 developers) including all of the teams leadership, Nate R., Nate S., and Jeremy A. whom immediately left and moved to the new studio. Roughly 2 months later after StarTheory failed to find new contract work in March 2020 the owners closed the studio and a large portion of the staff that remained at StarTheory initially then rejoined their former colleagues at Intercept Games after the studio closure with some portion of the rest of them moving on to other endeavors. So they were roughly around that 25 - 30 head count during the original contracted development at StarTheory and the beginning of Intercept Games; largely being the same individuals. This is further evidence by comments made by Paul F. (former engineering lead) when he was laid off last year around the time of the EA launch. He noted being proud of building a team up from about 25 engineers to 45 engineers.

There is/was definitely a significant level of continuity in developers from before & after the studio change as you noted; and the core leadership team was those same individuals up until shortly after the EA announcement when Nate R. left and went to Epic Games.

[K^2]

9 hours ago, Skorj said:

K^2, slightly off-topic, but how hard is combining multi-threading physics and multi-player these days?  I've done a ton of distributed systems dev, but the big performance gains required accepting that the system was not quite deterministic.  As I understand it, that's unworkable for multi-player, unless someone's gotten clever in the past few years.

I'll start with tech overview, if you are interested. Physics and performance for custom engines has been my main area of work for the past few years, so I can go into a bit of a detail, but I'll add a tl;dr near the end.

 

Multithreading and multiplayer for physics are two separate problems, fortunately. Networking physics is hard because in most games multiplayer relies on rollbacks, and physics doesn't guarantee deterministic outcome on a rollback unless you have history of the constraint caches. The latter isn't too hard to keep, but you do need a few floats per joint or contact point in general for every frame of simulation you want to be able to go back. If you implement something like a circular buffer and have a strategy for dealing with blowing out the contact cache count, it's not too bad. I think that's roughly what Epic has been implementing for Chaos in their latest updates. (Though, I'm not fully caught up yet.) Unfortunately, it's not a standard feature of Havok, so you'd have to tinker to get it in. If you have custom physics, you're on your own, but then hopefully you have a good team to handle that. Note, however, that all of this is if you're doing rollbacks. If you're not, non-determinism isn't actually a problem.

Other than that, simulated objects are a lot like players. Each island can be predicted locally, but someone should have authority. In traditional server-client, the authority is always the server, so nothing special is going on, but you are in rollback territory if you want good response to player inputs on simulated rockets. In something like KSP2, you have an option of making each player an authority over their own ship. This can be done in P2P setup, or even server-client with client-trust on anything ship-related. The advantage here is that rollback isn't necessary. For the authority player, the inputs are taken into account immediately upon the first simulation, and for anyone else, rubberbanding is fine. If two players are trying to dock, both are either moving slowly enough where it shouldn't matter much, or too fast for anything but an explosion to follow. So this is a perfect game for this kind of an approach.

Networking aside, physics basically does four things. 1) Gather forces and torques, 2) Find collisions and generate contact constraints, 3) Solve for constraints forces, 4) Update velocities and positions based on net forces. Step 1 usually happens while you're updating all of your components or whatever game logic you have. E.g., applying thrust force to one of the rocket engines. The last step is almost trivial. It's O(N) and there's barely any math to do. So the steps that take a lot of computation and can benefit from being parallelized are 2 and 3. Collisions are very easy to split between threads, since each collision test is independent of all the others. Just have an atomic for next rigid body to grab, and ignore anything with a smaller index if it comes up in a relevant BVH leaf. Oh, and you have to have a strategy for the contact cache that allows for multithread access in some way, but that's a standard mutex problem. You will probably trip over the CPU cache a lot, so if you want the most out of this, it becomes a harder problem, where you want to respect the BVH when you are assigning out the work to threads. And if you have any static colliders (planet surface, etc) you really want to make sure you organize things in memory for it. But that's squeezing out the last 10-20% of the gain here, so not as critical as just having a way to run it in parallel.

Step 3, solving constraints, is the hardest to divide up. Usually, it's not where you end up with the worst cases on performance, though. Standard way of optimizing this solver is to just run one iteration per island. So long as your contact cache isn't changing often, and the spikes in forces aren't too dramatic, the constraint forces don't change much frame-to-frame, so one iteration is enough. And in something like KSP2, you do have very clear islands - most of the time rockets aren't colliding, so you can assume each rocket is its own island for the purposes of simulation, and dividing islands between threads is trivial. It does mean you'll have to do extra work if you found an overlap between colliders from two different rockets, including running a couple of extra iterations on the solver, because you'll have new and likely catastrophic forces, but that's sort of how you get the most out of physics in a game like this.

How much of this can you implement in Unity? Depends on what you're doing for physics. The stock physics is an old branch of PhysX which is just bad. It's single-threaded, has poor quality (as it's pre-Box2D revolution) and you can't do much to change any of that, since it's basically just a black box in Unity. It is, unfortunately, what KSP2 was stuck with. At the time KSP2 work started at ST, it was basically the only option, and by the time Intercept rebooted the work, alternatives were still new and untested.

Recently, Havok became an option if you run your game with ECS and DOTS. KSP2 was written with old MonoBehavior components, but if we get a development restart, the new team might chose to overhaul the components and go with ECS. I think that's what I'd want to do, but it depends a lot on the team. With Havok, you'd get threading for free, but Havok does absolutely nothing for networked physics. So you basically have to go with no-rollback solution for multiplayer, and it will have to be an entirely custom networking to handle all of that.

Finally, custom physics is always an option. I'd still cut a branch from an existing physics engine for collision detection, because it's just a lot of work to write from scratch, but you have options there. Bullet has decent enough collision detection and is open source under ZLib license, so you can use it in proprietary software. Then write custom contact cache solution that plays nicely with my networking and threading strategies, and write a custom solver that keeps KSP2 specifics in mind. There are a lot of cool optimizations you can do when you know that your rockets are almost rigid bodies, and that you can always use a rigid body solution as a starting point for a lot of the math, working with smaller deltas as a result, leading to much better physics fidelity with no computational overhead. The downside of this is that there are maybe a few dozen game dev engineers who have experience with this. Most of them work at Epic. The rest would have come primarily from id, Havok, Crystal, or Blizz. So rare and not cheap.

 

tl;dr: With KSP2 codebase as it is and under Unity, there are no real options. It's old branch of PhysX, which will be single threaded and awful for multiplayer. If we get an overhaul with the new team, that can be swapped out for Havok, which will give you multithreading for free, but will be a pain for multiplayer team to deal with. Still, probably the best possible outcome. And finally, custom physics is probably a non-starter, even though it's by far the best solution for something like KSP2.

In all of these cases, multiplayer is going to be solved differently than in most games, and your best option is a no-rollback approach with each player having authority over their own rocket to avoid non-determinism problems. Any standard server-client solution is going to be a disaster with KSP2.

 

 

2 hours ago, PopinFRESH said:

The head count at Uber Entertainment when they started KSP2 was reported to have been approximately 30 developers.

I'd check on that. The only sources I've seen for ~30 is for late 2019, when KSP2 went from ST to Intercept. That could easily have been 20 in late 2017 at the start of the development, and hired up to 30 for work on KSP2 specifically. If you have sources for ~30 developers at the late 2017, I'd appreciate the link. I know there were more people at the studio, but that was largely on administrative side, and these were never transferred over to the Intercept.

 

Edited May 16 by K^2

[PopinFRESH]

4 hours ago, K^2 said:

...I'd check on that. The only sources I've seen for ~30 is for late 2019, when KSP2 went from ST to Intercept. That could easily have been 20 in late 2017 at the start of the development, and hired up to 30 for work on KSP2 specifically. If you have sources for ~30 developers at the late 2017, I'd appreciate the link. I know there were more people at the studio, but that was largely on administrative side, and these were never transferred over to the Intercept.

 

It's a Bloomberg paywalled article but here is a quote except from it.

Quote

About a dozen of Star Theory’s 30 employees wound up leaving for Take-Two’s new studio, while the rest stuck around in an attempt to save the business, they said. By January, the remaining team had a plan in place: Each employee would spend the next two months brainstorming new ideas and building prototypes. Then they would pitch the best ones to publishers at the Game Developers Conference in mid-March in the hope of securing a new deal, the five workers said.

That was from anonymously interviewing 5 of the devs who remained at StarTheory.

In regards to prior to that I'm basing that on personally following the Planetary Annellation kick starter and the failed Human Resources kick starters and seeing who was involved in Uber Entertainment's development team. They were consistently around 30 developers (used in the more broad sense of engineers, designers, etc.) with not much turnover. (I was a fan of MNC and SuperMNC; and Jon Mavor having played thousands of hours of TA growing up at lan parties with friends. I really liked what they did prior to deciding to shaft PA kickstarter backers). If you want to go back an login to linked in via the wayback machine and look at their company profile and then view the insights about employees you'll see they were about that size prior to 2017.

http://www.linkedin.com/company/1513137 

That is the now defunct company profile for Uber Entertainment.

Edited May 17 by PopinFRESH

[K^2]

1 hour ago, PopinFRESH said:

In regards to prior to that I'm basing that on personally following the Planetary Annellation kick starter [...] They were consistently around 30 developers [...] If you want to go back an login to linked in via the wayback machine and look at their company profile and then view the insights about employees you'll see they were about that size prior to 2017.

Fair enough.

[Skorj]

@K^2 Thanks, I appreciate the detailed reply.  I'm really surprised rollbacks aren't a standard part of physics engines these days.  (It occurs to me that if you have enough to do rollbacks, you're most of the way towards being able to snapshot and then save in a different thread, which is something else I'm surprised isn't common.)  If you're doing custom work anyway, is Godot a better starting point than Unity?   At least then it's not a black box.

12 hours ago, K^2 said:

Just have an atomic for next rigid body to grab ...

I think I shouted "next BATCH of items to grab" out loud.   Old habits die hard, I guess.  I really need to deep dive what the fast way is in C# to do this correctly (this is much easier in Java, because the answer is "everything sucks, shoulda used a different language if you cared").  I'm used to thinking of any cache-piercing actions as being really slow, but I guess you can safely assume a single-socket system for gaming and my intuitions are probably all wrong. 

I like your "islands" approach a lot - if there were KSP1-style physics bubbles around what each player was controlling, you could get good results without so much work whenever they didn't overlap.  Of course, maybe docking is where you have the least tolerance for rubberbanding, so hmmm.  I think if I were attempting it on a small budget I'd just add a docking computer component to the game as a work around. OTOH, perhaps with smart use of part welding, so that as you mention a simple rocket was just a single rigid body, you could probably get away with a lot of not-very-performant stuff.  Especially the case where it's just two Kerbals running across the surface next to each other (or both interacting with static controls at e.g. some colony), which might be a lot of the gameplay.

 

[K^2]

14 hours ago, Skorj said:

I'm really surprised rollbacks aren't a standard part of physics engines these days.

Until very recently, Havok was basically the only dedicated engine worth considering, and they've, erm, always lagged a little behind top-of-the-line. Some in-house engines have had that feature. But being able to handle that kind of a network load in the first place is a relatively new thing in gaming. So it's not a huge surprise. Chaos is the first one that's going to be fully networkable out of the box, and it kind of makes sense. Epic ships with a networking solution already included and, despite some problems with it, it's what everyone will at least start with. So making physics work with that just makes sense.

14 hours ago, Skorj said:

I think I shouted "next BATCH of items to grab" out loud.   Old habits die hard, I guess.  I really need to deep dive what the fast way is in C# to do this correctly

Well, you wouldn't do this in C# either even if you were building it for Unity. A Unity plugin can call native code, and you'd put all your serious code into that. So C++, likely. If you have worker threads already spinning, they can just increment an atomic to get the next index to work on. That's typically the simplest thing to do, and gives every thread a sensible stop condition.

14 hours ago, Skorj said:

I like your "islands" approach a lot - if there were KSP1-style physics bubbles around what each player was controlling, you could get good results without so much work whenever they didn't overlap.

Not mine. I don't know who introduced the term "physics island" - might be Erin Catto, but at any rate, it seems to be fairly well established in the trade. It doesn't have to be spatially-limited. Just any collection of rigid bodies and constraints that are guaranteed not to interact with anything from another island. A number of engines only ever use one island (hence guaranteeing the requirement is met) and I know of at least one that still uses the island terminology for that lonely island.

If you do multi-island solver, you can take an optimization that static colliders (like planet's surface) can be interacted with by multiple islands without causing the two islands to merge. Since the planet's infinitely massive (for the purpose of simulation) it can't transfer recoil from one island to another. Colonies are another matter however.

Actually, colonies deserve their own note. If you're doing custom physics (and maybe even if you don't) I would make colonies rigid. You can still simulate joint limits without allowing the joints to flex. The advantage is that you only have to solve the constraints matrix once per configuration. So unless you enter build mode or something breaks from impact, you don't have to recompute it. And even when you do, you might be able to spread it between several frames, and you'll have a good starting point for iterations. So you just compute the inverse of a matrix and store it. When an island touches the colony, you solve for the constraints as if the colony was perfectly rigid. Once you have the contact force, you apply these to relevant structures in the colony, and run these though the stress matrix to see if any joints are overloaded. If they do, stuff breaks, shenanigans ensue. Otherwise, the colony remains as solid as part of the terrain.

This breaks a little with orbital colonies in theory, as you could have forces on one ship docked to a space colony transfer recoil through colony structure to another ship docked at the same colony, at which point they're really become one island even if you can treat the colony as rigid, but also, maybe just don't? While being able to boost a space colony with a docked ship seems like an interesting game mechanic, I think having colonies be fixed in the orbit once constructed would be a reasonable simplification. In that case, a space colony's basically a tiny moon with "infinite" mass, and then you treat it exactly the same as surface colonies, and problem solved. In this approach, two ships only become the same island if they collide, and again, that ought to be either slow and gentle, where things will probably settle naturally, or catastrophic, where you don't care about precision nearly as much, making it a simpler problem.

This approach to colonies would work even with stock physics, so long as you can basically create separate collision scenes for each ship. Or maybe each player? The latter would allow each client to run their own physics simulation for whatever ships are "theirs", adding other ships into sim only if some bounding box collision is detected, at which point they're removed from the owning player's simulation.  It's something I'd have to experiment with in Unity to see how workable that is, but it lets you split the work between clients, at least, and still work together on building the same colonies, including any docking procedures that don't directly involve two ships touching each other.

14 hours ago, Skorj said:

If you're doing custom work anyway, is Godot a better starting point than Unity?

Tough question, and I'm probably not the best person to answer it. I have limited experience with both. From what I do know, Godot is better if you have a small team that's more engineer/tech-art heavy. You can do more custom stuff that fits your team's needs. For a larger or more art-heavy teams, Unity's probably still a better choice, because it has more built-in tools and pipelines. But the problem is that Unity's been getting more and more bloated, and project restart and build times are getting obscene for larger games. Basically, they're hurting exactly where Unity needs to be the strongest to be competitive. So the domain where Unity is the preferred engine is shrinking, with Godot and Unreal taking larger and larger bites from it from each end.

For 2017 Star Theory composition and size, I think Unity was still the best choice. For 2020 Intercept, in retrospect, I would say they should have switched to Unreal. Problem is, a lot of it is hindsight. I'm sure if you searched, you can find some posts of mine from 2020-2021 where I say they had no reason to switch. I was involved in evaluation of Unreal 5 for a developer in 2021, and my early impressions of Chaos were not as good as they are now. Also, I don't think we really understood the scope of the new KSP2 design back then. Knowing where Unreal 5.4 is now and what KSP2 looks like, Unreal was a better choice. A lot of the tech Intercept spent a lot of time building for planets is just standard features in Unreal 5.4. They were raw and not production-ready in 5.0, which only officially released in 2022, but previews were available as early as late 2020 for some studios, and a lot of this was available as experimental features in Unreal 4.

So it would have been a very hard sell in 2020, but had Intercept gone with Unreal back then, they'd have a lot more time to work on gameplay, would have much better physics and networking out of the box, and would have a better art and asset pipelines available to them. All in all, KSP2 would be half a year to a year ahead of where it is now, possibly nearing 1.0 release. Assuming the team was able to hire Unreal specialists early enough, but that's generally a lot easier to do these days than some specialist roles they had to find to hack these features into Unity.

[Skorj]

1 hour ago, K^2 said:

Until very recently, Havok was basically the only dedicated engine worth considering, and they've, erm, always lagged a little behind top-of-the-line. Some in-house engines have had that feature. But being able to handle that kind of a network load in the first place is a relatively new thing in gaming. So it's not a huge surprise.

Yeah, it does make sense that the AAA studios would have that as an advantage of their custom engines.  Surprised it has taken this long to trickle down, but it's cool that Unreal is on it.

1 hour ago, K^2 said:

Well, you wouldn't do this in C# either even if you were building it for Unity. A Unity plugin can call native code, and you'd put all your serious code into that. So C++, likely. If you have worker threads already spinning, they can just increment an atomic to get the next index to work on. That's typically the simplest thing to do, and gives every thread a sensible stop condition.

(Totally agree on the threading approach.) I don't know, you can get some good performance out of C# these days if you know what you're doing, but I suppose it's a case of "do the hard stuff in the language you know best".  Thanks for inspiring me to read up on std:atomic, since that was added about the time I switched to distributed systems (which ironically don't usually use clever multi-thread optimizations).  Interesting choice by the committee to only offer the slow-but-consistent fetch-and-add based increment, and avoid the fast-but-inconsistent atomic increment.  The latter was always a footgun, of course.

On 5/17/2024 at 6:17 PM, K^2 said:

Actually, colonies deserve their own note. If you're doing custom physics (and maybe even if you don't) I would make colonies rigid.

I'd certainly do orbital colonies as rigid, unless everything was working great and the team had time left over.  The whole "do the minimum viable product first" approach.  IMO this was the big leadership failure in the KSP2 dev process, they aimed too high without a fallback that was still a good game.  I don't understand why they didn't start with "just KSP1 without the crashes", then incrementally add cool new ideas until the funding ran out.  That's such a basic best-practice for running a project.

For surface colonies, is there canned solution yet for "I want a large static map, except for some small cutouts here and there where terrain is replacable/deformable as a rare exception"?  KSP2 rolled their own ambitious solution, and small indie teams just seem to live with the restriction of trying to do base-building on non-deformable terrain.  Seems like the kind of common problem a game engine should solve, but maybe they're working on "just make maps of all deformable terrain fast" instead? 

Thanks for your insights on engines!  Unreal certainly turned some corner in the past few years, as all the "small indie team sim games" I follow switched to Unreal as easier in their current work.  But for KSP2 I'd do all the orbital physics as custom anyway, leveraging real-world solutions, so it would probably come down to what the art side liked better. 

[K^2]

6 hours ago, Skorj said:

I don't understand why they didn't start with "just KSP1 without the crashes", then incrementally add cool new ideas until the funding ran out.  That's such a basic best-practice for running a project.

It looks like that's what they were trying for. The most critical failure was overengineering the ship/world data without adequate testing along the way. They ended up with some configurations being usaveable, others resulting in conflict between logical configuration and physics configuration, and others yet breaking the editor. Between symmetry modes, constraints, and the difference between save-state data and live data, it turned into an unworkable mess. I don't know who screwed it up, but somebody should have escalated problems with it a lot earlier.

The other problems were milder. Relocation problems and KSC "teleporting" was probably still related to the same world state issues. And the only other major problem was that whoever implemented flight physics didn't understand how the integration errors in orbital mechanics work. Which, to be fair, is niche topic. I know a lot of competent physics engineers who think that they can always throw an RK4 at the problem and get consistent integration. Whereas, in space engineering it's a known problem that none of these standard methods conserve energy or momentum when integrating gravity. There are symplectic methods for gravity, but they come from papers like this one, and are generally tricky to implement. The two physics engineers they have had did not come from the field of physics where they'd likely be aware of these kinds of problems with the methods they were deploying. As a result, you had various decaying orbits persisting throughout the development, with many attempted fixes.

Everything else seemed to have been on track to resolution. I know the rendering was still way over the budget, but that's kind of typical for a project at a stage where you're still tuning a bunch of things to see what even works for the overall look of the game.

 

On the net, everything salvageable. I'd probably start the world/ship state from scratch and ensure it works correctly on every step. I'd implement LoD system with intermediate rigid bodies for the ships, to greatly reduce the joint count and to make sure you don't have large mass differences between the two sides the joints hold together (it's a known limitation of all modern physics engines). And finally, I'd replace cartesian integration of orbits to the one in Hamilton-Jacobi formulation, which lets you keep things on rails unless an external force acts on them. This works pretty well even for a restricted 3-body problem. Coupled with a symplectic integrator, the solution can be almost perfect for anything you can encounter in KSP.

It's a few months of work for a good team, but you really can't throw rookies at it. You need someone who has done physics in games to TD and architect it. That's a small-ish pool of people, so it won't be cheap, but better drop 7 figures into that then just keep burning it with a team that has to stumble into the correct solutions over a much longer period of time.

[TablesRUs]

34 minutes ago, K^2 said:

I'd replace cartesian integration of orbits to the one in Hamilton-Jacobi formulation

Yes, I always found it very strange that a energy-centric approach wasn't taken from the very beginning; this seems like an obvious beginner's mistake made in the first game and Hamiltonian-esque mechanics really does seem like the natural formulation mechanics in this setting.

Edited May 20 by TablesRUs
Expand

[K^2]

4 hours ago, TablesRUs said:

Yes, I always found it very strange that a energy-centric approach wasn't taken from the very beginning; this seems like an obvious beginner's mistake made in the first game and Hamiltonian-esque mechanics really does seem like the natural formulation mechanics in this setting.

The math does get a bit hairy. Not at a point where it should scare anyone with a physics degree, but it can be rough for a lot of game devs in general. And even if you have a Ph.D., this is niche. Meaning, you should be able to follow along, but building it from scratch without the experience? That is iffy. The constants of motion for inverse central potential are E, L, ν₀, ω. (Energy, angular momentum, anomaly at epoch, and argument of the periapsis.) Note that for a given gravitational parameter μ, combination of E and vector L gives us the standard orbital elements a, e, i, Ω and vice versa. (Semi-major axis, eccentricity, inclination, and argument of the ascending node.) So the constants of motion representation and orbital elements representation are essentially equivalent, and you'll probably swap between the two depending on whether you're doing physics or updating positions or any displayed values. So far pretty standard.

The fun part starts when you apply thrust F. The E and L updates are trivial and depend only on the position relative to the center of attraction r and velocity vector v. dE/dt = v·F and dL/dt = r⨯F. But then the update for ν₀ and ω actually get quite messy. (Left as exercise to the reader, because I don't want to.) The advantage, though, is that if you're doing a one-body SoI, you're done. This form automatically conserves energy and momentum - the only change to these two comes from the thrust equations, so long as we keep deriving the local cartesian coordinates from the orbital elements. So for the entirety of Kerbol system, you have a perfect sim.

If you do have a restricted three-body, such as the Risk twins, and you want to use the above, you'll probably still want to pick your primary SoI in whatever reduced system makes sense, then apply additional forces as correction. So it'd be as if there was a source of external force that isn't the ship's engines. Here we're back to having concerns about the conservation laws. In practice, the orbits get chaotic enough that you probably don't care, and you can just use an RK4 and save yourself sleep. But if you really want to, the symplectic integrator approach from the previous article is done in generalized coordinates, so you can apply it to the constants of motion of the unperturbed Hamiltonian to derive something truly precise.

This is the point where we should chat about the warp, though. Maximum warp of KSP is 10⁵. Alpha Centauri is 4.3ly from Earth. The interstellar distances are reportedly "realistic," which in the most generous interpretation means 0.4 Kerbin light-years at realistic c of 3x10⁸m/s. So most generously, we're looking at something like 10¹⁴m. A 1g torch ship would traverse this in ~73 real world days. At 10⁵ warp that's about 1 minute. Which is fine, but only if you really can torch the whole way, your frame rate holds, and there's nothing at significantly higher distances. In other words, 10⁵ must be supported in KSP2 as an absolute minimum, and needs to be rock solid. Ideally, you want to be able to push it to 10⁶ while maintaining a 1:1 simulation.

So, say you're running a busy 10³ objects (ships, satellites, debris) and you want 10⁶ warp. If you wanted to maintain fidelity equivalent to a 30Hz simulation, leaving you with about... 0.1 instructions per object? Ok, so 30Hz is clearly a no go, but do we need to? Obviously, no. For any object without external forces (so not in Risk sytem and with engines idle) it's sufficient to check for next SoI intersection and set a reminder for then. This part is actually an interesting code challenge in itself. Clearly, a lot of objects are just never going to go anywhere. A satellite near Kerbin with PE above atmo and AP below Mun is going to stay in that orbit forever. But something interplanetary might clip into SoI of another planet at some point in the future. Should you always predict this? Personally, I'd ignore it for debris above certain warp and just say that if you have no crew/cores and you're confined to current SoI, ignore inner SoI intersections. That leaves us only with solving for the motion of things that are either under power or might go under power (due to crew, probes). And this is where the method you use for integration matters. A full symplectic integrator in constants of motion coordinates might get into thousands of cycles per step. This will likely be your limiting factor. So the question becomes, can we cut this further?

And yeah... Remember how I said that E and L updates are trivial? They still are, and they're still what we most care about. For objects under thrust (or other external force) you can simulate in E, L, r, v coordinates. Note the redundancy. You want to update r and v as a Verlet, and renormalize to E and L, while the latter are updated from force integration only. This will have a precession drift, but that should be tiny compared to true precession due to applied force and it won't yeet your objects out of the system like flees as integrating only r and v would. As a final piece, apply variable time steps to this approach. What that last bit gives you is that anything far away from a gravity source will travel in mostly a straight line, where you can do coarse steps. And anything going around a heavy object won't stay there for long, because if it's applying enough thrust to make you go to fine steps, it'll either escape the system or crash into the massive object. Either way, you won't have to care about it for too long, and your overall simulation rate gets to stay high. With this approach you really can have tends of ships under power with thousands of pieces of debris running 10⁶ warp and not causing half of your satellites to disappear because they crashed into something they shouldn't have.

It does still mean that the physics will have to be frozen on the ships themselves, just like in non-physics warp in KSP1, and you'll have to get clever with resource use, and how that's updated. (And not whatever unholy nightmare was plaguing both the KSP1 and KSP2...) But all of these problems are now independent of the coordinate systems and how we do the integration. They're just something KSP2 team would have had to deal with to support ships that continue using up fuel as they're traveling under warp.

So hopefully above gives a bit of an outline of both what KSP2 team should have been doing with orbital physics, and why they probably didn't think about it. The physics experts the team had knew basically nothing about the game development. And the game engineers on the project were Unity devs, for whom it's very natural to think of every rocket as a collection of scene nodes with mono behaviors. And yeah, you can't do that in a game they were trying to make. I hope they were going to figure it out eventually, but I can see why they didn't from the start.

[Skorj]

33 minutes ago, K^2 said:

The fun part starts when you apply thrust F. The E and L updates are trivial and depend only on the position relative to the center of attraction r and velocity vector v. dE/dt = v·F and dL/dt = r⨯F. But then the update for ν₀ and ω actually get quite messy. (Left as exercise to the reader, because I don't want to.) The advantage, though, is that if you're doing a one-body SoI, you're done. This form automatically conserves energy and momentum - the only change to these two comes from the thrust equations, so long as we keep deriving the local cartesian coordinates from the orbital elements. So for the entirety of Kerbol system, you have a perfect sim.

This.  The code would be dead simple (relative to "rocket sim").  Further, if you remove game physics for anything not in the physics bubble and, keep track of orbital parameters and not coordinates, then you don't even need to know where anything is until it needs to be displayed.  You could have any number of bodies in elliptical orbits and give them 0 computational effort until you needed to display them.  Perfect for high timewarp, and orbits cannot decay, or for that matter wander due to cumulative FP rounding error.  As you say, SOI-crossing orbits are a matter of setting a timer (which they should be getting right anyway to support Alarm Clock in the base game.).

For high timewarp, I'd certainly start development with treating the boosting ship as a rigid body.  Modeling joints under boost at high timewarp seems like a "nice to have" at best.

Until you mentioned the Risk twins, it didn't occur to me why you were discussing 3-body sim.   Wow, talk about starting with an overly ambitious dream.  Man, talk about stuff to defer until after everything else is working!  That was the tail end of the roadmap, and it seems like they started with it.

Upthread you mentioned the problems in KSP2 with save data and live data diverging.  Is there some common problem with saving/restoring full physics data for the public engines?  I've seen small indie teams using both Unity and Unreal just treat that as an unsolvable problem and just not save physics state despite being physics-heavy sims.  I'm curious whether that's actually difficult with stock physics engines, or it's a case of buying some game save tool off the store without understanding it, so they're stuck with the positional-only information it came with.

[steveman0]

14 hours ago, Skorj said:

I don't know, you can get some good performance out of C# these days if you know what you're doing, but I suppose it's a case of "do the hard stuff in the language you know best"

Certainly modern C# has come a long way, but I know Unity for the longest time was stuck on much older versions of C# that wouldn't see these benefits especially where Mono is concerned.  I'm not familiar with the version of Unity they are running and what the latest version of C# it supports, but I could certainly see the limitations here making C++ notably more performant in addition to being more familiar as you noted.

[Skorj]

8 minutes ago, steveman0 said:

Certainly modern C# has come a long way, but I know Unity for the longest time was stuck on much older versions of C# that wouldn't see these benefits especially where Mono is concerned.  I'm not familiar with the version of Unity they are running and what the latest version of C# it supports, but I could certainly see the limitations here making C++ notably more performant in addition to being more familiar as you noted.

Yeah, C# has had a lot of improvement in the past few years, which is surprising for such a mature language.

It's baffling why Unity is stuck with such an old version of C#, perhaps they see Android as their primary platform now, but it's been clear for a while that they've lost their way.  But of of course that wasn't obvious in 2017.

[K^2]

28 minutes ago, Skorj said:

Upthread you mentioned the problems in KSP2 with save data and live data diverging.  Is there some common problem with saving/restoring full physics data for the public engines?  I've seen small indie teams using both Unity and Unreal just treat that as an unsolvable problem and just not save physics state despite being physics-heavy sims.  I'm curious whether that's actually difficult with stock physics engines, or it's a case of buying some game save tool off the store without understanding it, so they're stuck with the positional-only information it came with.

There are two parts to it. One is just making sure the two models match and the other is stability.

The first part's solvable. You just have to be thorough. It's very clear that there are situations where you would load a game in KSP2 and some connectors are marked as decoupled in logic, but the physics joint is still there. And that's just a programming error. The fact that there are lots of such errors stems from some bad architectural choices, but even so would be resolved iteratively after some number of patches.

The second part is stability. A solver has to have an internal state, because otherwise convergence will take too many iterations for every frame. And you can in theory dump that internal state and restore it from a save. It's not even hard if you have your own custom engine. But neither Unity nor Unreal give you easy access to it. So instead, when you start the simulation, the internal cache is zeroed out. If you had a very wobbly rocket and you saved the game and loaded, you might come back to a rocket that's rapidly disassembling itself, due to the forces exceeding joint limits before the simulation had time to settle. There are ways to improve on that too. You can run simulation at a much shorter time step until it settles, for example. It would cause the game to be slow-motion for a few seconds on a state save, but I think that'd be very much acceptable. So it's a problem, but hardly game-breaking.

[cocoscacao]

2 hours ago, K^2 said:

If you had a very wobbly rocket and you saved the game and loaded, you might come back to a rocket that's rapidly disassembling itself, due to the forces exceeding joint limits before the simulation had time to settle.

Wait... What?! 

[K^2]

6 minutes ago, cocoscacao said:

Wait... What?! 

The solver approximates tension in joints iteratively. To avoid iterating many times per frame, the result from last frame is used as a starting point, and only one-two iterations are done to correct for changes. If you reloaded a scene, these tensions are set to zero, and if that's far from what it takes to keep the rocket together, for whatever reason, first few frames will over-correct, resulting in excessive forces.

Have you noticed how big rockets do a bouncy-bounce when they spawn in on the pad? And how if they survive that, it's probably alright? Same principle, but for a complex enough build, you don't even need gravity. Something somewhere is probably under a bit of tension just due to how the distances work out, and that will ripple out. With the unfortunate enough configuration of the joints and joint strengths, insta-kraken. Still happens even in KSP1, albeit, rarely.

[Scarecrow71]

14 minutes ago, K^2 said:

Have you noticed how big rockets do a bouncy-bounce when they spawn in on the pad?

You know, that's something I never could figure out as to why that happens.  I mean, I get that IRL rockets are rolled slowly from where they are built to the pad, and that in the game they spawn on the pad.  I just never understood why the physics would have the rocket start jumping up and down upon being spawned.  It's as if the game spawns the rocket ABOVE the ground, and then it falls (albeit very slightly) to the ground, causing it to bounce around.

Am I close on that?  Or do I need a new set of eyes?

[cocoscacao]

33 minutes ago, K^2 said:

The solver approximates tension in joints iteratively. To avoid iterating many times per frame,

I thought it runs on each engine "tick" and I didn't know it approximates. I assume it at least goes through every part on each iteration...?

Edited May 20 by cocoscacao
updated

[steveman0]

1 hour ago, K^2 said:

The solver approximates tension in joints iteratively. To avoid iterating many times per frame, the result from last frame is used as a starting point, and only one-two iterations are done to correct for changes. If you reloaded a scene, these tensions are set to zero, and if that's far from what it takes to keep the rocket together, for whatever reason, first few frames will over-correct, resulting in excessive forces.

Have you noticed how big rockets do a bouncy-bounce when they spawn in on the pad? And how if they survive that, it's probably alright? Same principle, but for a complex enough build, you don't even need gravity. Something somewhere is probably under a bit of tension just due to how the distances work out, and that will ripple out. With the unfortunate enough configuration of the joints and joint strengths, insta-kraken. Still happens even in KSP1, albeit, rarely.

I never thought too deeply before about how the solution is actually achieved. The iterative nature makes perfect sense though. This brought to mind the other stability concerns I had observed in the game. Namely the spotaneous rapid disassembly bug.

This doesn't require it to occur immediately after a scene load, though it often does. I assumed this was due to the lack of sensible dampening forces leading to rounding errors becoming blown out by some singularity. I assumed that this would be something applied outside to overlay the physics model, but it is sounding like there might be more reliance on the canned solution that makes it more opaque in understanding what is at fault.

I'm curious to get your thoughts on this. Would the solver have something built-in to suppress spurious noise that could lead to this kind of result, or is that something that would be left to the developer to layer over the physics model to address? Anytime I've built a model for some project from scratch, I would plan some kind of corrective force to ensure a stable state for predictability. Could some of the problems observed be explainable by the absense of something similar here? This seems necessary to ensure stability even if it might not be 100% accurate to life given the constraints imposed by the boundary conditions.