[KSP >= 1.3.1] Distant Object Enhancement (DOE) /L - Under new Management - 2.1.1.15 - 2024-0103

[Lisias]

10 hours ago, darthgently said:

Given even the lastest consoles have multiple cores it is way overdue for games to make use of them.  

I completely agree.

 

10 hours ago, darthgently said:

 A lot of KSP involves switching between craft, and then back again.  It would be great if the most recent flight scene were somehow kept alive on another core so that switching back would be nearly instantaneous, along with the Tracking Center being constantly alive on its own core.  Background Resource processing could also be running constantly.

Exactly. There's a pretty old Design Pattern called DocView, created by Microsoft for MS Word and it's a simplification of the MVC one. The Office Team (the group where the real improvements happened in the past, as DDE, COM, DCOM, etc) realised that they don't need to have all the visual thingies tied to the code that handles the bits of the document, but yet the MCV was overkill for them. So they wrote two components, the Doc (where the Controller and the Models live) and the View responsible for the U.I. And then suddenly you can have multiple views working fine on the same Doc (one view for preview the print, other to edit, perhaps yet another one to edit another part of the same document - you got the drill).

Games should be doing something similar - assuming, of course, they are not doing it already - at least, the ones made with better 3D engines than Unity (that it's not exactly bad - it's just not good neither).

 

10 hours ago, darthgently said:

The limiting factor would likely be RAM

Give a peek on how the venerable PlayStation 2 works. The Emotion Engine is still respectable even nowadays.

You can do a lot with low memory footprints when you know what you are doing.

Edited October 27, 2021 by Lisias
Tyops, as usulla….

[Grimmas]

Games and game engines have been doing something even more complicated for the better part of two decades now, it's called an entity component system. It breaks down game behavior into data and/or functionality blocks and uses those blocks to compose different game objects. It's baked into Unity and into many other games/engines. Visual and Physics are merely two of the many blocks (some games have 200+ such "components"). In KSP, an easy to understand example are modules, which you add to a part to give it some functionality.

However this is not as trivial to multithread as you think. Games are inherently sequential - you need to process frame by frame, sixty+ times per second (less often for physics, sometimes). And logic components often need to influence the state of the world, not only know about it. If objects influence the state of the world in different order then this can be the source of nasty hard to track down bugs (resulting for example in one player's game where Han shot first and another player's game where Han died because the other guy got processing time and shot first instead).

What people have come up with so far that works reasonably well is to massively parallelize only at certain stages (physics processing being a big one, for instance) and then sync it all back together before moving sequentially again. In KSP, since a vessel does not really meaningfully interact with another vessel most of the time, they also live in separate threads. 

[Gotmachine]

13 hours ago, darthgently said:

It would be great if the most recent flight scene were somehow kept alive on another core so that switching back would be nearly instantaneous

The loading times on scene switch is due to a bad design decision to have the core game database belonging to scene objects instead of being DontDestroyOnLoad root objects, which cause KSP to make almost the whole game to go through an useless full serialization/deserialization cycle.  To their discharge, managing that pattern is quite a lot easier in recent Unity versions and the "additive scenes" feature.

32 minutes ago, Grimmas said:

In KSP, an easy to understand example are modules, which you add to a part to give it some functionality.

KSP doesn't use the ECS system. It's entirely built on the GameObject system. And BTW, Unity's ECS implementation is merely a few years old, and has been out of "experimental" only since last year.

32 minutes ago, Grimmas said:

In KSP, since a vessel does not really meaningfully interact with another vessel most of the time, they also live in separate threads. 

This is a (common) urban legend. There is zero threading involved in any part of KSP, at the notable exception of the stock delta-v calcs.
The only threading happening is on the Unity renderer side and on the PhysX engine side.

Edited October 27, 2021 by Gotmachine

[Grimmas]

Any time you prefer composition over inheritance in gamedev you basically have an ECS. Doesn't matter if it's the ECS built into Unity or the KSP home grown ECS. You are composing game objects out of different components that provide discrete functionality. This is about the pattern, not about the concrete tools, and ECS has been a common programming pattern in gamedev since around the first time Scott Bilas first presented it at GDC way back in 2002. They used it in Dungeon Siege back then, though they didn't call it an ECS at the time, the fancy terminology came later. 

Edited October 27, 2021 by Grimmas

[Lisias]

23 hours ago, Gotmachine said:

This is a (common) urban legend.

It's more like a false cognate. We should be using the term Concurrency, but on modern software development (at least on system and comercial level), nowadays concurrency and threads are (wrongly) used as synonymous.

Co Routines, as used on Unity, are a kind of concurrency: on the old days of Win16, Workbench (Amiga), TOS (Atari ST), MacOS (on the 680x0 times) et all, the Concurrency were achieved by cooperative multitasking that is, in essence, Co Routines with steroids, but still Co Routines - we had the same problems and the same limitations from what we have on Unity (and used some of the same hacks too).

But it's nearly THIRTY YEARS since everybody is using threads to achieve MultiTasking, and so everybody ends up using both words as synonymous (including me).

It worths to mention that a lot of the problems I faced on Unity are copycats of problems that I had faced when giving maintenance on older Win16 code on Win95 (when finally Windows started to have Preemptive Multitasking, besides half baked). And I can't describe properly the huge pain that was porting these code into to Windows NT later, where Preemptive Multitasking were done right.

 

23 hours ago, Grimmas said:

What people have come up with so far that works reasonably well is to massively parallelize only at certain stages (physics processing being a big one, for instance) and then sync it all back together before moving sequentially again. In KSP, since a vessel does not really meaningfully interact with another vessel most of the time, they also live in separate threads. 

As was said above, not exactly - there're a lot of missed opportunities to explore parallelism on KSP - and, frankly, not all due legacy code. You can rewrite system level functionalities without breaking (well behaving) userspace programs, and the Linux Kernel is one of the most formidable examples on how this can be done. Even Win32 is not that bad (it's messy, but not bad - go see how was POSIX on the old days), most programs compiled agains Win32s (on the Win16 yet) could be compiled and run on Windows 95 almost without a change.

Even Win16 using WinG could be recompiled to Win32 on Windows95, damn it. And it worked (or the most near to work you can expect from Windows 95).

But a huge amount of software could not be ported to Win32, and one of the reasons are very similar to the problems we have nowadays on KSP: completely ignoring Concurrency concepts. Sometimes, I think there're some MS-DOS game developers lingering around.  (not going pejorative on them, by Kraken's sake - these guys were heroes on a war time. But we are not on a war anymore!).

Going back to the multi-vessel problem, one problem on going multi-thread on them is the Reference Frame. KSP has only one Reference Frame, the user PoV. So, when you switch vessels, the whole Universe is transposed in order to make the User's current position as the Origin (0,0,0) - essentially, the user never moves, everything else does it instead.

This hack was needed because Unity (at that time) used floats for everything (what made sense, we were using 32 bits CPUs at that time), and floats don't have the needed resolution to allow precise calculations on two far points of the Universe - so transposing everything to make the User the Origin alleviated the problem by removing from the user's visible space the problems related to low float resolution. Interesting enough, DOE is an excellent way to check exactly that, since it works from Unity 5 to Unity 2019 - recreating my demo scenario (the SpySat thingy) on KSP 1.4.3 and on 1.8.1 gave me radically different rendering results on the 1.3.1 (due Unity 5), from texture fighting to weird polygons.
 

Spoiler

The irony of it is that, nowadays, doubles are 30% faster than floats (as everything is 64 bits now), we would get a substantial improvement in performance by converting everything to 64 bits - assuming it can be done, we have a lot of legacy coded on 32 bits yet and, as I said, I'm pretty sure a lot of that code was node programmed on a portable way. I had survived the 8 to 16 bits transition, and then the 16 to 32 - experience taught me to avoid optimism on such transitions.

So a possible way to solve this mess is to decouple the Origin from the User's position, create many Reference Frames (one for each living vessel) and allow the User to transition between the different Reference Frames as a discrete entity (and not as the Origin of the Universe). This would decouple the vessels from the float problem, you could have loaded vessels scattered about the Universe at the same time without needing to unload them into Rails all the time (as long you have CPU to handle everything).

Reference Frames should be handled on 64 bits, vessels inside each Reference Frame can still be processed on 32 bits - not differently from what was made on Windows 95 to run WIn16 programs.  

In essence, each Reference Frame can run on a thread or core concurrently with the other ones. And the process that macro handles the Reference Frames could also run on a dedicated thread/coroutine/whatever too.

As long the vessels don't get near each other.  

Once two vessels enters into the each other's physics range, you will need to merge their Reference Frames (and later split it again when they get far from each other). I think this is where things can get prohibitively complicated.

[addendum]

It's worth to remember that we are terribly underusing our GPUs nowadays. The machines are made to munch numbers, and we could offload a lot of heavy computing from the CPU into the GPU, as long we can cope with the limitations.

GPUs handle floats tens orders of magnitude faster than they handle doubles, so keeping the Reference Frames on floats is desirable as it at least allows us to have hope that we could offload these into the GPU someday.

[/addendum]

 

23 hours ago, Grimmas said:

What people have come up with so far that works reasonably well is to massively parallelize only at certain stages (physics processing being a big one, for instance) and then sync it all back together before moving sequentially again. In KSP, since a vessel does not really meaningfully interact with another vessel most of the time, they also live in separate threads. 

What's not different from system programming. Parallel compiling is one perfect example of this problem being solved on real life.

 

22 hours ago, Grimmas said:

Any time you prefer composition over inheritance in gamedev you basically have an ECS. 

The two concepts are not mutually exclusive. You can use both, exploring the strengths and avoiding the weakness. What we need is good interfaces on well defined borders/layers - give me a good interface, and I can make an Apple II running ProDOS work together with a SAP box running on a Sun M9000, or use a Raspberry PI to offload some work from it.

Edited October 28, 2021 by Lisias
Addendum

[SiCaRiO31]

On 10/20/2021 at 7:22 AM, Lisias said:

ANNOUNCE

Release 2.1.1.6 is available for downloading, with the following changes:

• Some brain-farts of mine on handling Scene switch were fixed
  • Thanks for the report, @darthgently!
  • And for the one from @Krazy1too! While fixing the previous, I detected what could be happening on this one and (hopefully) fixed it.
• Testings down to KSP 1.3.1 suggests it works on these, but
  • "Development" was done on KSP 1.4.1 and 1.4.3, and then tested against 1.7.3 and 1.12.2 and no problems (others than my own ones) were found! #HURRAY!!
  • KSP 1.3.1 appears to work, but I didn't "certified" it yet. Try at your own risk
  • On the bottom line, the thing runs downto 1.3.1, but I'm not confident enough yet.

I essentially remade VesselDraw this time, as the previous coding style was becoming hard to "escalate" (how is the exactly term in EN-US?). This fixed a bunch of issues by itself, as they were originate by my attempt to prevent doing exactly that (if it's working, don't fix it! It only happened that it was not working as I needed, so now I fixed it).

I think  it will work fine on KSP 1.3.1, I surely did some test runs on it. But not close from the scrutiny I'm doing on KSP 1.4x to 1.12.2 (1.4.3, 1.4.5, 1.7.3, 1.8.1 and 1..12.2 - as experience tells me that if the damned thing works on these ones, it almost surely with work on the others too! KSP 1.5.1 and 1.6.1 passed the smoke tests at least, I ran out of time to keep testing the Unity 2019 series, except by 1.12.2, of course). As I said before, it was by plain luck I didn't used anything not available (and working) on KSP 1.3.1, so we still may get strike by something. So I prefer to deferrer an official statement for the WeekEnd, when I will have time to play with the thing.

No screenshots this time, I'm in hurry.  

See OP for the links.

— — — — —

This Release will be published using the following Schedule:

• GitHub. Right Now.
• CurseForge. Tomorrow night (or a bit sooner)
• SpaceDock. (By Friday, probably - SpaceDock is scheduled to have a major update and I will wait to see what happens - just in case).

The rationale is to gradually deploy the thing in order to cope with eventual mishaps before it reaches too much people.

will this be avaliable in CKAN ? I wanted to upgrade but couldnt find this version there. 

[Lisias]

On 10/29/2021 at 6:37 AM, SiCaRiO31 said:

will this be avaliable in CKAN ? I wanted to upgrade but couldnt find this version there. 

Yes. CKAN is updated together SpaceDock, and I do it on Fridays. On Working Days I'm not always available for support if anything goes wrong, and I ended up settling on updating these two only when I know I will be able to handle things after the update.

This will happen today more or less 21:00 UTC.

Edited October 30, 2021 by Lisias
tyop! Surprised?

[monstah]

On 10/20/2021 at 2:22 AM, Lisias said:

 

I essentially remade VesselDraw this time, as the previous coding style was becoming hard to "escalate" (how is the exactly term in EN-US?).

"didn't scale well", I believe  

On 10/29/2021 at 7:12 AM, Lisias said:

Yes. CKAN is updated together SpaceDock, and I do it on Fridays.

So, every time I fire CKAN in the week and try to se what's new, I'm just making a fool of myself?

[Lisias]

6 minutes ago, monstah said:

So, every time I fire CKAN in the week and try to se what's new, I'm just making a fool of myself?

Not every time.

On some weeks, my  day-job workload is pretty manageable and so I update things on workdays. Important bug fixes that don't risk breaking things are also uploaded fast.

I usually postpone things to Friday when my dayjob is demanding (as I will not be able to respond on a reasonable time) or when the update are… humm… fearsome  and I need to reserve time to do proper support to people bitten by borderline situations I didn't anticipate (or plain let them pass trough - hey, it happens!  ).

[Lisias]

Hey!! DOE's thread was nominated one of the Threads of the Month for 2021/11 !!!

Spoiler

 

 

[zer0Kerbal]

congrats! 

[OrbitalManeuvers]

So I have a zip file called Distant_Object_Enhancement_Continued-v2.0.3.1.zip which contains a .version file that says this:

{
  "NAME": "DistantObjectEnhancement",
  "URL": "https://raw.githubusercontent.com/TheDarkBadger/DistantObject/master/GameData/DistantObject/DistantObject.version",
  "DOWNLOAD": "https://github.com/TheDarkBadger/DistantObject/releases",
  "VERSION": {
    "MAJOR": 2,
    "MINOR": 3,
    "PATCH": 1
  },

Am I correct that 2.3.1 was just a typo for a hot minute, and your current release is the direct replacement for what I have above?

[zer0Kerbal]

@OrbitalManeuvers

https://www.curseforge.com/kerbal/ksp-mods/distant-object-enhancement-l/files

https://github.com/net-lisias-ksp/DistantObject/releases

I believe [DistantObject-2.1.1.6-CurseForge.zip] is the file name for the most recent release.

Suggest Downloading from CurseForge - to help support your mod author.

[Lisias]

3 hours ago, OrbitalManeuvers said:

So I have a zip file called Distant_Object_Enhancement_Continued-v2.0.3.1.zip which contains a .version file that says this:

{
  "NAME": "DistantObjectEnhancement",
  "URL": "https://raw.githubusercontent.com/TheDarkBadger/DistantObject/master/GameData/DistantObject/DistantObject.version",
  "DOWNLOAD": "https://github.com/TheDarkBadger/DistantObject/releases",
  "VERSION": {
    "MAJOR": 2,
    "MINOR": 3,
    "PATCH": 1
  },

Am I correct that 2.3.1 was just a typo for a hot minute, and your current release is the direct replacement for what I have above?

UGH! It's a typo!  It was meant to be 2.0.3.1...

The most up to date release has this version file:

{
	"NAME"		: "Distant Object Enhancement (DOE) /L",
	"URL"		: "https://raw.githubusercontent.com/net-lisias-ksp/DistantObject/master/DistantObject.version",
	"DOWNLOAD"	: "https://github.com/net-lisias-ksp/DistantObject/releases",
	"CHANGE_LOG"	: "https://raw.githubusercontent.com/net-lisias-ksp/DistantObject/master/CHANGE_LOG.md",
	"CHANGE_LOG_URL": "https://raw.githubusercontent.com/net-lisias-ksp/DistantObject/master/CHANGES.md",
	"VERSION":{
		"MAJOR":2,
		"MINOR":1,
		"PATCH":1,
		"BUILD":6
		},
	"KSP_VERSION_MIN":{
		"MAJOR":1,
		"MINOR":4,
		"PATCH":1
		}
}

 

[Krazy1]

This is really trivial and you should ignore it... but could you make comet tails translucent to planets?

This is near a comet looking backward through the tail (while hovering the mouse on Duna). Ship is retrograde to the comet. 

[Lisias]

27 minutes ago, Krazy1 said:

This is really trivial and you should ignore it... but could you make comet tails translucent to planets?

This is near a comet looking backward through the tail (while hovering the mouse on Duna). Ship is retrograde to the comet. 

I'm not sure I understood (language barrier probably). Do you want to see Dune trough the comet's plumes, it's it?

[Krazy1]

12 minutes ago, Lisias said:

Do you want to see Dune trough the comet's plumes, it's it?

Right now planets can be seen through the plume. I'm asking if the plume could obscure the planet. In this picture, the dust is very thick so I think we should not be able to see Duna. Ideally the planet brightness would vary with the dust density. (Like the previous request where a planet can be seen through the atmosphere in orbit or through the sun.) But this is a rare use case! 

[SiCaRiO31]

I believe this was deemed unfixable, since the game renders first the atmosphere and then the decals. 

[Lisias]

4 minutes ago, SiCaRiO31 said:

I believe this was deemed unfixable, since the game renders first the atmosphere and then the decals. 

Yep, but there's still hope - I have some control over the decals, so I can achieve a similar effect by mangling the decals themselves. I don't think it will look perfect, but perhaps it will look good enough.

[SiCaRiO31]

7 hours ago, Lisias said:

Yep, but there's still hope - I have some control over the decals, so I can achieve a similar effect by mangling the decals themselves. I don't think it will look perfect, but perhaps it will look good enough.

ohhh. Cant wait

[Lisias]

On 11/30/2021 at 9:01 PM, SiCaRiO31 said:

ohhh. Cant wait

As a matter of fact, neither do I

(not exactly by the feature, but by having time to spend on it - dude, RL™ is bitting.  )

[Alioth81]

I posted this in the scatterer thread but reading the last posts here it is probably Distant Objects that is causing this or the sequence KSP renders the objects (therefore I repost it here)

I am Running the latest scatterer build based on some AVP configs and distant objects.
I just noticed something I never actively saw before. Maybe it was always there but I only notice it now because the atmosphere seems to be larger or it is something that changed. Eve seems to be drawn on top of the atmosphere instead of behind it.

I believe this is similar to what was reported with the comet plume correct?

Can I do something about it?

Maybe instead of black it could be transparent? Would this have any other side effects?

 

(you can also see Moho closer to the sun)

Edited December 28, 2021 by Alioth81

[Lisias]

On 12/28/2021 at 8:07 AM, Alioth81 said:

I posted this in the scatterer thread but reading the last posts here it is probably Distant Objects that is causing this or the sequence KSP renders the objects (therefore I repost it here)

I am Running the latest scatterer build based on some AVP configs and distant objects.
I just noticed something I never actively saw before. Maybe it was always there but I only notice it now because the atmosphere seems to be larger or it is something that changed. Eve seems to be drawn on top of the atmosphere instead of behind it.

Hi!

Yep, it's something on DOE. What happens is that the atmosphere is drawn before the planets and other distant objects (pun not intended), and so any attempt to use a z-index or similar technic is doomed - there's no way to override that.

But...

There's a stunt I think it may work (besides not being perfect). This is the very second github issue for DOE/L by the way! https://github.com/net-lisias-ksp/DistantObject/issues/2

I had planned to use exactly these days between Christmas and New Year to tackle DOE's issues down, but as usual Real Life™ overrun me - I'm currently diagnosing a pretty nasty problem on a critical component for a big customer that it'd expecting the product for the first days of 2022… As soon as I manage to put the day-job on rails (pun not intended again) I will work on DOE again, and this one will be mitigated for sure.

Cheers!

[1FingerSalute]

Thank you for your hard work Lisias.

[Krazy1]

Hi Lisias. One more request... make it work for asteroids. Right now it's showing only a small dot from physics range out to DOE setting for max rendering range. Then it suddenly blinks into full detail at the edge of physics range. With a class A it's probably not too noticeable but a Class E certainly is. It went from about 1 mm diameter white dot to 2 cm full detail as I approached it. 

It might be an extra challenge for new asteroids because they are "unowned" and KSP might not even spawn the details of the asteroid until it first enters physics range... just a guess. 

Thanks and good luck with this and IRL.