[1.12.4] ScrapYard (SYD) The Common Part Inventory - v2.2.99.0-prerelease `<Project Zelda II>` edition [08 Jan 2023]

[zer0Kerbal]

I have been watching, and until now there hasn't been news or a reason for me to comment.

... Demonstrated to me beyond a shadow of doubt that:

• part(s) of the issue is the KSP181-OnReRoot issue.... which doesn't seem to have an easy solution.
  • special shout out to @Lisias for not only discovering, but reporting and suggesting the fix! This bug was undiscovered/unreported from 1.8.1 to now.
  • KSPCF has rolled back the changes made in KSP1.8.1 that only seemed to aggravate the situation
  • however this fix-on-a-fix doesn't resolve the underlying root cause bug that still exists (AFAIK/U); still it is progress.
• part(s) of the issue might be a race condition where two or more things change/access a part at the same time for the same vessel.
• part(s) of the issue remain undisclosed because more research is needed to not only track down the root cause, but attempt to solve them 

SYD might have one issue (PersistentID) - because it's code was changed to be a good plugin and do things the way KSP wanted; was a bad move so the code will be changing back to using internal tracking instead of the KSP. Additional 'bug's might crawl out during the updating/debugging process - but this is not very likely until the actual problem can be ferreted out and resolved. It has been suggested internally that we could just "shut off" the logging of this issue, but that is like taking the batteries out of your fire/smoke/radon detectors.... the silence might be permanent. 

All other issues are extrinsic to ScrapYard.dll - maybe not KSPCF as had originally thought, but are not being caused by ScrapYard.dll; rather ScrapYard.dll is the messenger, or rather the canary in the coal mine.

We (the developers) of ScrapYard are working toward a new version, eta unknown at this point because this Quest for the Flowgraph Grail has lead us to discover the ReRoot (and now other) heretofor undiscovered (or undeclared) bugs in KSP (and others). We want to do this the right way - and not slap a beautiful patch of spaceTape on a big sucking bug.

Bug reports are very helpful, and should be directed to GitHub (https://github.com/zer0Kerbal/ScrapYard/issues) and FlowGraph/Parts not applying  reports here (https://github.com/zer0Kerbal/ScrapYard/issues/81)

and just a word about OhScrap! We still intend to release an update shortly. (soon™ there is a long list of released waving at me for attention) 

23 hours ago, Richmountain112 said:

When I assign a reused part to an action group it automatically uses a new one instead of reusing the one in the inventory.

I'd like it to ignore action groups created in the VAB. I don't know about in-flight edits to action groups.

 @Richmountain112 +1

Interesting idea worth considering - kindly make a suggestion here (https://github.com/zer0Kerbal/ScrapYard/issues/new?assignees=zer0Kerbal&labels=type%3A+request%2Ctype%3A+discussion&projects=&template=Feature_Request.yml&title=[Request]%3A+)

news when there is news

 

 

13 minutes ago, Lisias said:

There's no doubt that SYD is going to be changed. But without understanding why, you will just move the problem to another place and then someone will have to diagnose this problem again.

Until I'm convinced that changing the persistentId is really the source of the problem with reproducible evidences, I will keep advocating for looking the problem somewhere else in the SYD's code. AND AGAIN, I'M NOT ADVOCATING TO WORK AROUND IT SOMEWHERE ELSE, I want to understand what's happening and why.. 

And I was candidly ignoring a detail about this problem: exactly what happens when the LogSpam is triggered? What the real side effects to the game? Memory Leaks? Performance Issues? Savegame corruption?

Until this moment, I was focusing on the symptom, because I agree we should be aware about what is happening and why.

But if I create a Log filter on SYD and just omit the LogSpam from being written on the KSP.log (what would be a <piiiiii> move, to make it clear, and I do not condone it), exactly what would be the consequences?

Because, you see, this may be another case of false alarm, in the same sense that damned ADDON BINDER ERROR that happens when you merely probe if an Assembly is loaded or not.

well said.... and what I was referring to (well, mostly 

and thank you (so far) to all those who have posted concerning this issue.

Edited June 24, 2023 by zer0Kerbal
update links and lasso the birds

[Lisias]

32 minutes ago, zer0Kerbal said:

SYD might have one issue (PersistentID) - because it's code was changed to be a good plugin and do things the way KSP wanted; was a bad move so the code will be changing back to using internal tracking instead of the KSP.

By doing that you will create new issues on SYD, because you will loose the track of the part as it have it's id "changed" by a reason or another.

You need a new tracking method that would detect when the part was cloned and "renamed", otherwise your plugin will be rendered useless when there're living crafts from the same craft file flying around.

I'm not making such a fuss just because I'm an idiot, I'm doing it besides being an idiot (because, frankly, only an idiot would be dealing with some <insert your non forum compliant expletive here> "collaborators" instead of doing something else).

Edited June 24, 2023 by Lisias
Better phrasing

[TruePikachu]

19 hours ago, Lisias said:

THAT specific log. There're some more to compare with.

I was referring to that specific log because your description that accompanied the log implied the log was showing that the spam wasn't related to the persistentId, whereas I was indicating that conclusion is...errr...inconclusive. Also, the only 1.12.x log from your link that has any match for the regex /FlowGraph/ is the one from this comment, and it doesn't seem to have persistentId writes instrumented.

19 hours ago, Lisias said:

FOUR times the persistentID was changed, and no LogSpam was triggered, on the same log.

Just to note, looking at that log excerpt, something funky looks to be going on on behalf of whatever is setting the persistentId of four different parts all to the same value. While I'd probably call it a potentially-invalid test, I don't know enough about the actual trigger for the FlowGraph message to say for sure (e.g. I don't know if it's keying a set based on identity or by persistentId, nor if it's silently discarding duplicates by the key when added to the set).

19 hours ago, Lisias said:

Interesting, from my point of view it looks like you are cherry picking the pieces of logs I post, and completely ignoring that I posted the logs where the LogSpam happened, between a lot of logs where it didn't (and told about).

The big problem is that the spam hasn't been reliably triggered by everybody for a yet-unknown reason; for instance, @zer0Kerbal doesn't appear to have had it much if at all while I've had it happen reliably. As such, the lack of FlowGraph errors in a test session isn't conclusive evidence for the change being tested being an actual fix for FlowGraph errors, but the presence of FlowGraph errors indicates that something is still wrong. This is probably my least favorite thing about the bug altogether: I'm able to reliably reproduce it, but I have neither the experience with KSP's internals nor the time to put together and learn a modding environment in order to experiment with it, all for a mod I haven't been able to use (and therefore personally gauge how valuable fixing the bug would be) because of the bug creating issues in my installs. All I can really help with is philosophical views from my own experience with bug testing, and even that's starting to stretch my patience pretty thin. Also a testing environment for already-compiled instrumented plugins, though with the rising aggression I'm not sure I can offer that anymore for my own safety.

Edited June 25, 2023 by TruePikachu

[Richmountain112]

5 hours ago, zer0Kerbal said:

On 6/23/2023 at 6:00 PM, Richmountain112 said:

When I assign a reused part to an action group it automatically uses a new one instead of reusing the one in the inventory.

I'd like it to ignore action groups created in the VAB. I don't know about in-flight edits to action groups.

 @Richmountain112 +1

Interesting idea worth considering - kindly make a suggestion here (https://github.com/zer0Kerbal/ScrapYard/issues/new?assignees=zer0Kerbal&labels=type%3A+request%2Ctype%3A+discussion&projects=&template=Feature_Request.yml&title=[Request]%3A+)

It's no big issue for me.

But for someone with Oh, Scrap, it could get worse (but not by much). And for Kerbal Construction Time, it could make missions much quicker if action groups can be edited between flights.

Edited June 25, 2023 by Richmountain112

[TheAnonymous]

Hi, I downloaded this mod a long time ago in tandem with Kerbal Construction Time but lately I noticed that the Quick Apply button isn't working. Is there a fix for it? Also what else does this mod do? thanks

[RiceCakes85]

I don't know if the issues over the last several pages are still being tackled.  I stumbled upon the issue early in my save game and plunged down this rabbit hole.  I want to express my thanks to the mod developers.  You all are amazing.

[sjsharks39]

Hi Friends, what's the status of the discussions back in June? I would be curious about using this mod for an upcoming playthrough but having a hard time understanding the impact of the persistentID issue. Regardless of it being fixed or not, what did that actually cause to the player?

[OldMold]

1 hour ago, sjsharks39 said:

Hi Friends, what's the status of the discussions back in June? I would be curious about using this mod for an upcoming playthrough but having a hard time understanding the impact of the persistentID issue. Regardless of it being fixed or not, what did that actually cause to the player?

Lots and lots of logspam - game starts slowing to a crawl because of all the errors.

edit: +1 to patiently waiting and hoping this mod gets fixed someday. It made playing with the construction time mod interesting because saving stages helped reduce build times and re-use them. Also was great for reusing recovered spaceplanes in quick turnarounds. 

Edited October 12, 2023 by OldMold
added voice of support

[roxik0]

Hi Poeple  
So I tried to attack Flow Graph error.. or maybe better apply error   And I have one think to test if it works then I will put PR so Zer0Kerbal will be able to add this to release(and fix what I miss eventually).  

ScrapYard DLL

Expected Results:

- Flow Graph still occurs (But IMHO it's not a problem of ScrapYard it simply consequence of run event (OnEditorShipModified) with same ship).
- However your parts should be taken from inventory correctly. 

 

[Bimo1D]

Heyo, here is russian localisation for ur mod

[Ker Ball One]

On 10/11/2023 at 7:51 PM, OldMold said:

Lots and lots of logspam - game starts slowing to a crawl because of all the errors.

edit: +1 to patiently waiting and hoping this mod gets fixed someday. It made playing with the construction time mod interesting because saving stages helped reduce build times and re-use them. Also was great for reusing recovered spaceplanes in quick turnarounds. 

I haven't looked at logs, but I had to immediately uninstall SYD because every second it would lag in the VAB editor.  Made dragging parts difficult.

I assume this is the same problem.

Anything tracked as open on the GitHub?

 

[corbeau217]

@ the humans working on flowgraph error:
i have an idea for what the flowgraph lag might be caused but i dont know c# so cant verify (or how it handles under the hood other than generalist/from c/c++/java knowledge)

i think the lag might be due to the time it takes to perform I/O reads and writes, and waiting for locks on things like the file system or output streams

if it behaves in the way i think it is, every time there's a write for logging (kinda symptom now, but like "already exists in flowgraph" [which might be able to work with sorting and or using redblack trees or vertex cover bc it might be using struts for circles and then it's like network graphs, you gotta time to live or something, or modified dijkstra (i cant find the thing now and illegal to post screenshots of the lecture it was in at uni rip), or maybe it was bellman-ford idr ] might just be cycles or references to part A exist in child B's child which is called C and B might have C and D as a child, and D might have A as a child? [maybe the stronger joints mods do something with that? but i dont imagine LGG codes like that by the sheer number of mods they manage and what that implies about compsci knowledge]  )

but uhhh, mostly, (idk how familiar u r with logging/output streams but i love them a lot) they usually have a lock they gotta wait for access to write, and if multiple readers they gotta tell them to get lost with their lock before they can write. otherwise u get into spooky land. i think there's some goofiness with it in unity but i dont imagine it's much different from a standard System.out.println/printf/cout call. any time i use the console for displaying stuff or providing info it gets substantially slower just by it needing to do all the legwork of requesting access to the console and writing to it then letting go of access each time to make sure someone else isnt

i heard on another post that ksp is a bit strange with multithreading tho and that it's likely all singly threaded in logic? but idk if that's still the case, i keep forgetting to check dates. also idk if that means ksp burst is just somewhat multithreading or if it's all smoke and mirrors and we're just overworking the poor thread like that 1 network admin at a regional school
but uh, each thread could be trying to add the "next" item and end up trying to add one that already exists, and rather than just giving up straight away bc it's understandable with race conditions, it's hanging around and shouting all like "im leabing this gronp" and taking up resources/locking up that poor overworked thread with asking it about what fonts to use in a powerpoint presentation (guilty :') )
(i gotta be honest, im still not sure who made the flowgraph or who's playing around with it)

the other idea was if in the back-upping of ships that various mods im using are doing, maybe scrapyard too? i noticed a settings about parts and memory blurry, but it might be keeping those backups or references in an environment that the flowgraph algorithm can access so it's ransacking those references without it being the intended outcome. if C# has its own garbage collector (as implied by all the people saying it's microsoft take on java), then it might be straight up accessing the list of current active part references and then just linear search for ones that are part of the current ship, then checking if it's an option to traverse to, then if it's better to traverse to then trying for its descendants (re-doing the linear search :'), O(bruh) complexity)
now that i think about it i think scrapyard was trying to access an inventory for parts, but im still not sure on how that works but there was mention in difficulty settings about pricing of part selection and (i think) something about testing for if it could make the ship cost cheaper, and the interval for testing those things?


there's also the chance it's just a symptom of some rotten code smell buried in the unity game engine's asset tree structure and its navigation (again, no hablo C#/unity, but i've been accumulating resources so i can when i have free time. probably after doing computer graphics/distributed systems this semester, or after completing the intro gamedev units that all the C#/unity units are gated behind at my uni. absolute pain, but i get it )

uhh tangents there a few times, apologies if it's hard to read. i havent slept much lately, bc uni starts back in <2 weeks and i need to fix sleep schedule before then
vry vry vry hope this helps, i love your work and i love the mod and what some of those settings imply. i keep filling pages with brainstorming ideas of mods and then finding scrapyard/mks/oks(/wolf?)/konstruction/kis/usi does it already or is working on it lmao

tl;dr im mostly certain it's to do with time waiting for access to locks bc of an interaction with common concurrently used mods that all try to fix a thing or have different methods of fixing the thing based off assumptions of code state from last time the maker looked at the other mod (this sounds kinda finger pointy and like a water is wet but i dont mean it like that. u all have lives other than modding and cant all be 0-bug-hyperdimensional-beings like we want u to be)

[corbeau217]

ah rip, i dont think it was very clear after reading that back and ill forget in the morning what i was even on about

i mean console and logging and file system access interchangeably here since in ksp all three of those are linked (now that i say that, i wonder if it's still accessing all 3, which is 3 times the lock wait/free of a the naive approach to just directly `fprint(gameLogFsPtr,"[%s]: %s\n",System.getCurrentTime().formatAsUTC(), loggableContentStr);` or w/e it is in c#. also that kiinda points out to me that there's probably another lock again in accessing the current time which happens every update of anything at all. also soz for the java/c mix, ill rabbit hole if i google the C method for it)

(had to send as another post bc i am still <5 approved and ill forget in the morning bc i overloaded brain today. thank you mods for approving this one too )



EDIT: now I have the 5 approved priveledge and sleep, yet still remember: immediately after posting this last night, I was playing ksp and noticed the behaviour of the dv calculator from stock staging with parts. I kinda messed with it in the past a bit, and it was very clear to me that it wasnt removing deleted parts from the list of parts it was including in vessel mass portion of the dv calc

I'm wondering if this is because when you click to have one on your mouse, scrapyard (or adjacent mod that plays with parts and part selection) is adding it into `theRealPartList` and then when you try to delete by clicking in the part selector window to get the delete sound and have it leave your cursor, it's:
1. empties fuel
2. keeps the part in some local space for reference by mod that has a part inventory system
3. dv calc is using the same method as flowgraph, and can access these parts when trying to get the mass of current vessel (by this logic, other squad code might also be doing this for rendering and ship files etc)

KER installed at the same time seems to not see these hypothetical extra loitering references so must be just accessing the list of parts in the visible workspace? maybe the visible workspace for rendering is seperate to the purchasedPartList/theRealPartList which is some list or or array or tree in the background replacing what squad code expects to exist

maybe squad code actually seperated these out themself, so that you could get multicore behaviour with rendering and that it only links to the actual stuff if there's a change? could explain some of the krakens too (like as in, you have some list, you tell the gpu the list and it's like "on it boss", and renders it and handles the physics then if a part calls its "onDestroy" or maybe "onExplode" [maybe there's more than one with different behaviours] it'll then tell the part list to update whichever part was deaded? )

vry theory about it all. hope this helps c:

Edited February 9 by corbeau217
another post icky and i remembered some more stuff

[CubertFarnsworth]

On 7/14/2023 at 9:48 PM, TheAnonymous said:

Hi, I downloaded this mod a long time ago in tandem with Kerbal Construction Time but lately I noticed that the Quick Apply button isn't working. Is there a fix for it? Also what else does this mod do? thanks

@zer0Kerbal

yeah same. D-loaded this mod recently. Love the idea, have to force the used parts since Quick Apply does not work. Also if you remove an upper section and place it back, all associated used parts with that section are reset to "New Parts". 

--edit: Does anyone know how to identify in the SPH or VAB which parts are damaged to replace or how to repair them. I have to sell and rebuild a new plane every time since they are not repaired in the hangar. Defeats the whole point of recovery and not having to rebuild/pay for a new plane. ---

But huge thank yous to the modders. 

Edited April 10 by CubertFarnsworth
another "bug" added