[1.8.x] Oh Scrap!- A ScrapYard based Part Failure and Reliability Mod 2.0.1 (07/12/2019)

[eightiesboi]

Hey @severedsolo,

I am getting part failures without notifications. Sometimes I get the notification that a part has failed, and others I get no message, just the highlighted part and the possibility of crashy goodness. Logs are here: https://1drv.ms/u/s!AoyHZiRU1jT-zbdPmwwoaYTI2u38dg?e=RXTksZ

This is not critical; as far as I can tell, the mod is otherwise functioning normally.

Let me know if you need anything else. Thank you!

Update: immediately after experiencing the no notification failure, in the same game and without reloading (although on a different vessel and after changing scenes at least twice) I got a failure WITH notification. Logs are uploaded. The first logs without the parenthetical #1 are immediately after having a failure without notification, the second logs are immediately following a normal (with notification) failure.

Edited September 8, 2019 by eightiesboi
Update with additional logs

[severedsolo]

7 hours ago, eightiesboi said:

Hey @severedsolo,

I am getting part failures without notifications. Sometimes I get the notification that a part has failed, and others I get no message, just the highlighted part and the possibility of crashy goodness. Logs are here: https://1drv.ms/u/s!AoyHZiRU1jT-zbdPmwwoaYTI2u38dg?e=RXTksZ

This is not critical; as far as I can tell, the mod is otherwise functioning normally.

Let me know if you need anything else. Thank you!

Update: immediately after experiencing the no notification failure, in the same game and without reloading (although on a different vessel and after changing scenes at least twice) I got a failure WITH notification. Logs are uploaded. The first logs without the parenthetical #1 are immediately after having a failure without notification, the second logs are immediately following a normal (with notification) failure.

Just a quick note to say I've seen this and will look at when I get the chance (probably Tuesday). Raised #20

Edited September 8, 2019 by severedsolo

[eightiesboi]

Thank you! 

[severedsolo]

On 9/8/2019 at 6:13 AM, eightiesboi said:

Hey @severedsolo,

I am getting part failures without notifications. Sometimes I get the notification that a part has failed, and others I get no message, just the highlighted part and the possibility of crashy goodness. Logs are here: https://1drv.ms/u/s!AoyHZiRU1jT-zbdPmwwoaYTI2u38dg?e=RXTksZ

This is not critical; as far as I can tell, the mod is otherwise functioning normally.

Let me know if you need anything else. Thank you!

Update: immediately after experiencing the no notification failure, in the same game and without reloading (although on a different vessel and after changing scenes at least twice) I got a failure WITH notification. Logs are uploaded. The first logs without the parenthetical #1 are immediately after having a failure without notification, the second logs are immediately following a normal (with notification) failure.

The failure with the notification, was it the same part that failed? There's nothing in your first log to indicate a failure, but what you are describing with the second time, sounds like what would happen the second time a failed part was loaded.

(I'm not doubting it happened. I think I've seen it myself a couple of times, I'm just trying to pin down where to start looking)

Edited September 17, 2019 by severedsolo

[eightiesboi]

9 hours ago, severedsolo said:

The failure with the notification, was it the same part that failed? There's nothing in your first log to indicate a failure, but what you are describing with the second time, sounds like what would happen the second time a failed part was loaded.

(I'm not doubting it happened. I think I've seen it myself a couple of times, I'm just trying to pin down where to start looking)

Unfortunately, I don't remember for certain. I am going to try to recreate it with a separate install to see if I can make it happen. I do know it was early in my restarted career, and I was repeatedly launching the same rocket over and over in order to fulfill contracts and also to enhance part reliability. There weren't a lot of parts on the vessel. And yeah, I took a quick look at the logs before I uploaded them and also didn't see the failure, but I figured that was because I was missing something obvious. I will see if I can make it happen again this afternoon and report back. Also, normally if I were testing a mod I would go with a clean install and only the affected mod and its dependencies, but if you have no objection I am going to use a copy of my same install to see if I can make it happen (or, in the case of the missing notification, not happen) again. If I can, I'll upload the logs and then if you'd like me to start again clean I will.

Thank you for looking into this!

[eightiesboi]

Ok, after a whole bunch of launches with the same vessel, it happened. On launch, one of the SRBs suddenly failed. Red highlight. Interestingly, the logs show that Oh Scrap didn't roll a failure. I'm not playing with any other failure mods. I don't have a clue. Let me know what i can do to help track this down, and thank you!

Logs: https://1drv.ms/u/s!AoyHZiRU1jT-zboxlfo62rLNevINpA?e=u9XcuI

[severedsolo]

4 hours ago, eightiesboi said:

Ok, after a whole bunch of launches with the same vessel, it happened. On launch, one of the SRBs suddenly failed. Red highlight. Interestingly, the logs show that Oh Scrap didn't roll a failure. I'm not playing with any other failure mods. I don't have a clue. Let me know what i can do to help track this down, and thank you!

Logs: https://1drv.ms/u/s!AoyHZiRU1jT-zboxlfo62rLNevINpA?e=u9XcuI

Ohh it was an SRB? Was it an SRB last time too? - that explains alot. SRBs step outside the usual failure process and do their own thing.

[eightiesboi]

Yes, I think it was an SRB each time.

[severedsolo]

1 minute ago, eightiesboi said:

Yes, I think it was an SRB each time.

Thanks, that's probably enough to track it down. I'm doing some work with OhScrap at the moment, so I'll probably get it fixed up soon.

Edited September 18, 2019 by severedsolo

[eightiesboi]

Just now, severedsolo said:

Thanks, that's probably enough to track it down. I'm doing some work with OhScrap at the moment, so I'll probably get it fixed up soon.

Awesome! Please let me know if there is anything I can do to help!

[eightiesboi]

I just had another happen, and I had "One Window" by @DMagic open at the time. The screenshot, as you can see, is just after the failure and crash. According to One Window, there was a message in the top center that said that the SRB had failed to ignite. I may have missed it with the green text against the blue sky and chaotic flaming mess. Lol. However, I didn't get the standard message added to the message queue, and interestingly enough, the log says: 

[LOG 14:15:37.402] [OhScrap]: [OhScrap]: Calculated chance of failure in next 2 minutes is 11%
[LOG 14:15:37.402] [OhScrap]: Failure Chance: 0.02, Rolled: 0.0144548788734036 Succeeded: True
[LOG 14:15:37.402] [OhScrap]: Failure Event! Safety Rating: 9, MET: 0.240000000223517
[LOG 14:15:37.404] [OhScrap]: No parts failed this time. Aborted failure

I respectfully disagree that the failure was aborted. 

I added the new logs and the screenshot to the same folder link: https://1drv.ms/u/s!AoyHZiRU1jT-zboxlfo62rLNevINpA?e=u9XcuI

I hope this helps.

 

[severedsolo]

18 hours ago, eightiesboi said:

I just had another happen, and I had "One Window" by @DMagic open at the time. The screenshot, as you can see, is just after the failure and crash. According to One Window, there was a message in the top center that said that the SRB had failed to ignite. I may have missed it with the green text against the blue sky and chaotic flaming mess. Lol. However, I didn't get the standard message added to the message queue, and interestingly enough, the log says: 

[LOG 14:15:37.402] [OhScrap]: [OhScrap]: Calculated chance of failure in next 2 minutes is 11%
[LOG 14:15:37.402] [OhScrap]: Failure Chance: 0.02, Rolled: 0.0144548788734036 Succeeded: True
[LOG 14:15:37.402] [OhScrap]: Failure Event! Safety Rating: 9, MET: 0.240000000223517
[LOG 14:15:37.404] [OhScrap]: No parts failed this time. Aborted failure

I respectfully disagree that the failure was aborted. 

I added the new logs and the screenshot to the same folder link: https://1drv.ms/u/s!AoyHZiRU1jT-zboxlfo62rLNevINpA?e=u9XcuI

I hope this helps.

 

Fixed in Dev. The messages about rolling failures / failures being aborted don't apply to SRB's. They do a one time check on launch, rather than being thrown into the mix with everything else.

The ScreenMessage is working correctly, but you are right that it wasn't posting to the messaging app. That's now fixed, so will be out with the next update

[eightiesboi]

1 minute ago, severedsolo said:

Fixed in Dev. The messages about rolling failures / failures being aborted don't apply to SRB's. They do a one time check on launch, rather than being thrown into the mix with everything else.

The ScreenMessage is working correctly, but you are right that it wasn't posting to the messaging app. That's now fixed, so will be out with the next update

Thank you so much! 

[severedsolo]

Oh Scrap 1.7 Released

• Fixed RCS thrusters being allowed to fail when RCS is turned on
• Parachutes will now only fail while deployed
• Fixed an NRE that could occur when other mods apply MM patches after the Failure Module has attached itself to a part.
• Failure Modules will now cache all modules they need to interact with, instead of grabbing them on each FailPart() call.
• timeBetweenChecksRocketsSpace has now been split into timeBetweenChecksLocalSpace (Homeworld or its Moons) and timeBetweenChecksDeepSpace (everywhere else)
• timeBetweenChecksLocalSpace reduced to 20 minutes
• timeBetweenChecksDeepSpace increased to every roughly every 7.5 hours.
• Fixed an issue where SRBs would not post failures to the Messages App or Log they had happened.

[eightiesboi]

Hey @severedsolo!

I think I have another bug.... I had an SRB failure and I received a message from Oh Scrap! Well, technically, I received 1600 messages from Oh Scrap! before i managed to get  out of the flight scene. Logs here: https://1drv.ms/u/s!AoyHZiRU1jT-zboxlfo62rLNevINpA?e=mOy6KG

On the bright side, I *did* get what I asked for: messages!

[severedsolo]

10 hours ago, eightiesboi said:

Hey @severedsolo!

I think I have another bug.... I had an SRB failure and I received a message from Oh Scrap! Well, technically, I received 1600 messages from Oh Scrap! before i managed to get  out of the flight scene. Logs here: https://1drv.ms/u/s!AoyHZiRU1jT-zboxlfo62rLNevINpA?e=mOy6KG

On the bright side, I *did* get what I asked for: messages!

XD One day I'll learn to test things before I deploy. Even if they are simple fixes. I may have accidentally put that code outside the if statement that's supposed to control when it's sent. Sorry about that.

Oh Scrap 1.7.1 Released

• Fixed SRBs constantly spamming the messaging app on failure.

 

 

[eightiesboi]

It's all good! I was actually ROTFL when it happened. Thank you for the quick fix!

[eightiesboi]

Hey @severedsolo, I was thinking about Oh Scrap! when a rescue mission with an excellent safety rating and very mature generation fuel tanks suffered a leak. I wrote the following while I was walking this morning, so it was done without reference to the forum or the game. I think this is an awesome mod as-is, but I was pondering what enhancements or changes I think would be interesting. If you get any good ideas (or bad ones) from this, that would be terrific, but if nothing else, I just want to point out that you've written code that kept me typing on my phone during a 6-mile walk. 

Spoiler

Thoughts on Oh Scrap:

• Do more complex vessels (i.e. those with more parts) have a greater likelihood of failure, assuming all parts are same generation? That is, is the failure check per vessel or per part?
  • If per vessel, a consequence of this is that more complex vessels potentially have more redundancies and are less likely to experience a catastrophic failure. Simple vessels are more likely to experience catastrophic failure by virtue of having fewer parts to fail. This might go against real world statistical behavior, in that more complex systems have more potential failure points, which is the reason why redundancies are built into them.
  • If per part, then more complex vessels would have a higher potential failure rate, which might better reflect real life, but may also be less fun / more frustrating.
  • Ideally, a combination of the two would occur. First, the system would check to see if there was a potential failure. If the check was successful, nothing would fail and it would move on. If the check was failed, a part by part check would occur to determine what failed. If all parts individually passed, the failure would be negated (but see below). More complex vessels would therefore be no more prone to failure than simple ones on the first check, but more prone on the second.
    • It would be neat if, when a failure was indicated on the per vessel but all parts passed, there was a Master Caution alarm to let the player know they'd dodged a bullet!
• Severity: currently, when a failure occurs (especially a tank failure), the player has little time to notice and react. Furthermore, trying to use the context menu UI to quickly transfer resources from the failing tank to a spare can be difficult in the best of times. And if the tank is a small one, a small delay in reaction time could have large consequences. It would be interesting if the failures then ranged in severity. For instance, a tank leak might range from a blowout (all resources instantly lost) to a slow trickle. Untested and newly researched parts might have a greater likelihood of severe failures, whereas late generation parts might tend towards minor failures, if any.
• Damage mitigation: currently there is only one truly viable technique for damage mitigation during a failure, and that is rapidly transferring resources (where possible). EVA for repairs during a burn can be difficult and usually takes too long to do any good. And while an engineer can effect repairs after the fact, in the case of an emptied tank it's a bit of closing the barn door after the horses have gone. However, what if you had the option to isolate a tank, stopping the resource drain but also disconnecting it entirely from the vessel and disallowing the transfer of resources. This action would not easily be reversed: a level zero kerbal could reopen the isolation valve, but not repair the tank (giving you more time to prepare a resource transfer but that's about it), but it would take a level one engineer (or perhaps higher) to repair the tank (changed from the current system). This would allow for quick mitigation of a tank failure (short of a blowout) without eliminating the negative effects and still retaining consequences.
• Cascade failures: one more reason to isolate a failing tank could be that a tank that is spewing its contents should make it more likely for nearby components to fail. The only problem I see in implementing this is that it may be difficult to turn off the increased likelihood of adjacent component failure once the leaking tank is empty. But if that could be implemented, it would give a player a good reason to shut the isolation valve on a failed tank.
• Auxiliary resources: KSP allows you to switch off a resource, but rarely have I found a use for this feature in-game. However, what if switched off resources were immune to failure checks while switched off? You could therefore have an auxiliary oxygen supply, for example, that you know would be safe from failure (until, of course, you switched it on!). 

 

[eightiesboi]

On 8/16/2019 at 10:59 PM, severedsolo said:

I'm going to assume by "a couple of generations" you mean new builds? (not applying parts from the ScrapYard inventory).

Parts will get gradually better with each new generation, by the time you reach Gen 10 there is no need to run tests, as they will have reached the minimum failure level.

Reusing your rocket is the way you are supposed to "test" them. After a successful test (ie, you use the rocket, and then use the same parts from the ScrapYard inventory again after recovery) - the parts will get the bonus for being tested. A part is at it's most reliable on it's second use, after that the reliability will gradually drop off again as it starts to wear out after multiple uses (but usually they have to get to at least 6 uses before they can be considered unreliable)

Hey!

How often should SRBs be failing? My Hammers are at least generation 70, yet whether they are new or tested I am getting about one failure for every six that successfully launch. 

Logs here: https://1drv.ms/u/s!AoyHZiRU1jT-zboxlfo62rLNevINpA?e=3gK7ih

Note that around line 161842 in the output log or 1028 hrs in KSP log, there are a lot of messages regarding the single SRB failure.

Let me know if it's me or a bug or something in between, and thank you!

[severedsolo]

2 hours ago, eightiesboi said:

Hey!

How often should SRBs be failing? My Hammers are at least generation 70, yet whether they are new or tested I am getting about one failure for every six that successfully launch. 

Logs here: https://1drv.ms/u/s!AoyHZiRU1jT-zboxlfo62rLNevINpA?e=3gK7ih

Note that around line 161842 in the output log or 1028 hrs in KSP log, there are a lot of messages regarding the single SRB failure.

Let me know if it's me or a bug or something in between, and thank you!

At max level they should be 1/20 roughly. The log spam is a separate issue. It's not more than one failure (as far as I can tell).

Can I get a (stockish) craft file that manifests the issue please.

 

[eightiesboi]

30 minutes ago, severedsolo said:

At max level they should be 1/20 roughly. The log spam is a separate issue. It's not more than one failure (as far as I can tell).

Can I get a (stockish) craft file that manifests the issue please.

 

Let me see what I can come up with, although here's a question: is it 1/20 per SRB or per vessel? If it's the former, it may be observation bias on my part.

I uploaded a test craft that has suffered more than a few explosions on launch. Stock + Making History + Universal Storage II + Cryogenic Engines (it also had ScanSat and Aviation Lights, but I removed them.) It's in the same folder as my logs.

Thank you!

PS - You've mentioned before that SRBs go through a different failure calculation process. Since I don't know what that process is, what follows is total speculation. It feels (and I hate relying on feelings, but here we go) like the more SRBs, the greater the failure risk (which makes sense) but that the failure rate rises faster than I would expect. Could it be that the mechanism that determines SRB failures is multiplying the chance of failure by the number of SRBs, instead of treating each SRB separately? That is, if there is a 5% chance of failure for each SRB, then each 5% chance should be "rolled" independently; however, if the number of parts were first multiplied by the failure chance, you would end up with a 20% chance of failure, which would be incorrect. I admit, I haven't played with statistics since grad school, but if my wild speculation were correct, that might account for my observations.

Or I could be wrong.

Edited October 12, 2019 by eightiesboi
Stray thoughts

[severedsolo]

8 hours ago, eightiesboi said:

Could it be that the mechanism that determines SRB failures is multiplying the chance of failure by the number of SRBs, instead of treating each SRB separately?

They are rolled separately, but you are half right, it's not that it's being multiplied by the number of SRBs, it's just that's how probability works

I'll use that craft you provided as an example. 4 SRBs each with a 5% chance of failure.

We have a 5% chance of failure, but 4 possible failure points, every failure point is rolled individually.

Chance of a failure not occurring at each point: 0.95

4 Failure points means that the chances of no failure at all occurring is: 0.95 ^ 4 = 0.81450625

In other words, we have a 81.4% chance that the launch will go well. Meaning there is a 18.6% chance that one of the failures will happen.

100/18.6 means that on average a failure will occur every 5.37 launches. Which is pretty darn close to your own observation.

So, I need to do something about SRBs. Quite what I'm not sure yet, but SRBs should not be orders of magnitude less reliable than their liquid fuel counterparts, so that's a bug.

For your part: you can turn down the "baseFailureRate" in the MM config for now to compensate.

[eightiesboi]

8 hours ago, severedsolo said:

They are rolled separately, but you are half right, it's not that it's being multiplied by the number of SRBs, it's just that's how probability works

I'll use that craft you provided as an example. 4 SRBs each with a 5% chance of failure.

We have a 5% chance of failure, but 4 possible failure points, every failure point is rolled individually.

Chance of a failure not occurring at each point: 0.95

4 Failure points means that the chances of no failure at all occurring is: 0.95 ^ 4 = 0.81450625

In other words, we have a 81.4% chance that the launch will go well. Meaning there is a 18.6% chance that one of the failures will happen.

100/18.6 means that on average a failure will occur every 5.37 launches. Which is pretty darn close to your own observation.

So, I need to do something about SRBs. Quite what I'm not sure yet, but SRBs should not be orders of magnitude less reliable than their liquid fuel counterparts, so that's a bug.

For your part: you can turn down the "baseFailureRate" in the MM config for now to compensate.

I *did* say my stats was rusty... lol

But I do think that an 18% chance of failure on launch with four SRBs is a little high... And, if I understand your intended modeling, I don't think that's right. Shouldn't the probability of failure be impacted by the overall failure of the vessel? For parts other than SRBs (as I understand it), there are two checks rolled: vessel failure, and then part failure. If the vessel passes, no part rolls. If the vessel fails, all the parts roll until either one fails or they all succeed. If the SRBs are rolling independently--without the vessel failure roll--then the probability that one of them will fail is both independent of the overall vessel AND is greater the number of SRBs on the vessel. With all of the other parts (again, based on my admittedly feeble understanding), the probability that one of them will fail is conditioned on the probability of failure for the overall vessel (i.e., the chance that a part will fail is irrelevant if the vessel passes its failure roll). If the overall chance that a vessel will experience a failure is 5%, and each part has an independent 5% chance of failure, and there are four parts that could fail, isn't the chance that a failure will occur closer to 1%?

Again, I haven't done statistics in over a decade, and my memories of HLM and MLM have long faded, so forgive me if I am totally wrong.

[severedsolo]

3 minutes ago, eightiesboi said:

I *did* say my stats was rusty... lol

But I do think that an 18% chance of failure on launch with four SRBs is a little high... And, if I understand your intended modeling, I don't think that's right. Shouldn't the probability of failure be impacted by the overall failure of the vessel? For parts other than SRBs (as I understand it), there are two checks rolled: vessel failure, and then part failure. If the vessel passes, no part rolls. If the vessel fails, all the parts roll until either one fails or they all succeed. If the SRBs are rolling independently--without the vessel failure roll--then the probability that one of them will fail is both independent of the overall vessel AND is greater the number of SRBs on the vessel. With all of the other parts (again, based on my admittedly feeble understanding), the probability that one of them will fail is conditioned on the probability of failure for the overall vessel (i.e., the chance that a part will fail is irrelevant if the vessel passes its failure roll). If the overall chance that a vessel will experience a failure is 5%, and each part has an independent 5% chance of failure, and there are four parts that could fail, isn't the chance that a failure will occur closer to 1%?

Again, I haven't done statistics in over a decade, and my memories of HLM and MLM have long faded, so forgive me if I am totally wrong.

Nah, you're right. SRBs are the one part that still use the old model of rolling individually (and this demonstrates precisely why I went away from that model, despite complaints from some users). I 100% agree with you, 18% is way too high, it's just this is the first time I've actually sat down and done the maths on it. When I got the answer (and double checked my work) I was, surprised.

SRBs don't use that model because the chance of a failure becomes vanishingly small, but I didn't take into account multiple SRBs.

So assuming I changed it to work exactly the same way that other parts work (calculations in the spoiler)
 

Spoiler

Chance of Event A (Overall vessel failure roll) = 0.01

Change of Event B (individual part failure) = 0.05

Chance of both events occuring: A*B = 0.01*0.05=0.0005

Chance of no event occuring: 0.9995

Assume 4 potential failure points (4 SRBs): 0.9995^4 = 0.998(rounded)

 

Chance of failure: 0.002%.

That's.... unacceptably low tbh. It works for other parts because they have multiple failure points (get rolled often) - but for SRBs I need to do something different.

What that's going to be I don't know yet but you are right that the current model is just as bad but the other way.

[eightiesboi]

17 minutes ago, severedsolo said:

Nah, you're right. SRBs are the one part that still use the old model of rolling individually (and this demonstrates precisely why I went away from that model, despite complaints from some users). I 100% agree with you, 18% is way too high, it's just this is the first time I've actually sat down and done the maths on it. When I got the answer (and double checked my work) I was, surprised.

SRBs don't use that model because the chance of a failure becomes vanishingly small, but I didn't take into account multiple SRBs.

So assuming I changed it to work exactly the same way that other parts work (calculations in the spoiler)
 

  Hide contents

Chance of Event A (Overall vessel failure roll) = 0.01

Change of Event B (individual part failure) = 0.05

Chance of both events occuring: A*B = 0.01*0.05=0.0005

Chance of no event occuring: 0.9995

Assume 4 potential failure points (4 SRBs): 0.9995^4 = 0.998(rounded)

 

Chance of failure: 0.002%.

That's.... unacceptably low tbh. It works for other parts because they have multiple failure points (get rolled often) - but for SRBs I need to do something different.

What that's going to be I don't know yet but you are right that the current model is just as bad but the other way.

My professor would be proud of me. 

Thank you for looking into this! I've had so many interesting (in many senses of the word) missions because of the feeling of realism (and real stakes) that this mod brings to the game. It's why I set up real abort sequences and use a LES.