Boeing 7*7: the saga continues…

[Nightside]

NY Times Link

Looks like this crash was caused by an automatic correction executed by Boeing’s autopilot based on data from a bad angle of attack sensor.Could a better program have prevented the outcome?

Update:

I'm reviving this thread after reading 2 fascinating articles this week about how Boeing's shift over the last couple decades from a company that valued it's engineers and had a world renowned culture of innovation and safety became one that appears to be driven by greed alone. Although the victims of the 737 Max planes operated by Lion Air and Ethiopian paid the price, Boeing's focus on money over good product also has ramifications for their spaceflight projects as well.

https://www.theatlantic.com/ideas/archive/2019/11/how-boeing-lost-its-bearings/602188/

https://www.newyorker.com/magazine/2019/11/18/the-case-against-boeing

And, not specifically related to Boeing or the 737 Max, but an interesting take on how adding complicated safety features to complex systems can have disastrous consequences.  

https://99percentinvisible.org/episode/cautionary-tales/

 

 

Edited August 23 by Nightside
New links

[p1t1o]

38 minutes ago, Nightside said:

NY Times Link

Looks like this crash was caused by an automatic correction executed by Boeing’s autopilot based on data from a bad angle of attack sensor.

Could a better program have prevented the outcome?

Do you mean, could a better program have compensated for a bad sensor?

Only if it knew it was bad, which would require another sensor. If that sensor is bad, then we are back at the beginning.

***

Was chatting with someone about this the other day and the 737 MAX is not fly-by-wire (not fully anyway, spoilers are FBW). In other words, the autopilot does not control the aerodynamic surfaces, it controls the stick/yoke in the cockpit, which controls the surfaces. So if the autopilot gives bad commands, at any time the pilot can just grab the control stick/yoke to override.

This makes me think that the "aggressive" or "abrupt" dive (as described by boeing) was SO aggressive that the pilots were prevented from taking this simple remedial action.

And thats quite a violent error. Unless Im missing some key piece of information.

[StrandedonEarth]

A tv news report I saw suggested that properly trained pilots would have overridden or taken over from the autopilot. OTOH, a critical sensor like that should have redundancy, or the computer should have been programmed to use other data (altitude, airspeed, climb rate) to determine that the sensor reading was erroneous. 

[wizzlebippi]

The 737 max is equipped with a stick pusher designed to keep the aircraft from encountering full aerodynamic stall. Based on the articles I've read, the aircraft had problematic airspeed indications, implying that Boeing can fire their pusher at a minimum airspeed. As a transport category aircraft, the 737 max should have at least 3 air data sources, allowing a malfunctioning system to be voted offline.

There should be a red override/disconnect button on both pilot's controls that if held would stop the pusher from firing. The aircraft may have been very nose down once the pusher was overridden, and the crew may have not recovered properly (throttles idle, speedbrakes extended, pull to 1.5-2g).

Edited November 9, 2018 by wizzlebippi
Editorial

[MaverickSawyer]

I, for one, am waiting for more detail than mindless and inexpert rambling by media "experts".

 

I also have a classmate who actually works on the 737MAX assembly line, and he has promised to keep the class in the loop on anything he hears. Right now, that's not much, as the investigation is still very early, and the FDRs and CVRs haven't been fully analyzed.

[kerbiloid]

One accident caused by a software mistake per how many ones caused by human malfunctions?

[Kerbart]

If the problem is with the inputs then fixing that should fix it. Software that compensates for faulty sensors is undoubtedly more complex. Complexity favors errors. You’ll probably introduce other errors by fixing the software. Fix the sensors (redundancy?) instead (less error sensitive, I mean) is what I would think is the way to go.

[YNM]

5 hours ago, Nightside said:

automatic correction executed by Boeing’s autopilot based on data from a bad angle of attack sensor.

Ah. As expected, electrical problems.

 

We've seen 2 patterns :

- QZ 8501 : Rudder traveler limit failure due to solder cracking.

- JT 610 : Bad sensor.

 

I don't know, but is it possible that electrical systems reeaally don't like warm, humid tropical climate ? Would it have any effect on the airplane system electrification progress ? (I really hope there'll be as in the past Kapton was found to be a really bad insulator material in warm, humid climate.)

The plane (PK-LQP) did have a problem before when flying from Denpasar (Bali) to Jakarta - Passengers had to disembark after the first boarding attempt as the engines seem to have trouble starting, then on the second attempt it did fly, but the aircon was not cooled and some said it was flying lower and slower than usual.

Rumours had it that the plane shown different values in the main display between left hand side and right hand side, also the flight before was flown entirely manual.

Edited November 9, 2018 by YNM

[wizzlebippi]

4 minutes ago, YNM said:

 

Rumours had it that the plane shown different values in the main display between left hand side and right hand side, also the flight before was flown entirely manual.

This could be the most interesting detail I've seen yet. The left and right pilot displays use different sensors to ensure that in case of a latent failure, someone should be looking at correct data. This also factors into autopilot operation in that guidance commands are computed twice by independent systems using data from independent sensors (displayed as flight directors on the left and right displays). If the flight directors don't agree within a given tolerance, the autopilot won't work.

[XB-70A]

22 minutes ago, kerbiloid said:

One accident caused by a software mistake per how many ones caused by human malfunctions?

Agree. However, Companies and manufacturers (in all departments) also tend to like when the whole reason can be imputed to ground and flight crew. From my point of view, the power of electronic systems and assistance is its almost "perfection". But, the number of flight crews complaining about having to reboot (literally) the electronics while flying is still alarming. Even worse is when a system is conceived improperly to human factors and is hard to detect: LOT 16... when a flight crew made a belly landing and were acclaimed as hero, until the inquiry clearly showed the lack of seriousness from the airline maintenance department, the failure of the same flight crew to detect a circuit breaker failure, and the lack of a redundant system. At the end, the flight crew could have used the alternate landing-gear extension system to lower the undercarriage ()

 

1 minute ago, YNM said:

The plane (PK-LQP) did have a problem before when flying from Denpasar (Bali) to Jakarta - Passengers had to disembark after the first boarding attempt as the engines seem to have trouble starting, then on the second attempt it did fly, but the aircon was not cooled and some said it was flying lower and slower than usual.

Even worse, PK-LQP was having problems during the last four flights:

On Nov 5th 2018, following the KNKT release confirming airspeed indicator problems during the last 4 flights of the aircraft [...] On Nov 8th 2018 the KNKT reported an angle of attack sensor had been replaced on Oct 28th 2018 following the flight JT-775 from Manado to Denpasar (the aircraft completed the subsequent flight JT-43 to Jakarta and suffered the crash the next flight JT-610). The aircraft subsequently flew to Jakarta, the crew however reported there were still problems.

 

---

For those of you (if any) who would be interested to read the recommendation published two days ago:

On Nov 7th 2018 Boeing issued an Operations Manual Bulletin (OMB) to all Boeing 737 MAX Operators stating that the investigation into the crash of PK-LQP found one of the Angle of Attack Sensors had provided incorrect readings, which could cause the aircraft's trim system to uncommandedly trim nose down in order to avoid a stall during manual flight. The OMB directs "operators to existing flight crew procedures to address circumstances where there is erroneous input from an AOA sensor." The OMB reiterates the Stabilizer Runaway non-normal checklist.

The flight Crew Operations Manual Bulletin TBC-19 reads:

The Indonesian National Transportation Safety Committee has indicated that Lion Air flight 610 experienced erroneous AOA data. Boeing would like to call attention to an AOA failure condition that can occur during manual flight only.

This bulletin directs flight crews to existing procedures to address this condition. In the event of erroneous AOA data, the pitch trim system can trim the stabilizer nose down in increments lasting up to 10 seconds. The nose down stabilizer trim movement can be stopped and reversed with the use of the electric stabilizer trim switches but may restart 5 seconds after the electric stabilizer trim switches are released. Repetitive cycles of uncommanded nose down stabilizer continue to occur unless the stabilizer trim system is deactivated through use of both STAB TRIM CUTOUT switches in accordance with the existing procedures in the Runaway Stabilizer NNC. It is possible for the stabilizer to reach the nose down limit unless the system inputs are counteracted completely by pilot trim inputs and both STAB TRIM CUTOUT switches are moved to CUTOUT.

Additionally, pilots are reminded that an erroneous AOA can cause some or all of the following indications and effects:

- Continuous or intermittent stick shaker on the affected side only.
- Minimum speed bar (red and black) on the affected side only.
- Increasing nose down control forces.
- Inability to engage autopilot.
- Automatic disengagement of autopilot.
- IAS DISAGREE alert.
- ALT DISAGREE alert.
- AOA DISAGREE alert (if the AOA indicator option is installed)
- FEEL DIFF PRESS light.

In the event an uncommanded nose down stabilizer trim is experienced on the 737 - 8 / - 9, in conjunction with one or more of the above indications or effects, do the Runaway Stabilizer NNC ensuring that the STAB TRIM CUTOUT switches are set to CUTOUT and stay in the CUTOUT position for the remainder of the flight.

Late Nov 7th 2018 the FAA released Emergency Airworthiness Directive (EAD) 2018-23-51 concerning all Boeing 737 Max aircraft reading:

This emergency AD was prompted by analysis performed by the manufacturer showing that if an erroneously high single angle of attack (AOA) sensor input is received by the flight control system, there is a potential for repeated nose-down trim commands of the horizontal stabilizer. This condition, if not addressed, could cause the flight crew to have difficulty controlling the airplane, and lead to excessive nose-down attitude, significant altitude loss, and possible impact with terrain.

The EAD requires operators to update the procedures in the Aircraft Flight Manuals within 3 days according to Boeing's Service Bulletin (see the text above), however, includes the possiblity the trim could move even after the cutout switches were set to cutout. The text of the proedure for a Runaway Stabilizer mandated reads:

Disengage autopilot and control airplane pitch attitude with control column and main electric trim as required. If relaxing the column causes the trim to move, set stabilizer trim switches to CUTOUT. If runaway continues, hold the stabilizer trim wheel against rotation and trim the airplane manually.

Note: The 737 - 8 /- 9 uses a Flight Control Computer command of pitch trim to improve longitudinal handling characteristics. In the event of erroneous Angle of Attack (AOA) input, the pitch trim system can trim the stabilizer nose down in increments lasting up to 10 seconds.

In the event an uncommanded nose down stabilizer trim is experienced on the 737 - 8 / - 9, in conjunction with one or more of the indications or effects listed below, do the existing AFM Runaway Stabilizer procedure above, ensuring that the STAB TRIM CUTOUT switches are set to CUTOUT and stay in the CUTOUT position for the remainder of the flight.

An erroneous AOA input can cause some or all of the following indications and effects:

- Continuous or intermittent stick shaker on the affected side only.
- Minimum speed bar (red and black) on the affected side only.
- Increasing nose down control forces.
- IAS DISAGREE alert.
- ALT DISAGREE alert.
- AOA DISAGREE alert (if the option is installed).
- FEEL DIFF PRESS light.
- Autopilot may disengage.
- Inability to engage autopilot.

Initially, higher control forces may be needed to overcome any stabilizer nose down trim already applied. Electric stabilizer trim can be used to neutralize control column pitch forces before moving the STAB TRIM CUTOUT switches to CUTOUT. Manual stabilizer trim can be used before and after the STAB TRIM CUTOUT switches are moved to CUTOUT.

 

This is just a bad copy-pass. More can be followed on The Aviation Herald.

[YNM]

18 minutes ago, wizzlebippi said:

The left and right pilot displays use different sensors to ensure that in case of a latent failure, someone should be looking at correct data. This also factors into autopilot operation in that guidance commands are computed twice by independent systems using data from independent sensors (displayed as flight directors on the left and right displays). If the flight directors don't agree within a given tolerance, the autopilot won't work.

Hmm... yeah. I live in the nation that the tragedy has happened in so it was pretty much constant bombardment through the media for every small details and progress that's been made.

For the detail I said above, I'd call it "half-rumour" - the pilot who flew the airplane on the flight in the night before hasn't been picked up by the media, but is certainly being asked by the investigators to help identify the failure that had happened. There was some detail about the crews only looking on one of the displays rather than both, I think it's the co-pilot display that was showing the correct data.

The unfortunate flight was flown by a pilot of Indian nationality, while the flight before was flown by Indonesian crews - it might be a bit stereotypical but some have said the pilot on the unfortunate flight was a bit too reliant on the automation system.

I really hope this crash brings light into electrical systems resilience, though, as I'd be inclined to think that while the "older" hydraulic systems fair poorly in "temperate" climate (given the freezing-heating weathers to result in tube cracks), "newer" electrical systems would fair poorly in (tropical) hot, humid climate (through corrosion). Not to say that these two crashes didn't have any negligence factor from airline maintenance however, as these two unfortunate flights are flown by LCCs, and might have more 'cracks' than otherwise.

12 minutes ago, XB-70A said:

Boeing would like to call attention to an AOA failure condition that can occur during manual flight only.

Huh. Well, idk but some have speculated the whole thing happens in autopilot somehow... not sure. I'll have to wait for the full report.

[wizzlebippi]

The only reason it's manual flight only is AP-ON operation likely has full envelope protection. This means the autopilot will never allow the aircraft to slow to the speed that the pusher activates.

I find it a bit disturbing that the pusher uses up to 10 seconds of trim movement. A 3 second mistrim is all that is required in testing. 10 seconds would have to be catastrophically high control forces, even with hydraulically boosted controls (which I think the 737 has).

I also find it disturbing that a failed/malfunctioning AoA vane has an impact on all other level A sensors. Someone really failed at the fault tree analysis for AoA. Though my favorite is still the Legacy 450 with a GPS dependant yaw damper.

For YNM's concern, Boeing has to have done extreme heat/humidity (105F at 100% relative humidity and 135F with 12+ hour exposure prior to testing) testing as well as extreme cold (-40F/C for 12+ hours is pretty popular). The McKinley Climactic Chamber at Eglin AFB is a popular destination for this testing. You are right, these conditions make a lot of systems not work. It generally involves one visit to see what breaks so it can be fixed, and another to actually certify. A damaged wire bundle could be the root cause, but I would like to think they would do a continuity test on the wiring.

[StrandedonEarth]

1 hour ago, wizzlebippi said:

Someone really failed at the fault tree analysis for AoA.

Well, it seems they decided to rely on the pilots to fly the plane when the computers started getting bad data. But they need to finish analyzing the  "black boxes" to figure out the sequence of events.

It's more concerning that the aircraft was back in the air when pilots were reporting problems...

[YNM]

3 hours ago, wizzlebippi said:

For YNM's concern, Boeing has to have done extreme heat/humidity (105F at 100% relative humidity and 135F with 12+ hour exposure prior to testing) testing as well as extreme cold (-40F/C for 12+ hours is pretty popular).

To be fair those would be higher than operating conditions - albeit I'll have to say that it's a lot shorter as well. Not sure how FAA achieved their kapton testing that proves them unsuitable in the past, might be a similar thing.

And yeah the trim down looks aggressive.

1 hour ago, StrandedonEarth said:

It's more concerning that the aircraft was back in the air when pilots were reporting problems...

Lion Air's spokeperson said in the early aftermath of the accident that "technicians have declared the problems to have been cleared", albeit now the head of technical department of Lion Air has been suspended.

Given these are LCCs I don't think they'd invest money and effort to look further than the minimums that's specified by the manufacturers, unlike flag carriers (ie. Garuda Indonesia which owns it's own full maintenance and overhaul facility (GMF AeroAsia) ). QZ 8501's report also suggested that there should've been additional actions from maintenance when the RTLU warning has been turned on in multiple occasions on the ground, which might've uncovered the cracked soldering and repaired it, though I can only imagine this action isn't mentioned on the minimum maintenance level provided by the aircraft manufacturer (probably expecting it to be uncovered fairly soon after in a full check or so).

Edited November 10, 2018 by YNM

[MaverickSawyer]

I'll restate my recommendation to sit back and quit being armchair accident investigators. We do NOT have anywhere close to enough facts publicly released from reputable sources to begin forming a hypothesis on our own, and we may not get that level of detail for months, if not longer.

[mrfox]

The system in question (and the subject of the EAD) is not the stick pusher, but the 737s speed trim system (STS). It is a stability "augmentation" system that adds trim in the opposite sense as airspeed and thrust settings change, in order to artifically increase the stick force required during manual flight. It is a very counter-intuitive system that trims against pilot input to create this additional stick force. All because the 737 is not naturally speed stable enough to meet certification requirements otherwise.

[YNM]

33 minutes ago, MaverickSawyer said:

We do NOT have anywhere close to enough facts publicly released from reputable sources to begin forming a hypothesis on our own, and we may not get that level of detail for months, if not longer.

I'm just voicing my layman concerns (the same way people that are afraid of flying do), which will be a massive relief if they all are proven wrong.

5 minutes ago, mrfox said:

... because the 737 is not naturally speed stable enough to meet certification requirements otherwise.

... Pardon ?

[mrfox]

2 hours ago, YNM said:

... Pardon ?

Aircraft certification requirements include specific stick force requirements at various speed and power configurations in order to provide stable handling characteristics. When Boeing stretched the 737 these handling characteristics degraded, so they came up with the STS, which artifically increased these stick forces on the cheap by running the trim system the opposite way the pilot is pulling. It ticks the box for the FAA requirements but is utterly counter-productive for the pilot, as he/she will have just have to untrim these inputs later.

It is a great example of how a bureaucracy can end up with the tail wagging the dog sometimes - a certification rule intended to provide ease of handling for the pilot ends up making handling worse by introducing counter-productive inputs in order to satisfy a "rule".

[mikegarrison]

6 hours ago, MaverickSawyer said:

I'll restate my recommendation to sit back and quit being armchair accident investigators. We do NOT have anywhere close to enough facts publicly released from reputable sources to begin forming a hypothesis on our own, and we may not get that level of detail for months, if not longer.

The more someone *actually* knows about something like this, the fewer specifics they will post in an open forum. Wait for the NTSB reports.

[YNM]

3 hours ago, mikegarrison said:

Wait for the NTSB reports.

That'd probably come next year or so. But at least we can roughly see what things are from ADs and operation manuals or such stuff if any, in the meantime (as they'd be published immediately upon realization they'd be very instrumental to aviation safety).

And there's a lot of parties that's interested in this : NTSC (KNKT) being the main investigation with GE, Boeing and FAA/NTSB giving full support and assistance. Plus all other transportation safety organization from other countries.

7 hours ago, mrfox said:

Aircraft certification requirements include specific stick force requirements at various speed and power configurations in order to provide stable handling characteristics. When Boeing stretched the 737 these handling characteristics degraded, so they came up with the STS, which artifically increased these stick forces on the cheap by running the trim system the opposite way the pilot is pulling.

... huh.

Alright, I'm definitely not flying on LCCs until we know all that went wrong.

Edited November 10, 2018 by YNM

[DerekL1963]

9 hours ago, mikegarrison said:

The more someone *actually* knows about something like this, the fewer specifics they will post in an open forum. Wait for the NTSB reports.

Look guys, if you don't want to participate in the discussion that's fine.  Nobody is forcing you to click on the thread and read it. 

But you're way out of line to suggest that other people not discuss it.

[wizzlebippi]

12 hours ago, mrfox said:

Aircraft certification requirements include specific stick force requirements at various speed and power configurations in order to provide stable handling characteristics. When Boeing stretched the 737 these handling characteristics degraded, so they came up with the STS, which artifically increased these stick forces on the cheap by running the trim system the opposite way the pilot is pulling. It ticks the box for the FAA requirements but is utterly counter-productive for the pilot, as he/she will have just have to untrim these inputs later.

It is a great example of how a bureaucracy can end up with the tail wagging the dog sometimes - a certification rule intended to provide ease of handling for the pilot ends up making handling worse by introducing counter-productive inputs in order to satisfy a "rule".

The problem is that insufficient speed stability resulted in a number of Leerjet crashes because the aircraft's pitching moment resulted in an increasing nose down pitch with increasing speed. The regs are written in blood. Never forget that.

[StrandedonEarth]

7 minutes ago, wizzlebippi said:

The problem is that insufficient speed stability resulted in a number of Leerjet crashes because the aircraft's pitching moment resulted in an increasing nose down pitch with increasing speed. The regs are written in blood. Never forget that.

I think the point is that instead of complying with regs by way of using a software/control workaround, they should find a way to make the design work without such workarounds, or realize that the stretch is simply too much of a stretch. Talk about Band-aid solutions when a tourniquet is called for. It all comes back to the K.I.S.S. principle, instead of requiring pilots to learn complex procedures for a single aircraft model when a computer falls victim to Garbage In Garbage Out

 

[mikegarrison]

1 hour ago, DerekL1963 said:

Look guys, if you don't want to participate in the discussion that's fine.  Nobody is forcing you to click on the thread and read it. 

But you're way out of line to suggest that other people not discuss it.

Well, I guess I did say "wait for the NTSB reports", but I didn't mean people can't discuss whatever they like. However, I stand by what I said that nobody with access to any actual facts is going to be posting here.

54 minutes ago, StrandedonEarth said:

I think the point is that instead of complying with regs by way of using a software/control workaround, they should find a way to make the design work without such workarounds, or realize that the stretch is simply too much of a stretch. Talk about Band-aid solutions when a tourniquet is called for. It all comes back to the K.I.S.S. principle, instead of requiring pilots to learn complex procedures for a single aircraft model when a computer falls victim to Garbage In Garbage Out

 

The speed stability system has been on 737s since at least the Next Gen. It's not a new system. It has worked as intended for more than 20 years.

[mrfox]

7 hours ago, mikegarrison said:

The speed stability system has been on 737s since at least the Next Gen. It's not a new system. It has worked as intended for more than 20 years.

From what I can gather from various reports, it appears that unlike previous 737 generations, the STS in the MAX also takes data directly from the AOA probes.