Lessons Learnt from Ethiopian Airlines: Should Human Intelligence Override AI?

It was a mid air struggle. On one hand there was the Ethiopian Airlines' Boeing's MCAS computer system erroneously stalling the flight (letting the plane go down); and on the other, the flight's captain, trying to get the aircraft up so that it does not crash. In the end, the helpless captain failed, the computer won, and 189 passengers died as the flight crashed.

There are several cases of indicative IT failure leading to tragedies. The France Air Airbus 330 fell into the Atlantic Ocean in June 2009 killing 228 passengers on board. The last 10 years has seen many IT malfunction and failure related accidents and near accidents - killing and injuring hundreds. Quantas flight 72, AeroPehru flight 603, Adam Air flight 574, SpanAir flight JK 5022 are a few one can recollect.

What is noteworthy, from the recollection of news of those accidents, was that the reason of the accidents was mostly attributed to computer malfunction and failure. And the pilots could not do much. But the case of Ethiopian Airlines crash is bizarre. The pilots knew what was going wrong and that the flight was stalling, they tried to take control in their hands, but the computer did not allow that. It was only during the postmortem analysis that it was discovered that the captain had the power to switch the MCAS computer off and take charge, but he did not. Clearly, Boeing engineers had a very high degree of confidence in their computer system; to the extent that almost till the end, flight's captain had no control over the computer. His only option was to switch it off. Maybe it was too late and he panicked and forgot to switch it off.

But that is the not my point. I am worried about the direction scientists and engineers are taking in terms of dependence on computers systems. With increasing technology deployment, will we see more such accidents spreading over to factories, trains, nuclear power plants, sports stadiums et al, putting hundreds or thousands of lives in jeopardy. A friend of mine whose elder brother is a manager with a leading car dealer tells me that many customers are often wary of automatic locking systems in the automobile. Will I and my family be able to open the door and run out in case of emergency, they ask.

People will forget the Ethiopian Airlines crash that killed 189 people as they get back to their rat race of daily life, but it is a fact that the world is rapidly moving towards a tech driven tech dependent world. Driver-less airplanes, trains, cars, and factories are already here; and robots as doctors, firemen, and foot soldiers is not a matter of just imagination now. Trials and pilots are on; and there is lots of excitement, money and glamour around it.

As safety is paramount, most countries regulate avionics and have standards, and the regulatory organizations in the US, EU, and Russia most affect international aviation development. Importance of IT is going up as avionics manufacturers see software as a way to add value without adding weight. Latest commercial aircrafts are flown by computer based auto-pilots, without the pilot's active intervention during certain phases of flight. Coming up soon are unmanned vehicles, missiles and drones which can take off, cruise and land without a pilot. According to an aviation veteran, "In many of these systems, failure has to be unacceptable. Unfortunately reliable software is not necessarily easy to use, often has not the best user interface, thus becoming part of many accidents"

While avionics hardware and software development is far more detailed and rigorous than commercial standards, and usually runs on a real time operating system, it might be time that the regulators and policy makers, along with technology companies, go back to the drawing board to review their list of products and processes that will run on AI, and those that will run on human intelligence.