There were six hours during the night of April 10, 2014, when the entire population of Washington State had no 911 service. People who called for help got a busy signal. One Seattle woman dialed 911 at least 37 times while a stranger was trying to break into her house. When he finally crawled into her living room through a window, she picked up a kitchen knife. The man fled.

The 911 outage, at the time the largest ever reported, was traced to software running on a server in Englewood, Colorado. Operated by a systems provider named Intrado, the server kept a running counter of how many calls it had routed to 911 dispatchers around the country. Intrado programmers had set a threshold for how high the counter could go. They picked a number in the millions.

Shortly before midnight on April 10, the counter exceeded that number, resulting in chaos. Because the counter was used to generate a unique identifier for each call, new calls were rejected. And because the programmers hadn’t anticipated the problem, they hadn’t created alarms to call attention to it. Nobody knew what was happening. Dispatch centers in Washington, California, Florida, the Carolinas, and Minnesota, serving 11 million Americans, struggled to make sense of reports that callers were getting busy signals. It took until morning to realize that Intrado’s software in Englewood was responsible, and that the fix was to change a single number.

Not long ago, emergency calls were handled locally. Outages were small and easily diagnosed and fixed. The rise of cellphones and the promise of new capabilities—what if you could text 911? or send videos to the dispatcher?—drove the development of a more complex system that relied on the internet. For the first time, there could be such a thing as a national 911 outage. There have now been four in as many years.

It’s been said that software is “eating the world.” More and more, critical systems that were once controlled mechanically, or by people, are coming to depend on code. This was perhaps never clearer than in the summer of 2015, when on a single day, United Airlines grounded its fleet because of a problem with its departure-management system; trading was suspended on the New York Stock Exchange after an upgrade; the front page of The Wall Street Journal’s website crashed; and Seattle’s 911 system went down again, this time because a different router failed. The simultaneous failure of so many software systems smelled at first of a coordinated cyberattack. Almost more frightening was the realization, late in the day, that it was just a coincidence.

“When we had electromechanical systems, we used to be able to test them exhaustively,” says Nancy Leveson, a professor of aeronautics and astronautics at the Massachusetts Institute of Technology who has been studying software safety for 35 years. She became known for her report on the Therac-25, a radiation-therapy machine that killed six patients because of a software error. “We used to be able to think through all the things it could do, all the states it could get into.” The electromechanical interlockings that controlled train movements at railroad crossings, for instance, only had so many configurations; a few sheets of paper could describe the whole system, and you could run physical trains against each configuration to see how it would behave. Once you’d built and tested it, you knew exactly what you were dealing with.

Software is different. Just by editing the text in a file somewhere, the same hunk of silicon can become an autopilot or an inventory-control system. This flexibility is software’s miracle, and its curse. Because it can be changed cheaply, software is constantly changed; and because it’s unmoored from anything physical—a program that is a thousand times more complex than another takes up the same actual space—it tends to grow without bound. “The problem,” Leveson wrote in a book, “is that we are attempting to build systems that are beyond our ability to intellectually manage.”

Our standard framework for thinking about engineering failures—reflected, for instance, in regulations for medical devices—was developed shortly after World War II, before the advent of software, for electromechanical systems. The idea was that you make something reliable by making its parts reliable (say, you build your engine to withstand 40,000 takeoff-and-landing cycles) and by planning for the breakdown of those parts (you have two engines). But software doesn’t break. Intrado’s faulty threshold is not like the faulty rivet that leads to the crash of an airliner. The software did exactly what it was told to do. In fact it did it perfectly. The reason it failed is that it was told to do the wrong thing. Software failures are failures of understanding, and of imagination. Intrado actually had a backup router, which, had it been switched to automatically, would have restored 911 service almost immediately. But, as described in a report to the FCC, “the situation occurred at a point in the application logic that was not designed to perform any automated corrective actions.”

This is the trouble with making things out of code, as opposed to something physical. “The complexity,” as Leveson puts it, “is invisible to the eye.”

The attempts now underway to change how we make software all seem to start with the same premise: Code is too hard to think about. Before trying to understand the attempts themselves, then, it’s worth understanding why this might be: what it is about code that makes it so foreign to the mind, and so unlike anything that came before it.

Technological progress used to change the way the world looked—you could watch the roads getting paved; you could see the skylines rise. Today you can hardly tell when something is remade, because so often it is remade by code. When you press your foot down on your car’s accelerator, for instance, you’re no longer controlling anything directly; there’s no mechanical link from the pedal to the throttle. Instead, you’re issuing a command to a piece of software that decides how much air to give the engine. The car is a computer you can sit inside of. The steering wheel and pedals might as well be keyboard keys.

Like everything else, the car has been computerized to enable new features. When a program is in charge of the throttle and brakes, it can slow you down when you’re too close to another car, or precisely control the fuel injection to help you save on gas. When it controls the steering, it can keep you in your lane as you start to drift, or guide you into a parking space. You couldn’t build these features without code. If you tried, a car might weigh 40,000 pounds, an immovable mass of clockwork.

Software has enabled us to make the most intricate machines that have ever existed. And yet we have hardly noticed, because all of that complexity is packed into tiny silicon chips as millions and millions of lines of code. But just because we can’t see the complexity doesn’t mean that it has gone away.

The programmer, the renowned Dutch computer scientist Edsger Dijkstra wrote in 1988, “has to be able to think in terms of conceptual hierarchies that are much deeper than a single mind ever needed to face before.” Dijkstra meant this as a warning. As programmers eagerly poured software into critical systems, they became, more and more, the linchpins of the built world—and Dijkstra thought they had perhaps overestimated themselves.

What made programming so difficult was that it required you to think like a computer. The strangeness of it was in some sense more vivid in the early days of computing, when code took the form of literal ones and zeros. Anyone looking over a programmer’s shoulder as they pored over line after line like “100001010011” and “000010011110” would have seen just how alienated the programmer was from the actual problems they were trying to solve; it would have been impossible to tell whether they were trying to calculate artillery trajectories or simulate a game of tic-tac-toe. The introduction of programming languages like Fortran and C, which resemble English, and tools, known as “integrated development environments,” or IDEs, that help correct simple mistakes (like Microsoft Word’s grammar checker but for code), obscured, though did little to actually change, this basic alienation—the fact that the programmer didn’t work on a problem directly, but rather spent their days writing out instructions for a machine.

“The problem is that software engineers don’t understand the problem they’re trying to solve, and don’t care to,” says Leveson, the MIT software-safety expert. The reason is that they’re too wrapped up in getting their code to work. “Software engineers like to provide all kinds of tools and stuff for coding errors,” she says, referring to IDEs. “The serious problems that have happened with software have to do with requirements, not coding errors.” When you’re writing code that controls a car’s throttle, for instance, what’s important is the rules about when and how and by how much to open it. But these systems have become so complicated that hardly anyone can keep them straight in their head. “There’s 100 million lines of code in cars now,” Leveson says. “You just cannot anticipate all these things.”

In September 2007, Jean Bookout was driving on the highway with her best friend in a Toyota Camry when the accelerator seemed to get stuck. When she took her foot off the pedal, the car didn’t slow down. She tried the brakes but they seemed to have lost their power. As she swerved toward an off-ramp going 50 miles per hour, she pulled the emergency brake. The car left a skid mark 150 feet long before running into an embankment by the side of the road. The passenger was killed. Bookout woke up in a hospital a month later.

The incident was one of many in a nearly decade-long investigation into claims of so-called unintended acceleration in Toyota cars. Toyota blamed the incidents on poorly designed floor mats, “sticky” pedals, and driver error, but outsiders suspected that faulty software might be responsible. The National Highway Traffic Safety Administration enlisted software experts from NASA to perform an intensive review of Toyota’s code. After nearly 10 months, the NASA team hadn’t found evidence that software was the cause—but said they couldn’t prove it wasn’t.

It was during litigation of the Bookout accident that someone finally found a convincing connection. Michael Barr, an expert witness for the plaintiff, had a team of software experts spend 18 months with the Toyota code, picking up where NASA left off. Barr described what they found as “spaghetti code,” programmer lingo for software that has become a tangled mess. Code turns to spaghetti when it accretes over many years, with feature after feature piling on top of, and being woven around, what’s already there; eventually the code becomes impossible to follow, let alone to test exhaustively for flaws.

Using the same model as the Camry involved in the accident, Barr’s team demonstrated that there were more than 10 million ways for key tasks on the onboard computer to fail, potentially leading to unintended acceleration.* They showed that as little as a single bit flip—a one in the computer’s memory becoming a zero or vice versa—could make a car run out of control. The fail-safe code that Toyota had put in place wasn’t enough to stop it. “You have software watching the software,” Barr testified. “If the software malfunctions and the same program or same app that is crashed is supposed to save the day, it can’t save the day because it is not working.”

Barr’s testimony made the case for the plaintiff, resulting in $3 million in damages for Bookout and her friend’s family. According to The New York Times, it was the first of many similar cases against Toyota to bring to trial problems with the electronic throttle-control system, and the first time Toyota was found responsible by a jury for an accident involving unintended acceleration. The parties decided to settle the case before punitive damages could be awarded. In all, Toyota recalled more than 9 million cars, and paid nearly $3 billion in settlements and fines related to unintended acceleration.

There will be more bad days for software. It's important that we get better at making it, because if we don't, and as software becomes more sophisticated and connected—as it takes control of more critical functions—those days could get worse.

The problem is that programmers are having a hard time keeping up with their own creations. Since the 1980s, the way programmers work and the tools they use have changed remarkably little. There is a small but growing chorus that worries the status quo is unsustainable. “Even very good programmers are struggling to make sense of the systems that they are working with,” says Chris Granger, a software developer who worked as a lead at Microsoft on Visual Studio, an IDE that costs $1,199 a year and is used by nearly a third of all professional programmers. He told me that while he was at Microsoft, he arranged an end-to-end study of Visual Studio, the only one that had ever been done. For a month and a half, he watched behind a one-way mirror as people wrote code. “How do they use tools? How do they think?” he said. “How do they sit at the computer, do they touch the mouse, do they not touch the mouse? All these things that we have dogma around that we haven’t actually tested empirically.”

The findings surprised him. “Visual Studio is one of the single largest pieces of software in the world,” he said. “It’s over 55 million lines of code. And one of the things that I found out in this study is more than 98 percent of it is completely irrelevant. All this work had been put into this thing, but it missed the fundamental problems that people faced. And the biggest one that I took away from it was that basically people are playing computer inside their head.” Programmers were like chess players trying to play with a blindfold on—so much of their mental energy is spent just trying to picture where the pieces are that there’s hardly any left over to think about the game itself.

John Resig had been noticing the same thing among his students. Resig is a celebrated programmer of JavaScript—software he wrote powers over half of all websites—and a tech lead at the online-education site Khan Academy. In early 2012, he had been struggling with the site’s computer-science curriculum. Why was it so hard to learn to program? The essential problem seemed to be that code was so abstract. Writing software was not like making a bridge out of popsicle sticks, where you could see the sticks and touch the glue. To “make” a program, you typed words. When you wanted to change the behavior of the program, be it a game, or a website, or a simulation of physics, what you actually changed was text. So the students who did well—in fact the only ones who survived at all—were those who could step through that text one instruction at a time in their head, thinking the way a computer would, trying to keep track of every intermediate calculation. Resig, like Granger, started to wonder if it had to be that way. Computers had doubled in power every 18 months for the last 40 years. Why hadn’t programming changed?

The fact that the two of them were thinking about the same problem in the same terms, at the same time, was not a coincidence. They had both just seen the same remarkable talk, given to a group of software-engineering students in a Montreal hotel by a computer researcher named Bret Victor. The talk, which went viral when it was posted online in February 2012, seemed to be making two bold claims. The first was that the way we make software is fundamentally broken. The second was that Victor knew how to fix it.

Bret Victor does not like to write code. “It sounds weird,” he says. “When I want to make a thing, especially when I want to create something in software, there’s this initial layer of disgust that I have to push through, where I’m not manipulating the thing that I want to make, I’m writing a bunch of text into a text editor.”

“There’s a pretty strong conviction that that’s the wrong way of doing things.”

Victor has the mien of David Foster Wallace, with a lightning intelligence that lingers beneath a patina of aw-shucks shyness. He is 40 years old, with traces of gray and a thin, undeliberate beard. His voice is gentle, mournful almost, but he wants to share what’s in his head, and when he gets on a roll he’ll seem to skip syllables, as though outrunning his own vocal machinery.

Though he runs a lab that studies the future of computing, he seems less interested in technology per se than in the minds of the people who use it. Like any good toolmaker, he has a way of looking at the world that is equal parts technical and humane. He graduated top of his class at the California Institute of Technology for electrical engineering, and then went on, after grad school at the University of California, Berkeley, to work at a company that develops music synthesizers. It was a problem perfectly matched to his dual personality: He could spend as much time thinking about the way a performer makes music with a keyboard—the way it becomes an extension of their hands—as he could thinking about the mathematics of digital signal processing.

By the time he gave the talk that made his name, the one that Resig and Granger saw in early 2012, Victor had finally landed upon the principle that seemed to thread through all of his work. (He actually called the talk “Inventing on Principle.”) The principle was this: “Creators need an immediate connection to what they’re creating.” The problem with programming was that it violated the principle. That’s why software systems were so hard to think about, and so rife with bugs: The programmer, staring at a page of text, was abstracted from whatever it was they were actually making.

“Our current conception of what a computer program is,” he said, is “derived straight from Fortran and ALGOL in the late ’50s. Those languages were designed for punch cards.” That code now takes the form of letters on a screen in a language like C or Java (derivatives of Fortran and ALGOL), instead of a stack of cards with holes in it, doesn’t make it any less dead, any less indirect.

There is an analogy to word processing. It used to be that all you could see in a program for writing documents was the text itself, and to change the layout or font or margins, you had to write special “control codes,” or commands that would tell the computer that, for instance, “this part of the text should be in italics.” The trouble was that you couldn’t see the effect of those codes until you printed the document. It was hard to predict what you were going to get. You had to imagine how the codes were going to be interpreted by the computer—that is, you had to play computer in your head.

Then WYSIWYG (pronounced “wizzywig”) came along. It stood for “What You See Is What You Get.” When you marked a passage as being in italics, the letters tilted right there on the screen. If you wanted to change the margin, you could drag a ruler at the top of the screen—and see the effect of that change. The document thereby came to feel like something real, something you could poke and prod at. Just by looking you could tell if you’d done something wrong. Control of a sophisticated system—the document’s layout and formatting engine—was made accessible to anyone who could click around on a page.

Victor’s point was that programming itself should be like that. For him, the idea that people were doing important work, like designing adaptive cruise-control systems or trying to understand cancer, by staring at a text editor, was appalling. And it was the proper job of programmers to ensure that someday they wouldn’t have to.

There was precedent enough to suggest that this wasn’t a crazy idea. Photoshop, for instance, puts powerful image-processing algorithms in the hands of people who might not even know what an algorithm is. It’s a complicated piece of software, but complicated in the way a good synth is complicated, with knobs and buttons and sliders that the user learns to play like an instrument. Squarespace, a company that is perhaps best known for advertising aggressively on podcasts, makes a tool that lets users build websites by pointing and clicking, instead of by writing code in HTML and CSS. It is powerful enough to do work that once would have been done by a professional web designer.

But those were just a handful of examples. The overwhelming reality was that when someone wanted to do something interesting with a computer, they had to write code. Victor, who is something of an idealist, saw this not so much as an opportunity but as a moral failing of programmers at large. His talk was a call to arms.

At the heart of it was a series of demos that tried to show just how primitive the available tools were for various problems—circuit design, computer animation, debugging algorithms—and what better ones might look like. His demos were virtuosic. The one that captured everyone’s imagination was, ironically enough, the one that on its face was the most trivial. It showed a split screen with a game that looked like Mario on one side and the code that controlled it on the other. As Victor changed the code, things in the game world changed: He decreased one number, the strength of gravity, and the Mario character floated; he increased another, the player’s speed, and Mario raced across the screen.

Suppose you wanted to design a level where Mario, jumping and bouncing off of a turtle, would just make it into a small passageway. Game programmers were used to solving this kind of problem in two stages: First, you stared at your code—the code controlling how high Mario jumped, how fast he ran, how bouncy the turtle’s back was—and made some changes to it in your text editor, using your imagination to predict what effect they’d have. Then, you’d replay the game to see what actually happened.

Victor wanted something more immediate. “If you have a process in time,” he said, referring to Mario’s path through the level, “and you want to see changes immediately, you have to map time to space.” He hit a button that showed not just where Mario was right now, but where he would be at every moment in the future: a curve of shadow Marios stretching off into the far distance. What’s more, this projected path was reactive: When Victor changed the game’s parameters, now controlled by a quick drag of the mouse, the path’s shape changed. It was like having a god’s-eye view of the game. The whole problem had been reduced to playing with different parameters, as if adjusting levels on a stereo receiver, until you got Mario to thread the needle. With the right interface, it was almost as if you weren’t working with code at all; you were manipulating the game’s behavior directly.

When the audience first saw this in action, they literally gasped. They knew they weren’t looking at a kid’s game, but rather the future of their industry. Most software involved behavior that unfolded, in complex ways, over time, and Victor had shown that if you were imaginative enough, you could develop ways to see that behavior and change it, as if playing with it in your hands. One programmer who saw the talk wrote later: “Suddenly all of my tools feel obsolete.”

When John Resig saw the “Inventing on Principle” talk, he scrapped his plans for the Khan Academy programming curriculum. He wanted the site’s programming exercises to work just like Victor’s demos. On the left-hand side you’d have the code, and on the right, the running program: a picture or game or simulation. If you changed the code, it’d instantly change the picture. “In an environment that is truly responsive,” Resig wrote about the approach, “you can completely change the model of how a student learns ... [They] can now immediately see the result and intuit how underlying systems inherently work without ever following an explicit explanation.” Khan Academy has become perhaps the largest computer-programming class in the world, with a million students, on average, actively using the program each month.

Chris Granger, who had worked at Microsoft on Visual Studio, was likewise inspired. Within days of seeing a video of Victor’s talk, in January of 2012, he built a prototype of a new programming environment. Its key capability was that it would give you instant feedback on your program’s behavior. You’d see what your system was doing right next to the code that controlled it. It was like taking off a blindfold. Granger called the project “Light Table.”

In April of 2012, he sought funding for Light Table on Kickstarter. In programming circles, it was a sensation. Within a month, the project raised more than $200,000. The ideas spread. The notion of liveness, of being able to see data flowing through your program instantly, made its way into flagship programming tools offered by Google and Apple. The default language for making new iPhone and Mac apps, called Swift, was developed by Apple from the ground up to support an environment, called Playgrounds, that was directly inspired by Light Table.

But seeing the impact that his talk ended up having, Bret Victor was disillusioned. “A lot of those things seemed like misinterpretations of what I was saying,” he said later. He knew something was wrong when people began to invite him to conferences to talk about programming tools. “Everyone thought I was interested in programming environments,” he said. Really he was interested in how people see and understand systems—as he puts it, in the “visual representation of dynamic behavior.” Although code had increasingly become the tool of choice for creating dynamic behavior, it remained one of the worst tools for understanding it. The point of “Inventing on Principle” was to show that you could mitigate that problem by making the connection between a system’s behavior and its code immediate.

In a pair of later talks, “Stop Drawing Dead Fish” and “Drawing Dynamic Visualizations,” Victor went one further. He demoed two programs he’d built—the first for animators, the second for scientists trying to visualize their data—each of which took a process that used to involve writing lots of custom code and reduced it to playing around in a WYSIWYG interface. Victor suggested that the same trick could be pulled for nearly every problem where code was being written today. “I’m not sure that programming has to exist at all,” he told me. “Or at least software developers.” In his mind, a software developer’s proper role was to create tools that removed the need for software developers. Only then would people with the most urgent computational problems be able to grasp those problems directly, without the intermediate muck of code.

Of course, to do that, you’d have to get programmers themselves on board. In a recent essay, Victor implored professional software developers to stop pouring their talent into tools for building apps like Snapchat and Uber. “The inconveniences of daily life are not the significant problems,” he wrote. Instead, they should focus on scientists and engineers—as he put it to me, “these people that are doing work that actually matters, and critically matters, and using really, really bad tools.” Exciting work of this sort, in particular a class of tools for “model-based design,” was already underway, he wrote, and had been for years, but most programmers knew nothing about it.

“If you really look hard at all the industrial goods that you’ve got out there, that you’re using, that companies are using, the only non-industrial stuff that you have inside this is the code.” Eric Bantégnie is the founder of Esterel Technologies (now owned by ANSYS), a French company that makes tools for building safety-critical software. Like Victor, Bantégnie doesn’t think engineers should develop large systems by typing millions of lines of code into an IDE. “Nobody would build a car by hand,” he says. “Code is still, in many places, handicraft. When you’re crafting manually 10,000 lines of code, that’s okay. But you have systems that have 30 million lines of code, like an Airbus, or 100 million lines of code, like your Tesla or high-end cars—that’s becoming very, very complicated.”

Bantégnie’s company is one of the pioneers in the industrial use of model-based design, in which you no longer write code directly. Instead, you create a kind of flowchart that describes the rules your program should follow (the “model”), and the computer generates code for you based on those rules. If you were making the control system for an elevator, for instance, one rule might be that when the door is open, and someone presses the button for the lobby, you should close the door and start moving the car. In a model-based design tool, you’d represent this rule with a small diagram, as though drawing the logic out on a whiteboard, made of boxes that represent different states—like “door open,” “moving,” and “door closed”—and lines that define how you can get from one state to the other. The diagrams make the system’s rules obvious: Just by looking, you can see that the only way to get the elevator moving is to close the door, or that the only way to get the door open is to stop.

It’s not quite Photoshop. The beauty of Photoshop, of course, is that the picture you’re manipulating on the screen is the final product. In model-based design, by contrast, the picture on your screen is more like a blueprint. Still, making software this way is qualitatively different than traditional programming. In traditional programming, your task is to take complex rules and translate them into code; most of your energy is spent doing the translating, rather than thinking about the rules themselves. In the model-based approach, all you have is the rules. So that’s what you spend your time thinking about. It’s a way of focusing less on the machine and more on the problem you’re trying to get it to solve.

“Typically the main problem with software coding—and I’m a coder myself,” Bantégnie says, “is not the skills of the coders. The people know how to code. The problem is what to code. Because most of the requirements are kind of natural language, ambiguous, and a requirement is never extremely precise, it’s often understood differently by the guy who’s supposed to code.”

On this view, software becomes unruly because the media for describing what software should do—conversations, prose descriptions, drawings on a sheet of paper—are too different from the media describing what software does do, namely, code itself. Too much is lost going from one to the other. The idea behind model-based design is to close the gap. The very same model is used both by system designers to express what they want and by the computer to automatically generate code.

Of course, for this approach to succeed, much of the work has to be done well before the project even begins. Someone first has to build a tool for developing models that are natural for people—that feel just like the notes and drawings they’d make on their own—while still being unambiguous enough for a computer to understand. They have to make a program that turns these models into real code. And finally they have to prove that the generated code will always do what it’s supposed to. “We have benefited from fortunately 20 years of initial background work,” Bantégnie says.

Esterel Technologies, which was acquired by ANSYS in 2012, grew out of research begun in the 1980s by the French nuclear and aerospace industries, who worried that as safety-critical code ballooned in complexity, it was getting harder and harder to keep it free of bugs. “I started in 1988,” says Emmanuel Ledinot, the Head of Scientific Studies for Dassault Aviation, a French manufacturer of fighter jets and business aircraft. “At the time, I was working on military avionics systems. And the people in charge of integrating the systems, and debugging them, had noticed that the number of bugs was increasing.” The 80s had seen a surge in the number of onboard computers on planes. Instead of a single flight computer, there were now dozens, each responsible for highly specialized tasks related to control, navigation, and communications. Coordinating these systems to fly the plane as data poured in from sensors and as pilots entered commands required a symphony of perfectly timed reactions. “The handling of these hundreds of and even thousands of possible events in the right order, at the right time,” Ledinot says, “was diagnosed as the main cause of the bug inflation.”

Ledinot decided that writing such convoluted code by hand was no longer sustainable. It was too hard to understand what it was doing, and almost impossible to verify that it would work correctly. He went looking for something new. “You must understand that to change tools is extremely expensive in a process like this,” he said in a talk. “You don’t take this type of decision unless your back is against the wall.”

He began collaborating with Gerard Berry, a computer scientist at INRIA, the French computing-research center, on a tool called Esterel—a portmanteau of the French for “real-time.” The idea behind Esterel was that while traditional programming languages might be good for describing simple procedures that happened in a predetermined order—like a recipe—if you tried to use them in systems where lots of events could happen at nearly any time, in nearly any order—like in the cockpit of a plane—you inevitably got a mess. And a mess in control software was dangerous. In a paper, Berry went as far as to predict that “low-level programming techniques will not remain acceptable for large safety-critical programs, since they make behavior understanding and analysis almost impracticable.”

Esterel was designed to make the computer handle this complexity for you. That was the promise of the model-based approach: Instead of writing normal programming code, you created a model of the system’s behavior—in this case, a model focused on how individual events should be handled, how to prioritize events, which events depended on which others, and so on. The model becomes the detailed blueprint that the computer would use to do the actual programming.

Ledinot and Berry worked for nearly 10 years to get Esterel to the point where it could be used in production. “It was in 2002 that we had the first operational software-modeling environment with automatic code generation,” Ledinot told me, “and the first embedded module in Rafale, the combat aircraft.” Today, the ANSYS SCADE product family (for “safety-critical application development environment”) is used to generate code by companies in the aerospace and defense industries, in nuclear power plants, transit systems, heavy industry, and medical devices. “My initial dream was to have SCADE-generated code in every plane in the world,” Bantégnie, the founder of Esterel Technologies, says, “and we’re not very far off from that objective.” Nearly all safety-critical code on the Airbus A380, including the system controlling the plane’s flight surfaces, was generated with ANSYS SCADE products.

Part of the draw for customers, especially in aviation, is that while it is possible to build highly reliable software by hand, it can be a Herculean effort. Ravi Shivappa, the VP of group software engineering at Meggitt PLC, an ANSYS customer which builds components for airplanes, like pneumatic fire detectors for engines, explains that traditional projects begin with a massive requirements document in English, which specifies everything the software should do. (A requirement might be something like, “When the pressure in this section rises above a threshold, open the safety valve, unless the manual-override switch is turned on.”) The problem with describing the requirements this way is that when you implement them in code, you have to painstakingly check that each one is satisfied. And when the customer changes the requirements, the code has to be changed, too, and tested extensively to make sure that nothing else was broken in the process.

The cost is compounded by exacting regulatory standards. The FAA is fanatical about software safety. The agency mandates that every requirement for a piece of safety-critical software be traceable to the lines of code that implement it, and vice versa. So every time a line of code changes, it must be retraced to the corresponding requirement in the design document, and you must be able to demonstrate that the code actually satisfies the requirement. The idea is that if something goes wrong, you’re able to figure out why; the practice brings order and accountability to large codebases. But, Shivappa says, “it’s a very labor-intensive process.” He estimates that before they used model-based design, on a two-year-long project only two to three months was spent writing code—the rest was spent working on the documentation.

As Bantégnie explains, the beauty of having a computer turn your requirements into code, rather than a human, is that you can be sure—in fact you can mathematically prove—that the generated code actually satisfies those requirements. Much of the benefit of the model-based approach comes from being able to add requirements on the fly while still ensuring that existing ones are met; with every change, the computer can verify that your program still works. You’re free to tweak your blueprint without fear of introducing new bugs. Your code is, in FAA parlance, “correct by construction.”

Still, most software, even in the safety-obsessed world of aviation, is made the old-fashioned way, with engineers writing their requirements in prose and programmers coding them up in a programming language like C. As Bret Victor made clear in his essay, model-based design is relatively unusual. “A lot of people in the FAA think code generation is magic, and hence call for greater scrutiny,” Shivappa told me.

Most programmers feel the same way. They like code. At least they understand it. Tools that write your code for you and verify its correctness using the mathematics of “finite-state machines” and “recurrent systems” sound esoteric and hard to use, if not just too good to be true.

It is a pattern that has played itself out before. Whenever programming has taken a step away from the writing of literal ones and zeros, the loudest objections have come from programmers. Margaret Hamilton, a celebrated software engineer on the Apollo missions—in fact the coiner of the phrase “software engineering”—told me that during her first year at the Draper lab at MIT, in 1964, she remembers a meeting where one faction was fighting the other about transitioning away from “some very low machine language,” as close to ones and zeros as you could get, to “assembly language.” “The people at the lowest level were fighting to keep it. And the arguments were so similar: ‘Well how do we know assembly language is going to do it right?’”

“Guys on one side, their faces got red, and they started screaming,” she said. She said she was “amazed how emotional they got.”

Emmanuel Ledinot, of Dassault Aviation, pointed out that when assembly language was itself phased out in favor of the programming languages still popular today, like C, it was the assembly programmers who were skeptical this time. No wonder, he said, that “people are not so easily transitioning to model-based software development: They perceive it as another opportunity to lose control, even more than they have already.”

The bias against model-based design, sometimes known as model-driven engineering, or MDE, is in fact so ingrained that according to a recent paper, “Some even argue that there is a stronger need to investigate people’s perception of MDE than to research new MDE technologies.”

Which sounds almost like a joke, but for proponents of the model-based approach, it’s an important point: We already know how to make complex software reliable, but in so many places, we’re choosing not to. Why?

In 2011, Chris Newcombe had been working at Amazon for almost seven years, and had risen to be a principal engineer. He had worked on some of the company’s most critical systems, including the retail-product catalog and the infrastructure that managed every Kindle device in the world. He was a leader on the highly prized Amazon Web Services team, which maintains cloud servers for some of the web’s biggest properties, like Netflix, Pinterest, and Reddit. Before Amazon, he’d helped build the backbone of Steam, the world’s largest online-gaming service. He is one of those engineers whose work quietly keeps the internet running. The products he’d worked on were considered massive successes. But all he could think about was that buried deep in the designs of those systems were disasters waiting to happen.

“Human intuition is poor at estimating the true probability of supposedly ‘extremely rare’ combinations of events in systems operating at a scale of millions of requests per second,” he wrote in a paper. “That human fallibility means that some of the more subtle, dangerous bugs turn out to be errors in design; the code faithfully implements the intended design, but the design fails to correctly handle a particular ‘rare’ scenario.”

Newcombe was convinced that the algorithms behind truly critical systems—systems storing a significant portion of the web’s data, for instance—ought to be not just good, but perfect. A single subtle bug could be catastrophic. But he knew how hard bugs were to find, especially as an algorithm grew more complex. You could do all the testing you wanted and you’d never find them all.

This is why he was so intrigued when, in the appendix of a paper he’d been reading, he came across a strange mixture of math and code—or what looked like code—that described an algorithm in something called “TLA+.” The surprising part was that this description was said to be mathematically precise: An algorithm written in TLA+ could in principle be proven correct. In practice, it allowed you to create a realistic model of your problem and test it not just thoroughly, but exhaustively. This was exactly what he’d been looking for: a language for writing perfect algorithms.

TLA+, which stands for “Temporal Logic of Actions,” is similar in spirit to model-based design: It’s a language for writing down the requirements—TLA+ calls them “specifications”—of computer programs. These specifications can then be completely verified by a computer. That is, before you write any code, you write a concise outline of your program’s logic, along with the constraints you need it to satisfy (say, if you were programming an ATM, a constraint might be that you can never withdraw the same money twice from your checking account). TLA+ then exhaustively checks that your logic does, in fact, satisfy those constraints. If not, it will show you exactly how they could be violated.

The language was invented by Leslie Lamport, a Turing Award–winning computer scientist. With a big white beard and scruffy white hair, and kind eyes behind large glasses, Lamport looks like he might be one of the friendlier professors at the American Hogwarts. Now at Microsoft Research, he is known as one of the pioneers of the theory of “distributed systems,” which describes any computer system made of multiple parts that communicate with each other. Lamport’s work laid the foundation for many of the systems that power the modern web.

For Lamport, a major reason today’s software is so full of bugs is that programmers jump straight into writing code. “Architects draw detailed plans before a brick is laid or a nail is hammered,” he wrote in an article. “But few programmers write even a rough sketch of what their programs will do before they start coding.” Programmers are drawn to the nitty-gritty of coding because code is what makes programs go; spending time on anything else can seem like a distraction. And there is a patient joy, a meditative kind of satisfaction, to be had from puzzling out the micro-mechanics of code. But code, Lamport argues, was never meant to be a medium for thought. “It really does constrain your ability to think when you’re thinking in terms of a programming language,” he says. Code makes you miss the forest for the trees: It draws your attention to the working of individual pieces, rather than to the bigger picture of how your program fits together, or what it’s supposed to do—and whether it actually does what you think. This is why Lamport created TLA+. As with model-based design, TLA+ draws your focus to the high-level structure of a system, its essential logic, rather than to the code that implements it.

Newcombe and his colleagues at Amazon would go on to use TLA+ to find subtle, critical bugs in major systems, including bugs in the core algorithms behind S3, regarded as perhaps the most reliable storage engine in the world. It is now used widely at the company. In the tiny universe of people who had ever used TLA+, their success was not so unusual. An intern at Microsoft used TLA+ to catch a bug that could have caused every Xbox in the world to crash after four hours of use. Engineers at the European Space Agency used it to rewrite, with 10 times less code, the operating system of a probe that was the first to ever land softly on a comet. Intel uses it regularly to verify its chips.

But TLA+ occupies just a small, far corner of the mainstream, if it can be said to take up any space there at all. Even to a seasoned engineer like Newcombe, the language read at first as bizarre and esoteric—a zoo of symbols. For Lamport, this is a failure of education. Though programming was born in mathematics, it has since largely been divorced from it. Most programmers aren’t very fluent in the kind of math—logic and set theory, mostly—that you need to work with TLA+. “Very few programmers—and including very few teachers of programming—understand the very basic concepts and how they’re applied in practice. And they seem to think that all they need is code,” Lamport says. “The idea that there’s some higher level than the code in which you need to be able to think precisely, and that mathematics actually allows you to think precisely about it, is just completely foreign. Because they never learned it.”

Lamport sees this failure to think mathematically about what they’re doing as the problem of modern software development in a nutshell: The stakes keep rising, but programmers aren’t stepping up—they haven’t developed the chops required to handle increasingly complex problems. “In the 15th century,” he said, “people used to build cathedrals without knowing calculus, and nowadays I don’t think you’d allow anyone to build a cathedral without knowing calculus. And I would hope that after some suitably long period of time, people won’t be allowed to write programs if they don’t understand these simple things.”

Newcombe isn’t so sure that it’s the programmer who is to blame. “I’ve heard from Leslie that he thinks programmers are afraid of math. I’ve found that programmers aren’t aware—or don’t believe—that math can help them handle complexity. Complexity is the biggest challenge for programmers.” The real problem in getting people to use TLA+, he said, was convincing them it wouldn’t be a waste of their time. Programmers, as a species, are relentlessly pragmatic. Tools like TLA+ reek of the ivory tower. When programmers encounter “formal methods” (so called because they involve mathematical, “formally” precise descriptions of programs), their deep-seated instinct is to recoil.

Most programmers who took computer science in college have briefly encountered formal methods. Usually they’re demonstrated on something trivial, like a program that counts up from zero; the student’s job is to mathematically prove that the program does, in fact, count up from zero.

“I needed to change people’s perceptions on what formal methods were,” Newcombe told me. Even Lamport himself didn’t seem to fully grasp this point: Formal methods had an image problem. And the way to fix it wasn’t to implore programmers to change—it was to change yourself. Newcombe realized that to bring tools like TLA+ to the programming mainstream, you had to start speaking their language.

For one thing, he said that when he was introducing colleagues at Amazon to TLA+ he would avoid telling them what it stood for, because he was afraid the name made it seem unnecessarily forbidding: “Temporal Logic of Actions” has exactly the kind of highfalutin ring to it that plays well in academia, but puts off most practicing programmers. He tried also not to use the terms “formal,” “verification,” or “proof,” which reminded programmers of tedious classroom exercises. Instead, he presented TLA+ as a new kind of “pseudocode,” a stepping-stone to real code that allowed you to exhaustively test your algorithms—and that got you thinking precisely early on in the design process. “Engineers think in terms of debugging rather than ‘verification,’” he wrote, so he titled his internal talk on the subject to fellow Amazon engineers “Debugging Designs.” Rather than bemoan the fact that programmers see the world in code, Newcombe embraced it. He knew he’d lose them otherwise. “I’ve had a bunch of people say, ‘Now I get it,’” Newcombe says.

He has since left Amazon for Oracle, where he’s been able to convince his new colleagues to give TLA+ a try. For him, using these tools is now a matter of responsibility. “We need to get better at this,” he said.

“I’m self-taught, been coding since I was nine, so my instincts were to start coding. That was my only—that was my way of thinking: You’d sketch something, try something, you’d organically evolve it.” In his view, this is what many programmers today still do. “They google, and they look on Stack Overflow” (a popular website where programmers answer each other’s technical questions) “and they get snippets of code to solve their tactical concern in this little function, and they glue it together, and iterate.”

“And that’s completely fine until you run smack into a real problem.”

In the summer of 2015, a pair of American security researchers, Charlie Miller and Chris Valasek, convinced that car manufacturers weren’t taking software flaws seriously enough, demonstrated that a 2014 Jeep Cherokee could be remotely controlled by hackers. They took advantage of the fact that the car’s entertainment system, which has a cellular connection (so that, for instance, you can start your car with your iPhone), was connected to more central systems, like the one that controls the windshield wipers, steering, acceleration, and brakes (so that, for instance, you can see guidelines on the rearview screen that respond as you turn the wheel). As proof of their attack, which they developed on nights and weekends, they hacked into Miller’s car while a journalist was driving it on the highway, and made it go haywire; the journalist, who knew what was coming, panicked when they cut the engines, forcing him to a slow crawl on a stretch of road with no shoulder to escape to.

Although they didn’t actually create one, they showed that it was possible to write a clever piece of software, a “vehicle worm,” that would use the onboard computer of a hacked Jeep Cherokee to scan for and hack others; had they wanted to, they could have had simultaneous access to a nationwide fleet of vulnerable cars and SUVs. (There were at least five Fiat Chrysler models affected, including the Jeep Cherokee.) One day they could have told them all to, say, suddenly veer left or cut the engines at high speed.

“We need to think about software differently,” Valasek told me. Car companies have long assembled their final product from parts made by hundreds of different suppliers. But where those parts were once purely mechanical, they now, as often as not, come with millions of lines of code. And while some of this code—for adaptive cruise control, for auto braking and lane assist—has indeed made cars safer (“The safety features on my Jeep have already saved me countless times,” says Miller), it has also created a level of complexity that is entirely new. And it has made possible a new kind of failure.

“There are lots of bugs in cars,” Gerard Berry, the French researcher behind Esterel, said in a talk. “It’s not like avionics—in avionics it’s taken very seriously. And it’s admitted that software is different from mechanics.” The automotive industry is perhaps among those that haven’t yet realized they are actually in the software business.

“We don’t in the automaker industry have a regulator for software safety that knows what it’s doing,” says Michael Barr, the software expert who testified in the Toyota case. NHTSA, he says, “has only limited software expertise. They’ve come at this from a mechanical history.” The same regulatory pressures that have made model-based design and code generation attractive to the aviation industry have been slower to come to car manufacturing. Emmanuel Ledinot, of Dassault Aviation, speculates that there might be economic reasons for the difference, too. Automakers simply can’t afford to increase the price of a component by even a few cents, since it is multiplied so many millionfold; the computers embedded in cars therefore have to be slimmed down to the bare minimum, with little room to run code that hasn’t been hand-tuned to be as lean as possible. “Introducing model-based software development was, I think, for the last decade, too costly for them.”

One suspects the incentives are changing. “I think the autonomous car might push them,” Ledinot told me—“ISO 26262 and the autonomous car might slowly push them to adopt this kind of approach on critical parts.” (ISO 26262 is a safety standard for cars published in 2011.) Barr said much the same thing: In the world of the self-driving car, software can’t be an afterthought. It can’t be built like today’s airline-reservation systems or 911 systems or stock-trading systems. Code will be put in charge of hundreds of millions of lives on the road and it has to work. That is no small task.

“Computing is fundamentally invisible,” Gerard Berry said in his talk. “When your tires are flat, you look at your tires, they are flat. When your software is broken, you look at your software, you see nothing.”

“So that’s a big problem.”

* This article originally stated that there were 10 million ways for the Toyota Camry to cause unintended acceleration. We regret the error.

The dispatcher said, "Wait a minute. Everglades Park?"

"Everglades Holiday Park, along canal L-sixty-seven. You need to get your choppers in the air. I'm a pilot. I have a GPS. I'll give you coordinates."

"Okay, sir. What kind of plane did you say? Is it a large plane?"

"A large aircraft similar to a seven-twenty-seven or a umm ... I can't think of it."

This lapse was unimportant. The caller was a born accident observer—a computer engineer and a private pilot with pride in his technical competence and a passion for detail. His name was Walton Little. When he first saw the airplane, it was banked steeply to the right and flying low, just above the swamp. Later he filed an official report, in which he stated,

Nearby fishermen ducked into their boat for cover—but not Walton Little, who stood on his deck, facing "about 115 degrees," and watched the airplane hit the water. The shock wave passed through his body.

He called within a minute. After telling the dispatcher about the crash and reading off his latitude and longitude, he said, "I'm in a bass boat on the canal. I thought it was an aircraft from an air show or something, and..."

The dispatcher interrupted. "What did you ... Did you see flames and stuff come up, sir?"

"I heard the impact, and I saw dirt and mud fly in the air. The plane was sideways before it went out of my sight on the horizon about a mile from me."

"Yes, sir. Okay. You said it looked like a seven-twenty-seven that went down?"

"Uh, it's that type aircraft. It has twin engines in the rear. It is larger than an executive jet, like a Learjet."

"Yes, sir."

"It's much bigger than that. I won't tell you it's a seven-twenty-seven, but it's that type aircraft. No engines on the wing, two engines in the rear. I do not see any smoke, but I saw a tremendous cloud of mud and dirt go into the sky when it hit."

"Okay, sir."

"It was white with blue trim."

"White with blue trim, sir?"

"It will not be in one piece."

Walton Little was right. The airplane was a twin-engine DC-9 painted the colors of ValuJet, an aggressive young discount airline based in Atlanta. When it hit the Everglades, it was banked vertically to the right and pointed nearly straight down. The airplane did not sink mysteriously into the swamp, as reports later suggested, but shattered as it hit the surface with the furious force of a fast dive.

By the time Walton Little felt the shock wave, everyone aboard was dead—two pilots, three flight attendants, and 105 passengers. Their remains lay in a shallow, watery crater filled with liquid mud and grass. All that marked the surface was a fractured engine, a few dead fish, some jet fuel, and a scattering of personal papers, clothes, and twisted pieces of aluminum—the stuff of tragedy. During those first few days some officials worried aloud about the accident's effect on nature, but the swamp was not so fragile as that, and quickly resumed its usual life. The families of those who died have proved less resilient. Most will feel the poison forever.

For the rest of us, though, the accident should be finished business. The official investigation is over, a "cause" has been found, contributing factors have been acknowledged, and the Federal Aviation Administration has written new regulations. Editorialists have expressed their outrage, and individuals have been held responsible. After a long suspension ValuJet has returned to the air with a renewed commitment to safety. Other airlines, too, have promised to be more careful. And even the FAA has gone through a housecleaning. So by conventional standards the reaction to the tragedy has been admirable. And yes, we know anyway that flying is almost always safe. After years as a working pilot, I have a poetic idea of why: airplanes are fundamentally at home in the sky. Certainly my own experience is that passengers do not need to cower around the exit rows, or carry emergency "smoke hoods," or fear bad weather, or worry about some impending collapse of airline safety. Those are ideas promoted by aviation illiterates—overly cautious people who can always find an audience, and who would smother us in their fear of violent death. The public has the sense in the long run to ignore them. Nonetheless, the ValuJet accident continues to raise troubling questions—no longer about what happened but about why it happened, and what is to keep something similar from happening in the future. As these questions lead into the complicated and human core of flight safety, they become increasingly difficult to answer.

Consider, for simplicity, that there are three kinds of airplane accidents. The most common ones might be called "procedural." They are those old-fashioned accidents that result from single obvious mistakes, that can immediately be understood in simple terms, and that have simple resolutions. To avoid such accidents pilots must not fly into violent thunderstorms, or take off with ice on their wings, or descend prematurely, or let fear or boredom gain the upper hand. Mechanics, ramp agents, and air-traffic controllers must observe equally simple rules. As practitioners, we have together learned many painful lessons.

The second kind of accident could be called "engineered." It consists of those surprising materials failures that should have been predicted by designers or discovered by test pilots but were not. Such failures at first defy understanding, but ultimately they yield to examination and result in tangible solutions. An American Eagle ATR turboprop dives into a frozen field in Roselawn, Indiana, because its de-icing boots did not protect its wings from freezing rain—and as a result new boots are designed, and the entire testing process undergoes review. A USAir Boeing 737 crashes near Pittsburgh because of a rare hard-over rudder movement—and as a result a redesigned rudder-control mechanism will be installed on the whole fleet. A TWA Boeing 747 blows apart off New York because, whatever the source of ignition, its nearly empty center tank contained an explosive mixture of fuel and air—and as a result explosive mixtures may in the future be avoided. Such tragic failures seem all too familiar, but in fact they are rare, and they will grow rarer still as aeronautical engineering improves. One can regret the lives lost and deplore the slowness with which officials respond, but in the long run there is reason to be optimistic. The Wright brothers were products of the Enlightenment. Our science will prevail.

The ValuJet accident is different. I would argue that it represents the third and most elusive kind of disaster, a "system accident," which may lie beyond the reach of conventional solution, and which a small group of thinkers, inspired by the Yale sociologist Charles Perrow, has been exploring elsewhere—for example, in power generation, chemical manufacturing, nuclear-weapons control, and space flight. Perrow has coined the more loaded term "normal accident" for such disasters, because he believes that they are normal for our time. His point is that these accidents are science's illegitimate children, bastards born of the confusion that lies within the complex organizations with which we manage our dangerous technologies. Perrow is not an expert on commercial flying, but his thinking applies to it nonetheless. In this case the organization includes not only ValuJet, the archetype of new-style airlines, but also the contractors that serve it and the government entities that, despite economic deregulation, are expected to oversee it. Taken as a whole, the airline system is complex indeed.

Keep in mind that it is also competitive, and that if one of its purposes is to make money, the other is to move the public through thin air cheaply and at high speed. Safety is never first, and it never will be, but for obvious reasons it is a necessary part of the venture. Risk is a part too, but on the everyday level of practical compromises and small decisions—the building blocks of this ambitious enterprise—the view of risk is usually obscured. The people involved do not consciously trade safety for money or convenience, but they inevitably make a lot of bad little choices. They get away with those choices because, as Perrow says, Murphy's Law is wrong—what can go wrong usually goes right. But then one day a few of the bad little choices come together, and circumstances take an airplane down. Who, then, is really to blame?

We can find fault among those directly involved—and we probably need to. But if our purpose is to attack the roots of such an accident, we may find them so entwined with the system that they are impossible to extract without toppling the whole structure. In the case of ValuJet the study of system accidents presents us with the possibility that we have come to depend on flight, that unless we are willing to end our affordable airline system as we know it, we cannot stop the occasional sacrifice. Beyond the questions of blame, it requires us to consider that our solutions, by adding to the complexity and obscurity of the airline business, may actually increase the risk of accidents. System-accident thinking does not demand that we accept our fate without a struggle, but it serves as an important caution.

It was headed from Miami to Atlanta, flown by Captain Candalyn Kubeck, age thirty-five, and her copilot Richard Hazen, age fifty-two. They represented a new kind of commercial pilot, experienced not only in the cockpit but in the rough-and-tumble of the deregulated airline industry, where both had held a number of low-paid flying jobs before settling on ValuJet. It would have been no shock to them that ValuJet pilots were non-unionized, or that the company required them to pay for their own training. With 9,000 flight hours behind her, more than 2,000 of them in a DC-9, Kubeck earned what the free market said she was worth—about $43,000 a year, plus bonuses. Hazen, formerly in the Air Force and with similar experience, earned a bit more than half as much.

Pilots were not the only low-paid employees at ValuJet—flight attendants, ramp agents, and mechanics made a lot less there than they would have at a more traditional airline. So much work was farmed out to temporary employees and independent contractors that ValuJet was sometimes called a "virtual airline." FAA regulators had begun to worry that the company was moving too fast, and not keeping up with its paperwork, but there was no evidence that the people involved were inadequate. Many of the pilots were refugees from the labor wars at the old Eastern Airlines, and they were generally as competent and experienced as their higher-paid friends at United, American, and Delta. ValuJet was helping the entire industry to understand just how far cost-cutting could be pushed. Its flights were cheap and full, and its stock was strong on Wall Street.

But six minutes out of Miami, while climbing northwest through 11,000 feet, Richard Hazen radioed, "Ah, five-ninety-two needs an immediate return to Miami." In the deliberate calm of pilot talk this was strong language. The time was thirty-one seconds after 2:10 P.M., and the sun was shining. Something had gone wrong with the airplane.

The radar controller at Miami Departure answered immediately. Using ValuJet's radio name "Critter" (for the company's cartoonish logo—a smiling airplane), he gave the flight clearance to turn initially toward the west, away from Miami and conflicting traffic flows, and to begin a descent to the airport. "Critter five-ninety-two, ah roger, turn left heading two-seven-zero, descend and maintain seven thousand."

Hazen said, "Two-seven-zero, seven thousand, five-ninety-two."

The controller was Jesse Fisher, age thirty-six, a seven-year veteran, who had twice handled the successful return of an airliner that had lost cabin pressurization. He had worked the night before, and had gone home, fed his cat, and slept well. He felt alert and rested. He said, "What kind of problem are you having?"

Hazen said, "Ah, smoke in the cockpit. Smoke in the cabin." His tone was urgent.

Fisher kept his own tone flat. He said, "Roger." Over his shoulder he called, "I need a supervisor here!"

The supervisor plugged in beside him. On Fisher's radar screen Flight 592 appeared as a little oval and an associated group of numbers, including a readout of its altitude. Fisher noticed that the airplane had not yet started to turn. He gave the pilots another heading, farther to the left, and cleared them down to 5,000 feet.

Aboard the airplane Hazen acknowledged the new heading but misheard the altitude assignment. It didn't matter. Flight 592 was burning, and the situation in the cockpit was rapidly getting out of hand. One minute into the emergency the pilots were still tracking away from Miami, and had not begun their return. Hazen said, "Critter five-ninety-two, we need the, ah, closest airport available."

The transmission was garbled or blocked, or Fisher was distracted by competing voices within the radar room. For whatever reason, he did not hear Hazen's request. When investigators later asked him if in retrospect he would have done anything differently, he admitted that he kept asking himself the same question. Even without hearing Hazen's request he might have suggested some slightly closer airport. But given that the flight's position was only twenty-five miles to the northwest, Miami still seemed like the best choice, because of the emergency equipment there. In any case "Miami" was the request he had heard, and he intended to deliver it.

To Hazen he said, "Critter five-ninety-two, they're gonna be standing, standing by for you." He meant the crash crews at Miami. "You can plan Runway One-two. When able, direct to Dolphin now."

Hazen said, "... need radar vectors." His transmission was garbled by loud background noises. Fisher thought he sounded "shaky."

Fisher answered, "Critter five-ninety-two, turn left heading one-four-zero."

Hazen said, "One-four-zero." It was his last coherent response.

The flight had only now begun to move through a gradual left turn. Fisher watched the target on his screen as it tracked through the heading changes: the turn tightened and then slowed again. With each sweep of the radar beam the altitude readouts showed a gradual descent—8,800, 8,500, 8,100. Two minutes into the crisis Fisher said, "Critter five-ninety-two, keep the turn around, heading ah one-two-zero."

Flight 592 may have tried to respond—someone keyed a microphone without talking.

Fisher said, "Critter five-ninety-two, contact Miami Approach on—correction, no, you just keep on my frequency."

Two and a half minutes had gone by. It was 2:13 P.M. The airplane was passing through 7,500 feet when suddenly it tightened the left turn and entered a steep dive. Fisher's radar showed the turn and an altitude readout of XXX—code for such a rapid altitude change that the computer cannot keep up. Investigators later calculated that the airplane rolled to a sixty-degree left bank and dove 6,400 feet in thirty-two seconds. During that loss of control Fisher radioed mechanically, "Critter five-ninety-two, you can, ah, turn left, heading one-zero-zero, and join the Runway One-two localizer at Miami." He also radioed, "Critter five-ninety-two, descend and maintain three thousand."

Then the incredible happened. The airplane rolled wings-level again and pulled sharply out of its dive. It is highly unlikely that the airplane would have done this on its own. It is possible that the autopilot kicked in, or that one of the pilots, having been incapacitated by smoke or defeated by melting control cables, somehow momentarily regained control. Fisher watched the radar target straighten toward the southeast, and again read out a nearly level altitude—now, however, merely a thousand feet. The airplane's speed was almost 500 miles an hour.

The frequency crackled with another unintelligible transmission. Shocked into the realization that the airplane would be unable to make Miami, Fisher said, "Critter five-ninety-two, Opa-Locka Airport's about ah twelve o'clock at fifteen miles."

Walton Little, in his bass boat, spotted the airplane then, as it rolled steeply to the right. The radar, too, noticed that last quick turn toward the south, just before the final nose-over. On the next sweep of the radar the flight's data block went into "coast" on Fisher's screen, indicating that contact had been lost. The supervisor marked the spot electronically and launched rescue procedures.

Fisher continued to work the other airplanes in his sector. Five minutes after the impact another low-paid pilot, this one for American Eagle, radioed, "Ah, how did Critter make out?" Fisher didn't answer.

It is not reasonable to blame them for this, though. The NTSB is a technical agency, staffed by technicians, which occupies a central position in the stilted world of aviation. Its job is to examine important accidents and to issue nonbinding safety recommendations—opinions, really—to industry and government. Because the investigators have no regulatory authority and must rely on persuasion to influence events, it may at times be necessary for them to use official-sounding language. Even among its opponents, who often feel that its recommendations are impractical, the NTSB has a reputation for technical competence. The NTSB is a piece of engineering done right. In a world built on compromise, it manages to play the old-fashioned, unambiguous role of the public's defender.

The press plays a more difficult role, though one equally important to the public's safety. It has a classically symbiotic relationship with the NTSB, relying on the investigators for information while providing them with their only effective voice. Nonetheless, in the time of crisis immediately after an accident, a tension exists between the two. Working under pressure to get the story out, reporters resent the caution of the investigators and their reluctance to speculate anonymously. Working under pressure to get the story right, investigators, for their part, resent the reporters' incessant demands during the difficult first days of an accident probe—the recovery of human remains and airplane parts. By the time I got to Miami, nineteen hours after Flight 592 hit the swamp, the two camps had assumed their habitual positions and were passing each other warily in the hotel lobby.

Twenty miles to the northwest, deep in the Everglades, the recovery operation was already under way. The NTSB had set up a staging area—a "forward ops base," one official called it—beside the Tamiami Trail, a two-lane highway that traverses the watery grasslands of southern Florida. Within two days this staging area blossomed into a chaotic encampment of excited officials—local, state, and federal—with their tents and air-conditioned trailers, their helicopters, their cars and flashing lights. I quit counting the agencies. The NTSB had politely excluded most of them from the actual accident site, which lay seven miles north, along a narrow levee road.

The press was excluded even from the staging area, but was provided with two news conferences a day, during which investigators cautiously doled out tidbits of information. One NTSB official said to me, "We've got to feed them or we'll lose control." But the reporters were well behaved, and if anything a bit overcivilized. Near the staging area they settled into their own little town of television trucks, tents, and lawn chairs. The location gave them good Everglades backdrops and shots of alligators swimming by; the viewing public could not have guessed that they stood so far from the action. They acted impatient, but in truth this was not a bad assignment; at its peak their little town boasted pay phones and pizza delivery.

Maybe it was because of my obvious lack of deadline that the investigators made an exception in my case. They slipped me into the front seat of a Florida Game and Fish helicopter whose pilot, in a fraternal gesture, invited me to take the controls for the run out to the crash site. From the staging area we skimmed north across the swamped grasslands, loosely following the levee road, before swinging wide to circle over the impact zone—a new pond defined by a ring of turned mud and surrounded by a larger area of grass and water and accident debris. Searchers in white protective suits waded side by side through the muck, piling pieces of people and airplane into flat-bottomed boats. It was hot and unpleasant work performed in a contained little hell, a place that one investigator later described to me as reeking of fuel, earth, and rotting flesh—the special smell of an airplane accident. We descended onto the levee, about 300 yards away from the crash site, where an American flag and a few tents and trucks constituted the recovery base.

The mood there was quiet and purposeful, with no sign among the workers of the emotional trauma that officials had been worriedly predicting since the operation began. The workers on break sat in the shade of an awning, sipping cold drinks and chatting. They were policemen and firemen, not heroes but straightforward guys accustomed to confronting death. Not knowing who I was, they spoke to me frankly about the gruesome details of their work, and made indelicate jokes, but they seemed more worried about dehydration than about "taking the job home" or losing sleep. I relaxed in their company, relieved to have escaped for a while the expectation of grief.

It was, of course, a somber place to be. Human remains lay bagged in a refrigerated truck for later transport to the morgue. A decontamination crew washed down torn and twisted pieces of airplane, none longer than several feet. Investigators tagged the most promising wreckage, to be trucked immediately to a hangar at an outlying Miami airport, where specialists could study it. Farther down the levee I came upon a soiled photograph of a young woman with a small-town face and a head of teased hair. A white-suited crew arrived on an airboat and clambered up the embankment to be washed down. Another crew set off. A boatload of muddy wreckage arrived. The next day the families of the dead came on buses, and laid flowers and cried. Pieces of the airplane kept being hauled up for nearly another month.

Much was made of this recovery, which—prior to the offshore retrieval of TWA's Flight 800—the NTSB called the most challenging in its history. It is true that the swamp made the search slow and difficult, and that the violence of the impact meant that meticulous work was required to reconstruct the critical forward cargo hold. However, it is also true that the physical part of the investigation served to confirm what a look at a shipping ticket had already suggested—that ValuJet Flight 592 burned and crashed not because the airplane failed but, in large part, because the airline did.

To me as a pilot, the most impressive aspect of the investigation was the speed with which it worked through the false pursuit of an electrical fire—an explanation supported by my own experiences in flight, and all the more plausible here because the ValuJet DC-9 was old and had experienced a variety of electrical failures earlier the same day, including a tripped circuit breaker that had resisted the attentions of a mechanic in Atlanta, and then mysteriously had fixed itself. I was impressed also by the instincts of the reporters, who for all their technical ignorance seized on the news that Flight 592 had been loaded with a potentially dangerous cargo of chemical oxygen generators—more than a hundred little firebombs that could have caused this accident, and that indeed did.

Flight 592 crashed on a Saturday afternoon. By Sunday the recovery teams were pulling up scorched and soot-stained pieces. On Monday a searcher happened to step on the flight-data recorder, one of two required black boxes meant to help with accident investigations. The NTSB took the recorder to its Washington laboratory and found that a blip in the flight data six minutes after Flight 592's takeoff seemed to indicate a momentary rise in air pressure. Immediately afterward the recorder began to fail intermittently, apparently because of electrical-power interruptions. On Tuesday night, at a press conference at the hotel, Robert Francis, the vice-chairman of the NTSB and the senior official on the scene, announced in a monotone, "There could have been an explosion." A hazardous-materials team would be joining the investigation. The investigation was focusing on the airplane's forward cargo hold, which was located just below and behind the cockpit, and was unequipped with fire detection and extinguishing systems. Routine paperwork indicated that the Miami ground crew had loaded the hold with homeward-bound ValuJet "company material," a witch's brew of three tires—at least two of them mounted—and five cardboard boxes of old oxygen generators.

It is ironic that the airplane's own emergency-oxygen system was different—a set of simple oxygen tanks, similar to those used in hospitals, that do not emit heat during use. The oxygen generators in Flight 592's forward cargo hold came from three MD-80s, a more modern kind of twin jet, which ValuJet had recently bought and was having refurbished at a hangar across the airport in Miami. As was its practice for most maintenance, ValuJet had hired an outside company to do the job—in this case a large firm called SabreTech, owned by Sabreliner, of St. Louis, and licensed by the FAA to perform the often critical work. SabreTech, in turn, hired contract mechanics from other companies on an as-needed basis. It later turned out that three fourths of the people on the project were just such temporary outsiders. The vulnerability of American wageworkers could be sensed in their testimony after the accident. They inhabited a world of boss men and sudden firings, with few protections or guarantees for the future. As the ValuJet deadline approached, they worked in shifts, day and night, and sometimes through the weekend as well. It was their contribution to our cheap flying.

We will never know everyone at fault in this story. ValuJet gave the order to replace oxygen generators on the MD-80s, most of which had come to the end of their licensed lifetimes. It provided SabreTech with explicit removal procedures and general warnings about the dangers of fire. Over several weeks SabreTech workers extracted the generators and taped or cut off their lanyards before stacking most of them in five cardboard boxes that happened to be lying around the hangar. Apparently they believed that securing the lanyards would keep the generators from being fired inadvertently. What they did not do was place the required plastic safety caps over the firing pins—a precaution spelled out on the second line of ValuJet's written work order. The problem for SabreTech was that no one had such caps, or cared much about finding them. Ultimately the caps were forgotten or ignored. At the end of the job, in the rush to complete batches of paperwork on all three MD-80s, two mechanics routinely "pencil-whipped" the problem by signing off on the safety-cap line as well as on the others, certifying that the work had been done. SabreTech inspectors and supervisors signed off on the work too, apparently without giving the caps much thought.

The timing is not clear. For weeks the five boxes stood on a parts rack beside the airplanes. Eventually mechanics lugged them over to SabreTech's shipping-and-receiving department, where they sat on the floor in the area designated for ValuJet property. A few days before the accident a SabreTech manager told the shipping clerk to clean up the area and get all the boxes off the floor in preparation for an upcoming inspection by Continental Airlines, a potential customer. The boxes were unmarked, and the manager did not care what was in them.

The shipping clerk then did what shipping clerks do, and prepared to send the oxygen generators home to ValuJet headquarters, in Atlanta. He redistributed them equally among the five boxes, laying the canisters horizontally end to end, and packing bubble wrap on top. After sealing the boxes he applied address labels and ValuJet company-material stickers, and wrote "aircraft parts." As part of the load he included two large main tires and a smaller nose tire—at least two of which were mounted on wheels. The next day he asked a co-worker, the receiving clerk, to make out a shipping ticket, and to write "oxygen canisters—empty" on it. The receiving clerk wrote "Oxy Canisters" and then put "Empty" between quotation marks, as if he did not believe it. He also listed the tires.

The cargo stood for another day or two, until May 11, when the SabreTech driver had time to deliver the boxes across the airport to Flight 592. There the ValuJet ramp agent accepted the material, though federal regulations forbade him to, even if the generators were empty, since canisters that have been discharged contain a toxic residue, and ValuJet was not licensed to carry any such officially designated hazardous materials. He discussed the cargo's weight with the copilot, Richard Hazen, who also should have known better. Together they decided to place the load in the forward hold, where ValuJet workers laid one of the big main tires flat, placed the nose tire at the center of it, and stacked the five boxes on top of it around the outer edge, in a loose ring. They leaned the other main tire against a bulkhead. It was an unstable arrangement. No one knows exactly what happened then, but it seems likely that the first oxygen generator ignited during the loading or during taxiing or on takeoff, as the airplane climbed skyward.

Two weeks later and halfway through the recovery of the scorched and shattered parts a worker finally found the airplane's cockpit voice recorder, the second black box sought by the investigators. It had recorded normal sounds and conversation up to the moment—six minutes after takeoff—when the flight-data recorder indicated a pulse of high pressure. The pulse may have been one of the tires exploding. In the cockpit it sounded like a chirp and a simultaneous beep on the public-address system. The captain, Candalyn Kubeck, asked, "What was that?"

Hazen said, "I don't know."

They scanned the airplane's instruments and found sudden indications of electrical failure. It was not the cause but a symptom of the inferno in the hold—the wires and electrical panels were probably melting and burning along with other, more crucial parts of the airplane—but the pilots' first thought was that the airplane was merely up to its circuit-breaking tricks again. The recording here is garbled. Kubeck seems to have asked, "About to lose a bus?" Then, more clearly, she said, "We've got some electrical problem."

Hazen said, "Yeah. That battery charger's kickin' in. Oooh, we gotta ..."

"We're losing everything," Kubeck said. "We need, we need to go back to Miami."

Twenty seconds had passed since the strange chirp in the cockpit. A total electrical failure, though serious, was not in those sunny conditions a life-threatening emergency. But suddenly there was incoherent shouting from the passenger cabin, and women and men screaming, "Fire!" The shouting continued for thirteen seconds and then subsided.

Kubeck said, "To Miami," and Hazen put in the call to Jesse Fisher, the air-traffic controller. When Fisher asked, "What kind of problem are you having?" Kubeck answered, off-radio, "Fire," and Hazen transmitted his urgent "Smoke in the cockpit. Smoke in the cabin."

Investigators now presume that the smoke was black and thick, and perhaps poisonous. The recorder picked up the sound of the cockpit door opening, and the voice of the chief flight attendant, who said, "Okay, we need oxygen. We can't get oxygen back there." Did she mean that the airplane's cabin masks had not dropped, or that they had dropped but were not working? If the smoke was poisonous, the masks might not have helped much, since by design they mix cabin air into the oxygen flow. The pilots were equipped with better, isolating-type masks and with goggles, but may not have had time to put them on. Only a minute had passed since the first strange chirp. Now the voice recorder captured the sound of renewed shouting from the cabin. In the cockpit the flight attendant said, "Completely on fire."

The recording was of little use to the NTSB's technical investigation, but because it showed that the passengers had died in agony, it added emotional weight to a political reaction that was already spreading beyond the details of the accident and that had begun to call the entire airline industry into question. The public, it seemed, would not be placated this time by standard reassurances and the discovery of a culprit or two. The press and the NTSB had put aside their on-site antagonism and had joined forces in a natural coalition with Congress. The questioning was motivated not by an immediate fear of unsafe skies (despite the warnings of Mary Schiavo, a federal whistle-blower who claimed special insight) but rather by a more nuanced suspicion that competition in the open sky had gone too far, and that the FAA, the agency charged with protecting the flying public, had fallen into the hands of industry insiders.

After a good night's sleep Hinson might have tried to repair the damage. Instead he appeared two days later at a Senate hearing in Washington sounding like an unrepentant Prussian: "We have a very professional, highly dedicated, organized, and efficient inspector work force that do their job day in and day out. And when we say an airline is safe to fly, it is safe to fly. There is no gray area."

His colleagues must have winced. Aviation safety is nothing but a gray area, and the regulation of it is an indirect process of negotiation and maneuver. Consider the size of the airline business, the scale of the sky, and the loneliness of an airplane in flight. The FAA can affect safety by establishing standards and enforcing them through inspections and paperwork, but it cannot throw the switches or turn the wrenches, or in this case supervise the disposal of old oxygen generators. Safety is ultimately in the hands of the operators, the mechanics and pilots and their managers, because it involves a blizzard of small judgments. Hinson might have admitted this reality to the American public, which is certainly capable of understanding such subtleties, but instead, inexplicably, he chose to link the FAA's reputation to that of ValuJet. This placed the agency in an impossible position. Whether for incompetence or for cronyism, the FAA would now inevitably be blamed.

Within days it came out that certain inspectors at the FAA had been worried about ValuJet for some time and had described their concerns in their reports. Their consensus was that the airline was expanding too fast (from two to fifty-two airplanes over its two-and-a-half-year life) and that it had neither the procedures nor the people in place to maintain standards of safety. The FAA tried to keep pace, but because of its other commitments—including countering the threat of terrorism—it could assign only three inspectors to the airline. At the time of the accident they had run 1,471 routine checks on the operation and made two additional eleven-day inspections, in 1994 and 1995. This level of scrutiny was about normal. But by early 1996 concern had grown within the FAA about the disproportionate number of infractions committed by ValuJet and the string of small bang-ups it had had. The agency began to move more aggressively. An aircraft-maintenance group found such serious problems in both the FAA's surveillance and the airline's operations that it wrote an internal report recommending that ValuJet be "recertified" immediately—meaning that it be grounded and started all over again. The report was apparently sent to Washington, where for reasons that remain unexplained it lay buried until after the accident. Meanwhile, on February 22, 1996, headquarters launched a 120-day "special emphasis" inspection, a preliminary report on which was issued after the first week. This suggested a wide range of problems. The special-emphasis inspection was ongoing when, on May 11, Flight 592 went down.

As this record of official concern emerged, the question changed from why Hinson had insisted on calling ValuJet "safe" after the accident to why he had not shut down the airline before the accident. Trapped by his own simplistic formulations, he could provide no convincing answer. The press and Congress were sharply critical. The FAA launched an exhaustive thirty-day review of ValuJet, perhaps the most concentrated airline inspection in history, assigning sixty inspectors to perform in one month the equivalent of four years' work. Lewis Jordan, a founder and the president of ValuJet, complained that Hinson was, in effect, conducting a witch hunt that no airline could withstand. Jordan had been trying shamelessly to shift the blame for the deaths onto his own contractor, SabreTech, and he received little sympathy now. No one was surprised when ValuJet was grounded indefinitely five weeks after the accident.

Here now was proof that the FAA had earlier neglected its duties. The agency's chief regulator, Anthony Broderick, was the first to lose his job. Broderick was an expert technocrat, disliked by safety crusaders because of his conservative approach to instituting and applying regulations, and respected by aviation insiders for the same reason. Hinson let him take the fall: Broderick was a man of integrity and would accept responsibility for the FAA's poor performance. But if Hinson thought that he himself could escape with this sacrifice, he was wrong. Broderick's airline friends now joined the critics in disgust. Hinson announced his upcoming resignation.

In a sense, the system worked. The tragedy did have some positive consequences—primarily because the NTSB did an even better job than usual, not only pinpointing the source and history of the fire but also recognizing some of its larger implications. With a well-timed series of press feedings and public hearings the accident team kept the difficult organizational issues alive and managed to stretch the soul-searching through the end of the year and beyond. By shaking up the FAA, the team reminded the agency of its mandate to oversee the safety of the airlines—perhaps prodding the FAA into a renewed commitment to inspections and a resolution to hold airlines responsible for their actions and for the performance of outside shops.

For the airlines, the investigation served as a necessary reminder of the possible consequences of cost-cutting and complacency. Among airline executives smart enough to notice, it may also have served as a warning about the public's growing distrust of their motives and about widespread anger with the whole industry—anger that may have as much to do with the way passengers are handled as with their fears of dying. However one wants to read it, the ValuJet turmoil marked the limits of the public's tolerance. The airlines were cowed, and they submitted eagerly to the banning of oxygen generators as cargo on passenger flights. They then rushed ahead of the FAA with a $400 million promise (not yet fulfilled) to install fire detectors and extinguishers in all cargo holds. The desire to find hidden hazards runs up against the practical difficulties of inspecting cargo. Nonetheless, ground crews can be counted on for a while to watch what they load into airplanes and what they take out and throw away.

And the guilty companies? They lost money and were sued, of course. After firing the two mechanics who had falsely signed the work orders, SabreTech tried to put its house in order. Nonetheless, its customers fled and did not return. The Miami operation shrank from 650 to 135 employees, and in January of last year was forced to close its doors. Soon afterward, as the result of a two-month FAA investigation, SabreTech's new Orlando facility was forced to close as well. ValuJet survived its grounding, and under intense FAA scrutiny returned to the sky later in 1996, with a reduced and standardized fleet of DC-9s; it ultimately changed its name to AirTran. For a while it was probably the safest airline in the country. What, then, explains the feeling, particular to this case, that so little has in reality been achieved?

Charles Perrow's thinking is more difficult for pilots like me to accept. Perrow came unintentionally to his theory about normal accidents after studying the failings of large organizations. His point is not that some technologies are riskier than others, which is obvious, but that the control and operation of some of the riskiest technologies require organizations so complex that serious failures are virtually guaranteed to occur. Those failures will occasionally combine in unforeseeable ways, and if they induce further failures in an operating environment of tightly interrelated processes, the failures will spin out of control, defeating all interventions. The resulting accidents are inevitable, Perrow asserts, because they emerge from the venture itself. You cannot eliminate one without killing the other.

Perrow's seminal book Normal Accidents: Living With High-Risk Technologies (1984) is an unusual work—a hodgepodge of storytelling and exhortation, out of which this new way of thinking has risen. His central device is an organizational chart on which to plot the likelihood of serious system accidents. He does not append numerical values to the chart but uses a set of general risk indicators. In one quadrant stand the processes—like those of most manufacturing—that are simple, slow, linear, and visible, and in which the operators experience failures as isolated and containable events. In the opposite one stand the opaque and tangled processes characterized by a combination of what Perrow calls "interactive complexity" and "tight coupling." By "interactive complexity" he means not simply that there are many elements involved but that those elements are linked in multiple and often unpredictable ways. The failure of one part—whether material, psychological, or organizational—may coincide with the failure of an entirely different part, and this unforeseeable combination will cause the failure of other parts, and so on. If the system is large, the possible combinations of failures are practically infinite. Such unravelings seem to have an intelligence of their own: they expose hidden connections, neutralize redundancies, bypass "firewalls," and exploit chance circumstances that no engineer could have planned for. When the operating system is inherently quick and inflexible (like a chemical process, an automated response to missile attack, or a jet airliner in flight), the cascading failures can accelerate out of control, confounding the human operators and denying them a chance to jury-rig a recovery. That lack of slack is Perrow's tight coupling. Then the only difference between a harmless accident and a human tragedy may be a question, as in chemical plants, of which way the wind blows.

I ran across this thinking by chance, a year before the ValuJet crash, when I picked up a copy of Scott D. Sagan's book The Limits of Safety: Organizations, Accidents, and Nuclear Weapons (1993). Sagan, a Stanford political scientist who is a generation younger than Perrow, is the most persuasive of Perrow's interpreters, and with The Limits of Safety he has solidified system-accident thinking, focusing it more clearly than Perrow was able to. The Limits of Safety starts by placing high-reliability and normal-accident theories in opposition and then tests them against a laboriously researched and previously secret history of failures within U.S. nuclear-weapons programs. The test is a transparent artifice, but it serves to define the two theories. Sagan's obvious bias does not diminish his work.

Strategic nuclear weapons pose an especially difficult problem for system-accident thinking, for two reasons: first, there has never been an accidental nuclear detonation, let alone an accidental nuclear war; and second, if a real possibility of such an apocalyptic failure exists, it threatens the very logic of nuclear deterrence—the expectation of rational behavior on which we continue to base our arsenals. Once again the pursuit of system accidents leads to uncomfortable ends. Sagan is not a man to advocate disarmament, and he shies away from doing so in his book, observing realistically that nuclear weapons are here to stay. Nonetheless, once he has defined "accidents" as less than nuclear explosions (as false warnings, near launches, and other unanticipated breakdowns in this ultimate "high-reliability" system), Sagan discovers a pattern of accidents, some of which were contained only by chance. The reader is hardly surprised when Sagan concludes that such accidents are inevitable.

The book interested me not because of the accidents themselves but because of their pattern, which seemed strangely familiar. Though the pattern represented possibilities that I as a pilot had categorically rejected, this new perspective required me to face the unpredictable side of my own experience with the sky. I had to admit that some of my friends had died in crazy and unlucky ways, that some flights had gone uncontrollably wrong, and that perhaps not even the pilots were to blame. What is more, I had to admit that no matter how carefully I checked my own airplanes, and how cautiously I flew them, the same could happen to me.

That is where we stand now as a society with ValuJet Flight 592, and it may explain our continuing discomfort with the accident. The ValuJet case represents a nearly perfect system accident. It arose from a process that fits most of Perrow's technical requirements of unpredictability and interactive complexity and some of those of tight coupling. More important, it fits the most basic definitions of an accident caused by the very functioning of the system or industry within which it occurred. Flight 592 burned because of its cargo of oxygen generators, yes, but more fundamentally because of a tangle of confusions that will take some entirely different form next time. It is frustrating to fight such a thing, and wrongdoing is difficult to assign.

This required a gang of hard-pressed mechanics to draw a verbal distinction between canisters that were "expired," meaning most of the ones they were removing, and canisters that were not "expended," meaning many of the same ones, loaded and ready to fire, on which they were expected to put nonexistent caps. Also involved were canisters that were expired and expended, and others that were not expired but were expended. And then, of course, there was the set of new replacement canisters, which were both unexpended and unexpired. If this seems confusing, do not waste your time trying to figure it out—the SabreTech mechanics did not, nor should they have been expected to. The NTSB suggested that one problem at SabreTech's Miami facility may have been the presence of Spanish-speaking immigrants on the work force, but quite obviously the language problem lay on the other side—with ValuJet and the English-speaking engineers, literalists, who wrote the orders and technical manuals as if they were writing to themselves. The real problem, in other words, was engineerspeak.

Before the accident the worry was not about old parts but about new ones—the safe refurbishing of the MD-80s in time to meet the ValuJet deadline. The mechanics quickly removed the oxygen canisters from their brackets and wired green tags to most of them. The green tags meant "repairable," which these canisters were not. It is not clear how many of the seventy-two workers were aware that these canisters couldn't be used again, since the replacement of oxygen generators is a rare operation, though of the people questioned after the accident most claimed to have known at least why the canisters had to be removed. But here, too, there is evidence of confusion. After the accident two tagged canisters were found still lying in the SabreTech hangar. On one of the tags, under "Reason for Removal," someone had written, "out of date." On the other tag someone had written, "generators have been expired fired."

Yes, a mechanic might have found his way past the ValuJet work card and into the huge MD-80 maintenance manual, to chapter 35-22-01, within which line "h" would have instructed him to "store or dispose of oxygen generator." By diligently pursuing his options, the mechanic could have found his way to a different part of the manual and learned that "all serviceable and unserviceable (unexpended) oxygen generators (canisters) are to be stored in an area that ensures that each unit is not exposed to high temperatures or possible damage." By pondering the implications of the parentheses he might have deduced that the "unexpended" canisters were also "unserviceable"canisters and that because he had no shipping cap, he should perhaps take such canisters to a safe area and "initiate" them, according to the procedures described in section 2.D. To initiate an oxygen generator is of course to fire it off, triggering the chemical reaction that produces oxygen and leaves a mildly toxic residue within the canister, which is then classified as hazardous waste. Section 2.D contains the admonition "An expended oxygen generator (canister) contains both barium oxide and asbestos fibers and must be disposed of in accordance with local regulatory compliances and using authorized procedures." No wonder the mechanics stuck the old generators in boxes.

The supervisors and inspectors failed miserably here, though after the accident they proved clever at ducking responsibility. At the least they should have supplied the required safety caps and verified that those caps were being used. If they had—despite all the other errors that were made—Flight 592 would not have burned. For larger reasons, too, their failure is an essential part of this story. It represents not the avarice of profit takers but rather something more insidious—the sort of collective relaxation of technical standards that the Boston College sociologist Diane Vaughan has called "the normalization of deviance," and that she believes existed at NASA in the years leading up to the 1986 explosion of the space shuttle Challenger. The leaking O-rings that caused the catastrophic blow-by of rocket fuel were a well-known design weakness, and had been the subject of worried memos and conferences up to the eve of the launch. Vaughan's book The Challenger Launch Decision (1996) is a 575-page exercise in system-accident thinking. After a long immersion in NASA's technical culture, Vaughan concludes that the O-ring worries were put aside in part because the agency had gotten away with launching the O-rings before. As Perrow has argued, what can go wrong usually goes right—and then people draw the wrong conclusions. In a general way this is what happened at SabreTech. Some mechanics now claim to have expressed their concerns about the safety caps, but if they did, they were not heard. The operation had grown used to taking shortcuts.

But let us be honest—mechanics who are too careful will never get the job done. The airline system as it stands today requires people, in flight or on the ground, to compromise, to make choices, and sometimes even to gamble. The SabreTech crews went astray—but not far astray—by allowing themselves quite naturally not to worry about discarded parts. A fire hazard? Sure. The mechanics taped off the lanyards and may have shoved the canisters a little farther away from the airplanes they were working on. The canisters had no warnings about heat on them and none of the standard hazardous-materials placards. It probably would not have mattered anyway, because the work area was crowded with placards and officially designated hazardous materials, and people had learned not to take them too seriously. Out of curiosity a few of the mechanics fired off some canisters and listened to the oxygen come out—it went pssst. No one seems to have considered the possibility that the canisters might accidentally be shipped. The mechanics did finally carry the five cardboard boxes over to the shipping department, but only because that was where ValuJet property was stored—an arrangement that itself made sense.

When the shipping clerk got to work the next morning, he found the boxes without explanation on the floor of the ValuJet area. The boxes were innocent-looking, and he left them alone until he was told to tidy up. Sending them to Atlanta seemed like the best way to do that. He had shipped off "company material" before without ValuJet's specific approval, and he had heard no complaints. He knew he was dealing with oxygen canisters, but apparently did not understand the difference between oxygen storage tanks and generators designed to fire off. When he prepared the boxes for shipping, he noticed the green "repairable" tags mistakenly placed on the canisters by the mechanics, and misunderstood them to signify "unserviceable" or "out of service," as he variably said after the accident. He also drew the unpredictable conclusion that the canisters were therefore empty. He asked the receiving clerk to fill out a shipping ticket. The receiving clerk did as he was asked, listing the tires and canisters, and put quotation marks around the word "Empty." Later, when asked why, he replied, "No reason. I always put like, when I put my check, I put 'Carlos' in quotations. No reason I put that." The reason was that it was his habit. On the shipping ticket he also put "5 boxes" between quotation marks.

But a day or so later, over by Flight 592, the ValuJet ramp agent who signed for the cargo didn't care about such subtleties. ValuJet was not authorized to carry hazardous cargoes of any sort, and it seems obvious now that a shipping ticket listing tires on wheel assemblies and oxygen canisters (whether or not they were empty) should have aroused the ramp agent's suspicions. No one would have complained had he opened the boxes, or summarily rejected the load. There was no hazardous-materials paperwork associated with it, but he had been formally trained in the recognition of unmarked hazards. His ValuJet station-operations manual specifically warned, "Cargo may be declared under a general description that may have hazards which are not apparent, that the shipper may not be aware of this. You must be conscious of the fact that these items have caused serious incidents, and in fact, endangered the safety of the aircraft and personnel involved." It also said,

It is possible that the ramp agent was lulled by the company-material labels. Would the SabreTech workers ship hazardous cargo without letting him know? His conversation with the copilot, Richard Hazen, about the weight of the load may have lulled him as well. Hazen, too, had been formally trained to spot hazardous materials, and he would have understood better than the ramp agent the dangerous nature of oxygen canisters, but he said nothing. It was a routine moment in a routine day. The morning's pesky electrical problems had perhaps been resolved. The crew was calmly and rationally preparing the airplane for the next flight, a procedure that had always worked for them before. As a result the passengers' last line of defense folded. They were unlucky, and the system killed them.

The other way to regulate the airline industry is not economic but operational—detailed governmental oversight of all the technical aspects of flight. This is an approach we have taken since the birth of the airlines, in the 1920s, and it is what we expect of the FAA today. Strictly applied standards are all the more important in a free market, in which unchecked competition would eventually require airlines to cut costs to the point of operating unsafely, until accidents forced them out of business one by one. A company should not overload its airplanes or fly them with worn-out parts, but it also cannot compete effectively against other companies that do. Day to day, airline executives may resent the intrusion of government, but in their more reflective moments they must also realize that they need this regulation in order to survive. The friendship that has grown up between the two sides—between the regulators and the regulated—is an expression of this fact, which no amount of self-reform at the FAA can change. When after the ValuJet crash David Hinson, of the FAA, reacted to accusations of cronyism by going to Congress and humbly requesting that his agency's "dual mandate" be eliminated, so that it would no longer be required by law to promote the airlines, he and Congress (which did as he requested) were engaged in a particularly hollow form of political theater.

The FAA's critics had real points to make. The agency had become too worried about the reactions of its allies in the airline industry, and it needed to try harder to enforce existing regulations. Perhaps it needed even to write some new regulations. Like NASA before the Challenger accident, the FAAneeded to listen to the opinions and worries of its own lower-level employees. But there are limits to all this, too. When, at a post-crash press conference in Miami, a reporter asked Robert Francis, of the NTSB, "Shouldn't the government protect us against this kind of thing?" the best answer would have been "It cannot, and never will."

The truth helps, because in our frustration with such system accidents we may be tempted to invent solutions that, by adding to the obscurity and complexity of the system, may aggravate just those characteristics that led to the accidents in the first place. This argument for a theoretical point of diminishing safety is a central part of Perrow's thinking, and it seems to be borne out in practice. In his exploration of the North American early-warning system Sagan found that the failures of safety devices and backup systems gave the most dangerous false indications of missile attack—the kind that could have triggered a response. The radiation accidents at Chernobyl and Three Mile Island were both induced by failures in the safety systems. Remember also that the ValuJet oxygen generators were safety devices, that they were backup systems, and that they were removed from the MD-80s because of regulations limiting their useful lives. This is not an argument against such devices but a reminder that elaboration comes at a price.

Human reactions add to the problem. Administrators can think up impressive chains of command and control, and impose complex double checks and procedures on an operating system, and they can load the structure with redundancies, but on the receiving end there comes a point—in the privacy of a hangar or a cockpit—beyond which people rebel. These rebellions are now common throughout the airline business—and, indeed, throughout society. They result in unpredictable and arbitrary actions, all the more so because in the modern, insecure workplace they remain undeclared. The one thing that always gets done is the required paperwork.

Paperwork is a necessary and inevitable part of the system, but it, too, introduces dangers. The problem is not just the burden that it places on practical operations but also the deception that it breeds. The two unfortunate mechanics who signed off on the nonexistent safety caps just happened to be the slowest to slip away when the supervisors needed signatures. The other mechanics almost certainly would have signed too, as did the inspectors. Their good old-fashioned pencil-whipping is perhaps the most widespread form of Vaughan's "normalization of deviance." The falsification they committed was part of a larger deception—the creation of an entire pretend reality that includes unworkable chains of command, unlearnable training programs, unreadable manuals, and the fiction of regulations, checks, and controls. Such pretend realities extend even into the most self-consciously progressive large organizations, with their attempts to formalize informality, to deregulate the workplace, to share profits and responsibilities, to respect the integrity and initiative of the individual. The systems work in principle, and usually in practice as well, but the two may have little to do with each other. Paperwork floats free of the ground and obscures the murky workplaces where, in the confusion of real life, system accidents are born.

It would be wrong to conclude that we should join the alarmists in their prophesies of doom. Flying will remain safe, and for conventional reasons, including the admirable reaction we have seen to the ValuJet crash. But it should also be clear that there are structural limits to flight safety, and that any dream of a zero-accident future is probably about as realistic as the old ValuJet promise to put safety first. If that is true, we had better get used to it. Conventional accidents—those I call procedural or engineered—will submit to our solutions, but as air travel continues to expand, we can expect capricious system accidents to blossom. Understanding why might keep us from making the system even more complex, and therefore perhaps more dangerous, too.