Sometimes the best hook is your own slip. In a recent breakdown of autonomous systems regulation, I casually placed two numbers side by side like playing a trump card: ASIL D is 10⁻⁸ failures per hour, while in aviation catastrophic failures are regulated below 10⁻⁹ per flight hour per ARP4761. The second number sounds so neat it doesn't invite scrutiny: one chance in a billion, sounds like the result of the most delicate measurements.
I went to check—and got stuck. First, there's no 10⁻⁹ anywhere in ARP4761: it's a document about safety assessment methodology, and the numerical budgets live somewhere else. Second, and far more interesting, 10⁻⁹ isn't a measurement at all. It's a budget derived through reverse arithmetic: first they decided what an acceptable picture of the world should look like, then divided it by the number of aircraft, systems, and years.
And the whole construction rests on a number written down in 1982 in a book that's long out of print.
The most honest primary source here lies off to the side of textbooks and current standards: a letter in a closed professional mailing list. In August 2024, engineer Phil Koopman from Carnegie Mellon University was clarifying in a system safety mailing list what exactly "one in a billion" refers to, and Peter Ladkin from Bielefeld answered him with a direct quote from Lloyd & Tye's book "Systematic Safety," published by the UK Civil Aviation Authority in 1982. Page 37, verbatim:
If it is assumed that from the safety assessment of all the systems of an aircraft type as many as 100 separate catastrophic effects might emerge, the total maximum risk by the requirements would be 100 × 10⁻⁹ = 10⁻⁷. This would mean that with a fleet of 100 aircraft of one type, each flying 3000 hours a year, one or other catastrophic effect might manifest itself once in just over 30 years, which is close to the concept of "practically never."
Read this again, because the paragraph inverts the usual logic of proof. Not "we measured the frequency and it turned out to be this." But "we want this to happen practically never; therefore, the frequency should be this." Then the number just gets projected onto systems: 100 catastrophic effects per type, 100 aircraft, 3000 hours per year, airframe life around 33 years. Total exposure roughly 10⁷ flight hours and a budget of about 10⁻⁹ per hour per effect.
The most curious thing in this quote is the word "assumed." The authors honestly write that the figure 100 is an assumption, not a calculation.
The second surprise awaits in the regulatory document text. Open 14 CFR 25.1309—that very paragraph everyone cites: "systems and associated components must be designed and installed so that each catastrophic failure must be extremely improbable." That's it. There's no nine anywhere, because the regulator writes the requirement by quality, not by number. Same with "extremely remote" and "remote."
The numbers live in a different layer—in the Advisory Circular, a document with an explicit disclaimer: "acceptable means, but not the only means." And here begins the story of forty years of uncertainty. As noted in the MathPages breakdown, AC 25.1309-1 (1982) and its version -1A (1988) consisted literally of two ambiguous sentences without any justification—which spawned divergences in interpretation both in industry and among the regulators themselves. Only in the 2000s did the ARAC committee convene and develop a detailed appendix with quantitative risk apparatus, named Arsenal—after the football club and for no substantive reason.
From this compendium flows the number: the original design of the criteria was formulated as a requirement that the probability of a single system failure leading to catastrophe over 10 million flight hours be less than 1/100. Hence the normalized average probability per hour—10⁻⁹. And this number has the same special status as "100 catastrophic effects": it's not a threshold found in nature, but an agreed-upon scale for engineering judgment.
Arsenal was adopted by the European side (now EASA). FAA—did not adopt it. So today transatlantic certifications rest on different interpretations of the same number.
And another detail that breaks the entire illusion of precision. The current revision AC 25.1309-1B from August 2024 explicitly allows considering a factor of three to be "of an order of magnitude": for extremely remote and extremely improbable failure conditions "a factor of three may be considered 'on the order' for demonstrating compliance." 10⁻⁹ and 3×10⁻⁹ are officially considered the same number.
In the same document, in section 3.2.4.1.2, the regulator writes literally the following: "Strictly speaking, performance of quantitative analysis alone to show that a failure condition is extremely improbable is not sufficient, due to the variability and uncertainty in the analytical process itself." Forty years after the number appeared, its own regulator stated that the number can't be relied upon.
When by the 1990s it became clear that the figure would have to be somehow transferred to software, they got down to business in earnest—and immediately hit arithmetic. Ricky Butler and George Finelli from NASA's Langley Research Center calculated what simple statistical confirmation would cost. Their result in IEEE Transactions on Software Engineering (1993) deserves to be written out in a table, because no metaphor can convey it.
A failure probability of 10⁻⁹ per ten-hour mission means a mean time to failure on the order of 10¹⁰ hours. If you take one system under test and wait for the first failure, the expected test duration is about 1,141,550 years. Ten systems on the bench reduce this to 114,155 years. A hundred systems—11,415 years. Ten thousand systems give a modest 114 years, but that requires ten thousand copies of an expensive avionics complex.
A separate story—reliability growth models, which were then considered a progressive method. By the authors' calculations, removing the last defect and reaching a failure intensity on the order of 10⁻¹¹ per input required on average about 2.2 × 10¹⁰ test runs. Even with a thousand-fold acceleration relative to real time, that's 42 years of continuous testing. In the table for six real prototype programs, the spread came out from 19 to 66 years—per program, and that's just the last defect found.
A remarkable property of the number 10⁻⁹ is that you can't obtain it for a complex system at all without accepting one modest axiom: that failures of redundant components are independent. Only then do probabilities multiply, and only then does a number with nine zeros assemble from separately failing blocks. Butler and Finelli formulate this without circumlocution: "The key assumption that allows… to estimate failure probabilities on the order of 10⁻⁹ is that individual redundant components fail independently or nearly independently."
For hardware this is reasonable. Different enclosures, separate power supplies, electrical isolation, environmental shielding—physics really does allow us to count on independence of failures. For software the same move doesn't work for a reason the authors formulate poisonously and precisely: "software faults are products of human reasoning," that is, in the literal sense, incorrect reasoning. Different teams, reading one specification, make similar mistakes because identically constructed people err in the same way on the same problem.
This didn't remain at the theory level. In 1986, John Knight and Nancy Leveson published in IEEE Transactions on Software Engineering the result of an experiment that's still uncomfortable to recall: 27 versions of one program were independently written at two universities from a unified specification of an antimissile defense system, then run through a million tests against a reference oracle program.
Individual versions turned out good: six of 27 gave not a single failure, twenty-three passed more than 99.9% of tests. But tests in which more than one version failed turned out substantially more numerous than independence predicted. Five hundred fifty-one times exactly two versions failed on the same input. In the most extreme case, eight programs failed on the same test—and this happened twice.
The statistics came out humiliating for the axiom: significance at α = 0.01, z-score 100.55. The null hypothesis of failure independence rejected.
A year earlier, Dave Eckhardt and Larry Lee from NASA proposed a mathematical apparatus for this, introducing the concept of "coincident errors" and their intensity. They also posed a question industry still doesn't like: does reliability grow monotonically with the number of versions, or is there an optimum after which adding a fifth copy only increases cost without improving safety. In the limiting case, when all residual defects are common to all versions, the system doesn't differ from one randomly selected version.
Next begins the most elegant part of the story, and it relates more to behavior than to technique. Confronted with the fact that the number can neither be measured nor justified for software, civil aviation simply stopped assigning probabilities to software. Didn't abandon the requirement, but translated it into a different language.
Bev Littlewood and Lorenzo Strigini in their 1993 work in Communications of the ACM summarized the state of things then: "it appears that engineering practice must accept the fact that no solution exists—at least now—for validation of ultra-high dependability in systems relying on complex software." They enumerated why software falls out of probabilistic logic: design defects are hard both to avoid and to counter; software often implements radically new functions, so experience from past successful systems doesn't help; and most importantly—digital systems implement discontinuous input-output mappings, to which continuity assumptions that work for mechanics don't apply.
The consequence of this impasse we see in current practice every time an aircraft gets a firmware update. DO-178C doesn't assign software either failure risk or probability: it requires a Development Assurance Level (DAL), meaning confirmation of processes. Roughly speaking, industry said: since the number is unattainable for software, we'll measure the rigor of how we prepared for it.
But here's the conclusion that turns the whole picture upside down.
In the opinion of John Downer from the University of Bristol, probabilistic calculations in aviation don't work the way they describe themselves. In his article "The Aviation Paradox" and book Rational Accidents (MIT Press, 2024), it's shown that FAA investigates the service history of a system's predecessors, not the system itself: verifies that proposed changes won't spoil this history, then assumes the new system will be about as reliable. This process is "more actuarial than predictive." It works only thanks to two institutional norms: design stability (a new machine changes so incrementally it remains comparable to the past one—thus fly-by-wire and composites survived decades in military aviation before entering civil) and recursive practice (constant incremental refinement based on incident analysis results).
Three pillars, according to Downer, hold the whole system: service history, design stability, recursive practice.
The most vivid test of the probabilistic approach happened in 2019, and it failed not where expected. Two crashes of the Boeing 737 MAX took 346 lives, and NTSB's breakdown (report ASR-19-01, September 2019) showed that the key link in the safety system was an assumption not about hardware, but about humans.
Certification logic is arranged this way: a system is assessed as safe if it's assumed the pilot correctly recognizes a failure and executes the needed procedure. Boeing built this assumption into the hazard classification of MCAS activation, and FAA approved it. In reality, pilot behavior on all three examined flights differed from what was anticipated, and simultaneous issuance of multiple alerts prevented immediate selection of the needed procedure. NTSB formulates the conclusion cautiously but unambiguously: if the manufacturer assumes correct pilot response without analyzing all conceivable combinations of alerts, then the hazard classification and all resulting design, procedural, and training measures may not account for actual behavior.
This is more important than it looks. The number 10⁻⁹ was calculated by a formula where independent variables were components, and the assumption about the pilot stood alongside as a constant that nobody recalculated. The failure came precisely from the term that wasn't in the formula.
From aviation the number spread throughout all safety engineering. In automotive ISO 26262, the target value of the random hardware failure metric for the highest level ASIL D is less than 10⁻⁸ per hour, for ASIL C and B—less than 10⁻⁷ per hour. In industrial IEC 61508, the fourth safety integrity level SIL 4 in continuous operation mode is defined as a range from 10⁻⁹ to 10⁻⁸ dangerous failures per hour. Littlewood and Strigini mention another bar: a processor for urban trains must fail with probability less than 10⁻¹² per hour, three orders of magnitude stricter than an aircraft.
It's telling that all these numbers are copies of the same thought process, multiplied across industries. And just as in aviation, they relate to random hardware failures. Where software appears, probabilistic requirements either disappear or transform into process requirements.
Today the same choice confronts artificial intelligence developers. In November 2025, EASA issued NPA 2025-07—the first regulatory draft on AI reliability in aviation, implementing the European AI Act. And the document's structure reveals that the industry has already traveled this path: the quantitative paradigm doesn't transfer to learning models; instead, learning assurance is proposed—confirmation of how the model was trained, on what data, and how it was verified. Exactly the same pivot from "how many times will it fail" to "how carefully we prepared for this" that aviation made for software thirty years ago.
Separately—three places where the material resisted being laid into a neat story.
First. I myself almost made a false comparison.
According to IATA data for 2025—1.32 accidents per million flights and eight fatal accidents per 38.7 million flights. It was tempting to recalculate this "per hour" and compare with the benchmark from the Lloyd & Tye era—supposedly the promised "practically never" slipped by two orders of magnitude. But you can't compare this: 10⁻⁹ relates to one specific catastrophic failure of one system, not to the aircraft as a whole, and certainly not to all accidents including runway excursions. Here's exactly that confusion between failure intensity per hour and normalized probability per flight hour that the MathPages breakdown warns about: quantities seemingly in the same units, but meaning different things. I'm leaving this as is: more as a trap I almost fell into than as a conclusion.
Second, quite honestly. Koopman in that very letter writes about the assumption of ten catastrophic failure conditions: "I may be remembering the number ten incorrectly—I don't know if this is even written down anywhere." An expert who's been doing system safety for decades isn't sure whether the key assumption of his field is documented anywhere. Apparently, if it is documented, it's somewhere in a book from 1982 that's long out of print, not in a current standard.
And third, minor. The same set of rules exists in two revisions with different thresholds: Arsenal was adopted in Europe, FAA—didn't adopt it. An engineer certifying a machine for both jurisdictions formally proves compliance with two different numbers for the same design.