Survivorship Bias

What it is

There is no single originator. The reasoning failure has been described independently in statistics, in finance and in the history of science, and the version most people meet is the wartime aircraft armour story, which circulates in several incompatible forms.

The test is always the same and it is always the missing half. Who did this and failed? If that group cannot be found, the argument has no comparison and cannot say what the ingredient did. A book that profiles people who succeeded and also did X has established that X is compatible with success, which is a much weaker claim than the one usually being made.

In effect

The vault holds several book-length instances, and they differ usefully in purity.

Cal Newport's Deep Work draws its whole prescriptive apparatus from people who succeeded, and no failed practitioner appears anywhere in the book. The opening roll-call at pages 3 to 5 names seven famous people selected because they succeeded and also worked alone. The four depth philosophies are introduced through people and only through people, with the stated selection criterion being the ones the author has seen work exceptionally well in practice. The grand gesture chapter at pages 121 to 126 is seven named anecdotes and no research: a novelist finished a book in a hotel suite, so the suite gets the credit, and a physicist worked out a device locked in a hotel room after being scooped, which reads at least as well as a story about competitive rage. The conclusion rests on a single eight-week programming sprint in 1974, which Newport himself calls the strongest form of his argument. To his credit he raises one counterexample, a successful person who works in fragments, before quarantining him with a rule that is asserted rather than tested.

A book of fifty-six profiles of eminent creative figures is purer still, because the selection rule for entry is eminence and the question cannot be asked at all. Nobody appears in it who kept three notebooks, or went to the pub every day, or wrote through the night, and produced nothing. What that book adds is a presentational aggravation rather than a second bias: each page carries a running head in the imperative, an instruction rather than a name, so the inference from this person did X and was great to do X is never argued anywhere. It is typeset. An argument printed as prose can be disputed on the page it appears; an inference asserted as a running head has no proposition to disagree with, and it is read before the paragraph meant to support it. A format of extractable rules will also contain rules that cancel, and this one sets one designer's flat denial that creative block exists against another figure praised for rarely suffering it, with no argument anywhere that could adjudicate between them.

The honest version is worth recording too. The Elephant in the Brain names its own sampling problem at page 46 before using it: its account of ancestral life is drawn from modern forager societies, perhaps as few as twenty are known to anthropologists, they have been influenced by modernity and they have been relegated to marginal environments. The authors state the problem, the contamination and the inferential gap, and note at page 51 that theories about distant ancestors are necessarily somewhat speculative. That is the correct shape and it is still the missing-half problem exactly: the forager societies observable in the twentieth century are the ones that survived into it, and the ones that did not left no ethnography. Naming the limit does not remove it.

What it does not say

It does not say anecdotes are illegitimate. They are perfectly good for illustrating a mechanism, for showing that a practice is possible, and for making an argument readable. The failure is putting the general claim first and letting the case read as its confirmation.

It does not say the winners' habits are irrelevant. It says the sample cannot tell you which of them mattered, which is a claim about what the evidence can support rather than about what is true.

It does not apply only to people. Publication bias is the same structure happening to studies rather than to individuals, and a retrospective literature built from practitioners who are already experts carries the bias inside a research field rather than inside a book.

---