The Law of Small Numbers

What it is

The damage is done in two steps. First, small samples produce extreme results far more often than large ones, simply because there is less to average out. Second, those extreme results are handed a causal explanation they do not deserve. The pattern is real, the arithmetic that produced it is invisible, and the story arrives to fill the gap.

It is a fact about sampling variance rather than an empirical effect that might fail to replicate, and the psychological finding built on it, that people misread small-sample extremes as signal, is among the durable results in this area.

In effect

The clearest illustration is kidney cancer across American counties. Of the more than three thousand counties in the United States, the ones with the lowest incidence turn out to be mostly rural, sparsely populated and traditionally Republican, which invites an immediate explanation about clean living and fresh air. Then the same survey shows the counties with the highest incidence are also mostly rural, sparsely populated and traditionally Republican. The two explanations cannot both hold and neither is needed. The only factor doing any work is small population, which makes extreme rates in both directions more likely. The example is credited by Kahneman to Howard Wainer and Harris Zwerling.

The same structure cost real money. A survey of Pennsylvania schools found small schools substantially overrepresented among the highest performers, the conclusion drawn was that small schools produce better outcomes, and large sums followed, including an investment by the Gates Foundation in small schools that the book puts in the billions. But small schools also dominate the list of the worst performers, for the same reason the low-incidence and high-incidence counties look alike. Small units are more variable, so they crowd both tails.

The researchers were running the bias themselves. Kahneman records reading an article reporting that psychologists commonly chose sample sizes carrying roughly a one-in-two risk of missing a true effect, and concluding that some of his own odd findings were artefacts of his research method. The direction of that admission is what makes it worth keeping.

There is also a case where the researcher got there first. Margaret Mead, in an appendix to Coming of Age in Samoa in 1928, forty-three years before the bias was named, wrote that with only sixty-eight girls in her age band quantitative statements were practically valueless, because the probable error of the group was too large and the age classes too small, and said the same of her intelligence test results. Her argument for generalising anyway is not the ordinary error. She does not claim the small sample resembles the population; she claims the sample is internally uniform, and that low variability within a group rather than a large number in it is what supports generality. That is a real statistical intuition, and it is unverified, because the uniformity licensing the inference is asserted from the same six months of fieldwork that produced the inference. One of the three supporting arguments turns back on itself, holding that the drastic character of the conclusions and the few exceptions needed further validate the size of the sample. A small sample producing a striking result is the ordinary behaviour of small samples, which is the whole content of this concept.

What it does not say

It does not say small studies are worthless. It says an extreme result from one carries far less information than its size suggests, and that the remedy is to ask what the sample size was before asking what the finding means.

It does not say the illustrations here are all sourced. The kidney-cancer example is attributed; the Pennsylvania school survey and the article on statistical power are not attributed in the popular source, and the figures in both are reported as the book gives them and should be traced before use.

It does not mean stating the problem fixes it. Mead's warning was printed, by the author, in the book, in plain English. The conclusion travelled for a century and the warning did not, because the warning was in an appendix and the conclusion was in the chapters people read.

---