Publication Bias

What it is

Nobody has to behave badly for this to happen. A researcher who runs a study that finds nothing has a result that is harder to write up, harder to place, and less rewarded if placed. A journal editor with limited pages prefers the paper that reports something over the paper that reports the absence of something. A field that runs many small studies on the same question will, by chance alone, produce some that clear significance, and those are the ones that travel. Every actor makes a locally reasonable decision and the aggregate is a literature that overstates whatever it is about.

The effect compounds in a way that is easy to miss. Small studies are noisy, which means their results scatter widely, which means the ones that survive the filter are systematically the ones where the noise happened to run in the flattering direction. A published literature built from small studies does not just contain too few nulls. It contains positives that are too large, because only the large ones got through. That is why the corrective statistics look for a relationship between a study's size and its reported effect: if the small studies in a literature report bigger effects than the large ones, something other than the phenomenon is deciding what gets printed.

Two of those techniques appear repeatedly on this site. Small-study correction methods estimate what a literature would look like if the missing nulls were restored, and produce an adjusted effect size to compare against the published one. P-curve asks a different question, whether the distribution of significant p-values in a literature has the shape a real effect would produce, or the flatter shape produced by analytic flexibility. Neither method opens a drawer. Both work by asking whether the published numbers are internally consistent with the claim that only the phenomenon was doing the selecting.

Fraud is rare, and treating publication bias as a milder synonym for misconduct gets the problem backwards. Misconduct is an individual failure with an individual remedy. This is structural, it operates on whole literatures, and the people whose work is affected by it can be scrupulous throughout. The single most useful thing a non-researcher can take from the concept is a change in what counts as reassurance. "There are hundreds of studies" describes the output of a filter. It says nothing about what went in.

In effect

The reanalysis that needed no new data

Ego depletion is the cleanest worked case available. A 2010 meta-analysis by Martin Hagger and colleagues covering close to 200 experiments put the depletion effect at d = 0.62, medium to large, and large enough to plan a working day around. In 2014 Evan Carter and Michael McCullough took the same dataset, not a new one, and ran it through methods designed to detect and correct for small-study effects. They found very strong signals of bias and concluded that the corrected effect could be zero.

No new data. Better methods over the same numbers, and a different answer.

The later preregistered work is what makes the reanalysis worth citing rather than merely arguing about. A 23-laboratory preregistered replication in 2016 with 2,141 participants returned d = 0.04. A 36-laboratory test in 2021 with 3,531 participants returned d = 0.06. A 12-laboratory replication the same year with 1,775 participants returned d = 0.10, rising to 0.16 once participants who appeared to be answering at random were excluded. The honest summary is that the true effect sits somewhere between zero and about a tenth of a standard deviation, which is not zero and is nowhere near 0.62. The correction was pointing in the right direction four years before the large replications arrived.

The glucose half, checked with a different instrument

The claim that self-control runs on glucose and can be restored with a sugary drink was the most commercially useful part of the strength model, and it was tested with p-curve rather than with correction methods. Miguel Vadillo, Natalie Gold and Magda Osman published that analysis in Psychological Science in 2016 and found the literature's evidential value weak. Separate experiments testing glucose drinks and mouth rinses against depletion found nothing. The physiological objection had always been available too, since the quantity of sugar involved is far too small to change how a brain works, but the p-curve is the part that speaks to publication bias specifically: the significant results in that literature did not have the shape a real effect produces.

What it costs the person doing the citing

The commercial version of this problem is not abstract. Advice, training programmes and product decisions were built on d = 0.62, and the organisations that built them were reading the literature correctly as it stood. That is the uncomfortable part. Diligence at the level of "check that the claim is supported by published research" was not enough, because the filter operates above that level. For a publication whose authority rests on being right, the exposure is direct: citing a finding that exists mainly because the nulls were never printed is the fastest available way to lose the trust that makes the citation worth anything. The practical response is not to stop citing psychology. It is to prefer literatures where the nulls are visible, which is what preregistration makes possible.

What it does not say

It does not say that published research is worthless, or that a large literature is evidence of a problem. Publication bias is a reason to check, not a verdict. Plenty of well-supported findings sit in literatures with hundreds of papers, and the way to tell them apart is to look at whether the large, preregistered tests agree with the small ones, not to count papers in either direction.

It does not say that a bias-corrected estimate is the truth. Correction methods produce an inference about a literature under assumptions, not a measurement of the world. Carter and McCullough's conclusion was that the effect could be zero, which is a weaker and more defensible claim than "the effect is zero," and the multi-laboratory work that followed suggests something small and non-zero rather than nothing at all. Reporting the correction as a debunking overstates it in the same direction the original literature was overstated.

It does not diagnose individual studies. This is a property of literatures. The 1998 cookies-and-radishes experiment is not made invalid by the state of the field it launched, and a single well-run study with a null result is not evidence that its field is biased. Using the concept to dismiss a particular paper you dislike is a misuse of it, and a common one.

It also does not apply only to conclusions you find inconvenient. The reasoning is symmetric, and the same filter shapes literatures on the other side of any argument, including the ones that report failures to replicate. The only real defence is the boring one: prefer evidence produced under conditions where a null result would have been published anyway, and be suspicious of your own relief when a correction lands where you wanted it.

---