The Psychology You Learned Didn't Replicate
Psychology audited itself in 2015 and a third of the findings did not come back. The story stops there in most coverage. It is the next decade, and what survived, that tells you how to read a psychological claim now.
Key takeaways
- The Open Science Collaboration replicated 100 studies from three psychology journals. Ninety-seven per cent of the originals had reported significant results; thirty-six per cent of the replications did.
- Replication success was predicted by the strength of the original evidence, not by anything about the teams involved. Weak evidence stayed weak.
- The mechanism was structural, not personal: journals print surprising positives, nulls go in drawers, and the published literature stops being a sample of what was found.
- The Many Labs projects ran preregistered replications across 36 countries with over 15,000 participants and around half held up, which is a different diagnosis from the 2015 number.
- The findings that survived were the large, repeatedly demonstrated ones whose original evidence was strong rather than merely significant.
The story you have probably absorbed is that psychology got caught. That a field built on small studies and clever experiments turned out to be largely made up, and that the sensible response is to stop believing any of it.
That is not what happened, and the actual story is more useful. A field ran an audit on itself, published the result, and then spent a decade working out which of its findings were real. Some very famous ones were not. A surprising number were. The interesting question is not whether psychology survived. It is how you are supposed to read a psychological claim now that the audit exists.
What actually happened in 2015?
A group calling itself the Open Science Collaboration [replicated 100 experimental and correlational studies](https://thewhyweproject.com/articles/do-psychology-findings-hold-everywhere) from three psychology journals, using high-powered designs and, where they could get them, the original materials. Replication within the same kinds of sample is one test, and holding up in other populations is another. Ninety-seven per cent of the original studies had reported statistically significant results. Thirty-six per cent of the replications did. Replication effects came out at about half the magnitude of the originals.
One finding inside that finding matters more than the headline and gets quoted least. Replication success was better predicted by the strength of the original evidence than by anything about the original or the replication teams. Not that some labs were sloppy and others careful, which nobody serious had claimed. Weak evidence stayed weak when somebody looked again.
Why did so much of it fail?
Because of an arrangement nobody designed and everybody worked inside. A journal publishes surprising positive results. A career is built on publications. A study that finds nothing is harder to publish, so it goes in a drawer. Run that for forty years and the published literature is not a sample of what researchers found. It is a sample of what researchers found that was interesting enough to print.
Add small samples, which make an effect look larger than it is whenever it reaches significance at all, and a degree of freedom in how the analysis gets run that nobody thought of as cheating because nobody thought of it as anything.
You can watch the mechanism operating directly. Miguel Vadillo, Natalie Gold and Magda Osman took the experiments claiming that glucose is the resource behind willpower and ran a p-curve analysis over them, which examines the distribution of results across a literature instead of testing any single study. Their conclusion was that the effect sizes are possibly influenced by publication or reporting bias, and that even within the studies reporting significant results, the evidential value is weak. Their point was not that the individual papers were wrong. It was that the shape of the literature was the shape a literature takes when the disappointing runs are missing.
Weak evidence stayed weak when somebody looked again. That is the whole finding, and it is more useful than any list of casualties.
Which famous things did not survive?
A partial list, and each one is on this site with its sources. Take the proposed machinery of [ego depletion](https://thewhyweproject.com/articles/does-decision-fatigue-spoil-our-choices), the idea that self-control runs on a limited tank. The glucose mechanism does not hold up under the analysis above. The best-known rescue of the effect, that depletion happens only to people who believe willpower is limited, was tested again in a preregistered replication by Nicholas Carruth, Jairo Ramos and Akira Miyake with 187 participants, and neither the depletion effect nor its moderation by mindset appeared. That is two supports gone. Whether the underlying effect exists at all is argued elsewhere on this site and is not settled by either paper.
Red making women more attractive to men, from Andrew Elliot and Daniela Niesta's 2008 paper, which named marketers among its audience. Leonard Peperkoorn, Craig Roberts and Thomas Pollet then ran three experiments covering 830 men and found no red effect, having set out to refine the finding rather than to attack it. Gabrielle Lehmann and Robert Calin-Jageman obtained the original materials, preregistered the design and built in a positive control, and found a very weak effect for men rating women and a very weak one in the opposite direction for women rating men.
Nudging as a general proposition. When Maximilian Maier and colleagues reanalysed the choice architecture database with publication probability modelled directly, there was no evidence for an overall effect, though the category containing defaults came out undecided rather than negative. Painted eyes above an honesty box were sold as the cheapest nudge there is, and one article asks whether [watching eyes](https://thewhyweproject.com/articles/do-watching-eyes-make-people-behave) make people behave.
The bottomless soup bowl, and a good deal else from the same laboratory. We name researchers throughout this article whose findings did not replicate, because a non-replication is ordinary science and carries no imputation. We do not name this one, because the matter went beyond non-replication to a withdrawal of institutional assurance in 2018, and that is a different kind of claim about a living person. The fact a reader needs is that the figures are not evidence of anything now, whatever they showed.
Behavioural priming, in its strong form. The claim that a subtle cue could reach into behaviour broadly and automatically took the most damage, and the anchoring literature, once its best illustration, is now cited against it. Another popular claim that rests on thin evidence is taken apart in an article asking [whether brain training games work](https://thewhyweproject.com/articles/do-brain-training-games-work). The best-known claim that a role turns ordinary people cruel gets the same sorting in an article on what the [Stanford Prison Experiment](https://thewhyweproject.com/articles/the-stanford-prison-experiment-reread) showed.
What survived?
This is the part the coverage skipped, and it is larger than you would guess. Before the 2015 audit, Richard Klein and colleagues had already run a different design. Thirteen classic and contemporary effects, tested across 36 independent samples with 6,344 participants. Ten replicated consistently. One, imagined contact reducing prejudice, showed weak support. Two did not replicate at all, both of them priming effects, flag priming on conservatism and currency priming on system justification.
Then they did it properly. Twenty-eight published findings, protocols peer reviewed before anybody collected data, administered across 125 samples totalling 15,305 participants in 36 countries and territories. Fifteen of the replications, fifty-four per cent, produced a statistically significant effect in the same direction as the original. At a much stricter threshold, fourteen still did.
Read that next to the 2015 number, with one concession attached. The two projects did not sample the same way. The 2015 audit took a journal year largely as it came; the Many Labs projects chose findings that suited a multi-site design, which will flatter the rate. The comparison is therefore suggestive and not clean.
Even allowing for that, the picture changes shape. A third of a journal's back catalogue replicating is alarming. Half of a chosen set of classic findings replicating at high power across three dozen countries describes a field with real results in it and a serious contamination problem, a different diagnosis calling for a different response.
The effects that came through tend to share a feature. They are the large, obvious, repeatedly demonstrated influences and not the delicate ones, and their original evidence was strong and not merely significant. One famous colour result that never rested on what it was said to has an article of its own on [what the pink cell never proved](https://thewhyweproject.com/articles/does-pink-really-calm-people-down). Part of what looks like a finding may be arithmetic, as an article on whether the [Dunning-Kruger effect](https://thewhyweproject.com/articles/is-the-dunning-kruger-effect-real) is real explains.
So how should you read a psychological claim now?
Four questions, and they cost nothing. How many people were in it. A study of forty undergraduates that produced a startling result produced it partly by being small.
Has anybody done it again, with the original materials, having said in advance what they would find. That last clause is the one doing the work, and it is recent enough that most famous findings predate it.
Does the person citing it also tell you what the researchers said about their own limits. Almost every paper in this article contains its authors' own caution, and almost none of those cautions reach the audience.
And is the claim that something has a large general effect on behaviour, or that something has a specific effect in a specific setting. The second kind survived the audit at a much better rate.
The question nobody has answered
Here is what still bothers us, and we do not have a resolution for it. The audit tested findings that were old enough to be famous. Everything published since has been produced under better norms, with preregistration spreading, samples growing and journals more willing to print a null result. Nobody has audited that. Plenty of individual replications have run since, and there is no 2015-style audit of a journal year of post-2015 work, partly because the effort was enormous and partly because the incentive to run one is no stronger now than it was then.
So the honest position is that we know a great deal about the reliability of psychology's confident era and very little about its careful one. The reasonable expectation is that things have improved. Expectation is not evidence, which is the entire lesson of the decade this article describes.
Sources
Open Science Collaboration (2015). 'Estimating the reproducibility of psychological science.' Science, 349(6251), aac4716. One hundred replications; 97 per cent of originals statistically significant against 36 per cent of replications; replication effects half the magnitude; replication success better predicted by the strength of original evidence than by characteristics of original or replication teams. doi:10.1126/science.aac4716
Klein, R. A., Ratliff, K. A., Vianello, M., et al. (2014). 'Investigating variation in replicability: A many labs replication project.' Social Psychology, 45(3), 142-152. Thirteen effects across 36 samples and 6,344 participants; ten replicated consistently, one weakly, and two, flag priming and currency priming, did not. doi:10.1027/1864-9335/a000178
Klein, R. A., Vianello, M., Hasselman, F., et al. (2018). 'Many Labs 2: Investigating variation in replicability across samples and settings.' Advances in Methods and Practices in Psychological Science, 1(4), 443-490. Twenty-eight findings, protocols peer reviewed in advance, 125 samples, 15,305 participants, 36 countries; 54 per cent significant in the same direction, 50 per cent at a strict threshold. Findings were selected for suitability to the design, which this article states before comparing rates. doi:10.1177/2515245918810225
Vadillo, M. A., Gold, N., & Osman, M. (2016). 'The bitter truth about sugar and willpower.' Psychological Science, 27(9), 1207-1214. p-curve analysis of the glucose experiments; effect sizes possibly influenced by publication or reporting bias and weak evidential value even within significant studies. doi:10.1177/0956797616654911
Carruth, N. P., Ramos, J. A., & Miyake, A. (2023). 'Does willpower mindset really moderate the ego-depletion effect? A preregistered replication of Job, Dweck, and Walton (2010).' PLoS ONE, 18(6), e0287911. 187 participants; neither a significant main effect of depletion nor moderation by willpower mindset. doi:10.1371/journal.pone.0287911
Lehmann, G. K., & Calin-Jageman, R. J. (2017). 'Is red really romantic?' Social Psychology, 48(3), 174-183. doi:10.1027/1864-9335/a000296
Peperkoorn, L. S., Roberts, S. C., & Pollet, T. V. (2016). Evolutionary Psychology, 14(4). doi:10.1177/1474704916673841
Maier, M., Bartos, F., Stanley, T. D., et al. (2022). 'No evidence for nudging after adjusting for publication bias.' PNAS, 119(31), e2200300119. doi:10.1073/pnas.2200300119
A note on what is not claimed: this article does not say the ego depletion effect itself is dead. It says two of its supports failed. The direct multi-site replications of the effect are a separate literature and are not cited here.