John Protzko
John Protzko is a psychologist at Central Connecticut State University and first author of the paper that announced psychology's reforms had worked. Published in Nature Human Behaviour in November 2023, it reported that 86 per cent of newly discovered findings replicated under rigorous conditions. It was retracted in September 2024, partly because the analyses behind that claim had not been preregistered, contrary to what the paper said.
Who they are
Protzko works on intelligence, developmental interventions and metascience. The retracted paper was an ambitious piece of work and it is worth describing accurately, because the retraction is usually reported as though the study were shoddy, and it was not.
Four laboratories, at Stanford, UC Santa Barbara, Berkeley and Virginia, each generated four new findings under deliberately rigour-enhancing conditions: large samples, preregistration, blinding, confirmatory testing. Each of the sixteen discoveries was then replicated four times, once by the originating lab and three times independently. Eighty data collections in total. The headline results were that 86 per cent of replication attempts reached significance in the predicted direction and that the replication effect sizes were 97 per cent the size of the originals.
Joseph Bak-Coleman and Berna Devezer published the critique that prompted the retraction. Their point was that replicability had not been the project's original outcome of interest, and that the analyses supporting the claim in the title were not preregistered even though the paper asserted they were.
Nature Human Behaviour retracted the paper on 24 September 2024, citing lack of transparency, misstatement of the hypotheses the meta-study was designed to test, the preregistration discrepancy, selection of outcome measures with knowledge of the data, and incomplete reporting. All seventeen authors agreed to the retraction over the preregistration point while disputing the other concerns. They were invited to resubmit.
Two readings are usually offered and both are half right. The retraction is not evidence that the reform failed: a gap between what a paper claimed and what it did was found by outsiders, conceded by the authors and recorded in public inside ten months, which is the machinery working. It is also not evidence that 86 per cent is the right number, because the grounds for retraction are precisely the mechanism that inflates published estimates. The honest position is that nobody currently knows what proportion of new findings replicate under rigorous conditions.
Key papers
Key papers
Protzko, J., Krosnick, J., Nelson, L., Nosek, B. A., and colleagues (2024). Nature Human Behaviour, 8(2), 311-319. Retracted. First author. Four laboratories each generated four new findings under deliberately rigour-enhancing conditions, large samples, preregistration, blinding and confirmatory testing, and each of the sixteen was then replicated four times, once in house and three times independently, across eighty data collections. The reported headline was that 86 per cent of replication attempts reached significance in the predicted direction, at effect sizes 97 per cent the size of the originals.
Bak-Coleman, J., & Devezer, B. (2024). "Claims about scientific rigour require rigour." Nature Human Behaviour, 8(10), 1890-1891. The critique that prompted the retraction. Replicability had not been the project's original outcome of interest, and the analyses supporting the claim in the title were not preregistered although the paper said they were.
Retraction note (2024). Nature Human Behaviour, 8, 2067. DOI 10.1038/s41562-024-01997-3. The journal cited lack of transparency, misstatement of the hypotheses the meta-study was built to test, the preregistration discrepancy, selection of outcome measures with knowledge of the data, and incomplete reporting. All seventeen authors agreed to the retraction on the preregistration point while disputing the other concerns, and were invited to resubmit.
The consequence for anyone citing this literature is narrow and worth stating exactly. The figure is unusable rather than disproved, and no one currently knows what proportion of new findings replicate under rigorous conditions.
Key ideas
The most rigorous design still needs the rules followed. Blinding, large samples and independent replication did not substitute for a preregistration that matched the analysis.
A retraction is an outcome of the system, not a failure of it, and reading it either way alone is a mistake.
The number is unusable now. Not disproved, unusable, which is a different and more awkward thing to say.
Concepts on this site
Sources
- Protzko, J., Krosnick, J., Nelson, L., Nosek, B. A., et al. (2024). Nature Human Behaviour, 8(2), 311-319. Retracted.
- Retraction note (2024). Nature Human Behaviour, 8, 2067. DOI 10.1038/s41562-024-01997-3.
- Bak-Coleman, J., & Devezer, B. (2024). "Claims about scientific rigour require rigour." Nature Human Behaviour, 8(10), 1890-1891.