Uri Simonsohn

Who they are

Simonsohn's work on this site is about method rather than about any one result, and it operates in two directions at once.

The first is building infrastructure. He is a co-founder of AsPredicted, which is a preregistration service, and of Data Colada, a blog on research methods written with Joseph Simmons and Leif Nelson. With those two he is a co-author of the paper that popularised the term p-hacking, "False-Positive Psychology", which demonstrated that ordinary analytic flexibility, the everyday choices about which participants to exclude and which measures to report, can manufacture a statistically significant result almost at will. That paper is routinely cited as Simmons and Simonsohn, dropping Leif Nelson, who is its second author. This site names all three.

The second direction is correcting overstatement in the critical literature, which is less expected and more interesting. The 2015 Open Science Collaboration reproducibility project is usually summarised as showing that about sixty per cent of psychology failed to replicate. Simonsohn's 2016 reanalysis, published as Data Colada 47, partitions the hundred studies differently: roughly forty replicated, roughly thirty clearly failed, and roughly thirty were statistically inconclusive either way. An ambiguous result is not a refuted one. It is also not a rescue, and the honest version of the correction says both halves: the failure rate is lower than advertised, and forty out of a hundred, from a set of papers of which ninety-seven had reported a positive finding, is a bad number by any reading. What gets worse under his reanalysis is the informativeness of the original literature rather than its error rate.

The third piece the vault holds is a specific reanalysis. Brett Pelham, Matthew Mirenberg and John Jones reported in 2002 that people are disproportionately likely to enter careers and live in places resembling their own names, framed as implicit egotism, with the much-quoted finding of more dentists called Dennis than would be expected. Simonsohn argued in 2011 that these effects largely reflect confounds including ethnicity, birth cohort and reverse causality. Walter, for instance, is overwhelmingly a name of older generations. Pelham and colleagues have since published responses, and this site records the exchange rather than declaring it closed.

Key papers

Simmons, J. P., Nelson, L. D., & Simonsohn, U. "False-Positive Psychology." The demonstration that ordinary analytic flexibility can manufacture statistical significance, and the paper that popularised p-hacking. Joseph Simmons is first author and Leif Nelson, routinely dropped from the credit, is second. The year and journal are deliberately not recorded in the vault and are not supplied here.

Simonsohn, U. (2011). Journal of Personality and Social Psychology, 101(1), 1-24. The argument that name-letter effects on careers and residence largely reflect confounds such as ethnicity, birth cohort and reverse causality. Read against Pelham, B. W., Mirenberg, M. C., & Jones, J. T. (2002), Journal of Personality and Social Psychology, 82(4), 469-487. Pelham and colleagues have since published responses.

Simonsohn, U. (2016). Data Colada [47], 3 March 2016. The partition of the Open Science Collaboration results into roughly forty replicated, thirty failed and thirty statistically inconclusive. A blog post rather than a peer-reviewed paper, and this page says so.

His work on eBay auction timing, where sellers cluster their endings at the same hour without reasoning about the other sellers doing the same calculation, reaches the vault through another book's endnotes and is not described there in enough detail to cite as a study.

Key ideas

Analytic flexibility manufactures findings. The p-hacking demonstration is about ordinary undisclosed choices rather than about anyone's intent, and it is the reason preregistration exists.

Preregistration as infrastructure. AsPredicted is the practical follow-through: write down what you will test before you run it.

An inconclusive result is not a failed one. The partition of the reproducibility project into forty, thirty and thirty is the standing correction to the sixty-per-cent summary.

And the correction cuts both ways. Forty in a hundred, from papers of which ninety-seven reported a positive finding, is a bad number. What his reanalysis worsens is the informativeness of the original literature.

Confounds before conclusions. The name-letter reanalysis is the pattern in miniature: a striking effect, a plausible alternative explanation in cohort and ethnicity, and an exchange that the field should keep open rather than settle by citation count.

Name all three authors. Leif Nelson is the second author of the p-hacking paper and is usually left out of the credit.

---