Anne Scheel

Who they are

Scheel works on research methods and the epistemics of psychological science, including the argument that the field skipped a stage: it went straight to hypothesis testing without first doing the descriptive, exploratory work that tells you what there is to test.

The 2021 paper with Mitchell Schijen and Daniel Lakens is the one most often cited, and it is unusually well built. It is not a sample of Registered Reports; it is all of them, the complete population as of November 2018, which is seventy-one papers. The comparison set of 152 ordinary articles was drawn by reproducing Fanelli's earlier search method, so the two groups were identified the same way.

The result is a fifty-two point gap. Thirty-one of seventy-one Registered Reports reported a positive result against 146 of 152 standard reports.

Two things about that number are routinely misreported and this site holds to the authors' own definitions. "Positive" was not a p-value threshold. Following Fanelli, the coders recorded whether the authors themselves concluded that the first stated hypothesis was supported, and partial support counted as positive. Writing it as "96 per cent of psychology papers report significant results" is wrong. And the authors state the obvious caveat themselves: more than half the Registered Reports were close replications, which plausibly lowers their hit rate. Excluding those, the gap narrows and persists.

What the paper does not do is separate publication bias from Type I error inflation, and the authors say so. It establishes that something is filtering the ordinary literature. It does not say how much of the filtering happens at the journal and how much in the analysis.

Key papers

Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). "An excess of positive results: comparing the standard psychology literature with Registered Reports." Advances in Methods and Practices in Psychological Science, 4(2). doi:10.1177/25152459211007467

Fanelli, D. (2010). "'Positive' results increase down the hierarchy of the sciences." PLoS ONE, 5(4), e10068. The search method Scheel reproduced.

Key ideas

Measure the format, not the finding. The comparison is between publishing systems rather than between topics, which is what makes the gap interpretable.

A full population beats a sample. Using every Registered Report in existence removes the selection worry that would otherwise apply to the study of selection.

Define your terms. "Positive" means the authors said their hypothesis was supported, and reporting it as a significance rate overstates it.

---