Michael O'Donnell

Who they are

The finding under test was one of the most quotable in social psychology. Ap Dijksterhuis and Ad van Knippenberg reported in 1998 that participants asked to think about a professor before answering general-knowledge questions did better than participants asked to think about a football hooligan, with a difference of around thirteen percentage points. The result was read as evidence that merely activating a stereotype could change intellectual performance, and it travelled widely on that reading.

O'Donnell, with Leif Nelson, Evelina Ackermann and a large group of collaborators, ran it as a registered replication report. The design removes most of the room for argument after the fact. The protocol was agreed with the original authors in advance and is the one they preferred, the analysis was specified before the data existed, and every qualifying laboratory's result is included whatever it shows. Twenty-three laboratories took part and 4,493 participants were tested.

The effect came back at 0.14 per cent, with a confidence interval running from about minus 0.7 per cent to about 1 per cent. The interval contains zero comfortably and excludes anything close to the original thirteen points.

The general lesson is the one this site takes from the paper, and it is about arithmetic rather than about anyone's integrity. A study with a few dozen participants can only detect an effect if that effect is very large, so the only results such a study can publish are large ones, whether or not the underlying effect is large or exists at all. Run the same question at a hundred times the sample and the honest answer surfaces. That is the whole argument for statistical power, made once, at scale, with the original authors' own materials.

Key papers

O'Donnell, M., Nelson, L. D., Ackermann, E., et al. (2018). "Registered replication report: Dijksterhuis and van Knippenberg (1998)." Perspectives on Psychological Science, 13(2), 268-294. Twenty-three laboratories and 4,493 participants, run on the original authors' preferred protocol. Effect 0.14 per cent, with a confidence interval from about minus 0.7 per cent to about 1 per cent. O'Donnell is first author.

The study it tests:

Dijksterhuis, A., & van Knippenberg, A. (1998). The professor-priming result, reporting a gain of around thirteen percentage points in trivia performance. Full citation details are not recorded in the vault and are not supplied here.

Key ideas

Registered replication removes the arguments that usually follow a null result. Protocol agreed in advance with the original authors, analysis specified before the data, every qualifying laboratory included.

Thirteen points became a fifth of one point, with an interval that comfortably contains zero.

Small samples can only publish large effects. That is a fact about the arithmetic of detection, not about the honesty of the researchers, and it is the reason a literature can look consistent and be uninformative.

Run it at scale on the original authors' own materials, and there is very little left to dispute about the procedure.

---