Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Many Labs 5 tested the most common defence offered when a replication fails, that the replication was badly designed. The team selected ten studies from an earlier large replication project for which the original authors had objected to the protocol before the data were collected, only one of which had produced a significant effect. They revised each protocol, submitted the revisions for formal expert review before collecting anything, and then ran both the old protocol and the improved one in multiple laboratories with large samples. The revised protocols produced effects essentially identical to the original replication protocols. Both remained far below the effects reported in the original published studies, which were several times larger. So expert review of design before data collection, on this evidence, does not recover the missing effects. The paper is careful and pre-registered, and it closes off an explanation that had been used often. This vault cites it whenever a failed replication is dismissed on procedural grounds.
Many Labs 5 tested the most common defence offered when a replication fails, that the replication was badly designed. The team selected ten studies from an earlier large replication project for which the original authors had objected to the protocol before the data were collected, only one of which had produced a significant effect. They revised each protocol, submitted the revisions for formal expert review before collecting anything, and then ran both the old protocol and the improved one in multiple laboratories with large samples. The revised protocols produced effects essentially identical to the original replication protocols. Both remained far below the effects reported in the original published studies, which were several times larger. So expert review of design before data collection, on this evidence, does not recover the missing effects. The paper is careful and pre-registered, and it closes off an explanation that had been used often. This vault cites it whenever a failed replication is dismissed on procedural grounds.
Written from the abstract, OpenAlex.
What our sources record
Reproduced as it was written when this source was checked, figures and all.
Corker, K. S., et al. (2020). "Many Labs 5." AMPPS. Expert-revised, peer-reviewed-before-data-collection protocols produced r = .05 against r = .04 for the unrevised protocols, at a median N of 1,279.5 versus an original median of 76.
Where we use it
- What Forty-One Participants Buys, Why We Look
Reliability of this reference
The link below was matched against the publisher's record on title, first author and year, and all three agree.
The source
https://doi.org/10.1177/2515245920958687
DOI: 10.1177/2515245920958687