Evaluating replicability of laboratory experiments in economics
Camerer and colleagues repeated eighteen laboratory experiments published in the two leading economics journals between 2011 and 2014. The methodology matters as much as the result: every replication followed an analysis plan published in advance, and each was powered well enough to have a nine in ten chance of detecting the original effect if it was real and the same size. Eleven of the eighteen produced a significant effect in the same direction as the original, which is around three in five, and the average replicated effect was about two thirds the size of the published one. Other ways of scoring replicability put the rate between two thirds and three quarters, including a measure based on a prediction market in which other researchers bet on which studies would hold. That last point recurs across replication projects: the field can often tell in advance. This vault cites it alongside the psychology and cancer biology projects, since together they show the problem is not one discipline's.
Camerer and colleagues repeated eighteen laboratory experiments published in the two leading economics journals between 2011 and 2014. The methodology matters as much as the result: every replication followed an analysis plan published in advance, and each was powered well enough to have a nine in ten chance of detecting the original effect if it was real and the same size. Eleven of the eighteen produced a significant effect in the same direction as the original, which is around three in five, and the average replicated effect was about two thirds the size of the published one. Other ways of scoring replicability put the rate between two thirds and three quarters, including a measure based on a prediction market in which other researchers bet on which studies would hold. That last point recurs across replication projects: the field can often tell in advance. This vault cites it alongside the psychology and cancer biology projects, since together they show the problem is not one discipline's.
Written from the abstract, OpenAlex.
What our sources record
Reproduced as it was written when this source was checked, figures and all.
Camerer, C. F., et al. (2016). Science, 351(6280), 1433-1436, experimental economics at 11/18. Camerer, C. F., et al. (2018). Nature Human Behaviour, 2(9), 637-644, Nature and Science social science at 13/21.
Where we use it
- What Forty-One Participants Buys, Why We Look
Reliability of this reference
The link below was matched against the publisher's record on title, first author and year, and all three agree.
The source
https://doi.org/10.1126/science.aaf0918
DOI: 10.1126/science.aaf0918