Megastudy

What it is

Behavioural interventions are usually tested one at a time, in different places, with different measures. That makes it hard to say which of two ideas works better, because the studies differ in so many ways. A megastudy removes those differences. In the 2021 study, 30 scientists from 15 US universities worked in small independent teams to design 54 four-week digital programmes encouraging exercise among members of the 24 Hour Fitness chain. Members were randomly assigned to a programme, and every programme was judged on the same outcome, weekly gym visits.

In effect

About 45 per cent of the interventions significantly increased weekly gym visits, by 9 to 27 per cent. The best performer offered small rewards for returning to the gym after a missed workout. Two audiobook conditions did well: a free audiobook on its own ranked fourth of the 53 conditions compared with the placebo control, and a free audiobook with temptation bundling explained ranked ninth. Only 8 per cent of the interventions produced changes that were significant and measurable after the four weeks ended.

The study also asked professors, practitioners and members of the public to forecast the results in advance. Their forecasts failed to predict which interventions would be most effective, which the authors present as an argument for testing many ideas at once.

What it does not say

A megastudy does not make each of its arms an independent new study. The 2020 temptation bundling experiment led by Erika Kirgios was one of the twenty studies embedded in the same programme, so its results and the megastudy's audiobook results come from the same members.

Nor does a strong result during a four-week programme show a lasting habit. Most interventions that worked during the programme did not produce clearly detectable effects afterwards.

---