Effect Size Benchmarks

Effect Size Benchmarks

Reviewed by Faisal Zaidi

An effect size means nothing on its own. Whether 0.11 standard deviations is trivial or impressive depends on what it is being compared with, and the comparison distribution is almost never printed beside the number. In the growth mindset dispute the two camps use benchmarks that differ by a factor of about twenty at the median, which is enough on its own to explain why one side reads the same result as a success and the other as a failure. The rule this page exists to enforce: give the reader the yardstick before the number.

What it is

The first yardstick is 0.57 standard deviations, offered as a typical educational intervention effect. David Yeager of the University of Texas at Austin and Carol Dweck of Stanford object that this figure comes from a 1996 meta-analysis by Hattie, Biggs and Purdie of learning-skills manipulations, mostly measured at immediate post-test on researcher-designed measures that sit close to a manipulation check. On that standard, almost nothing in real schooling looks large.

The second is drawn from third-party preregistered field trials with no file drawer. Matthew Kraft, in 2020, located every study funded by the United States federal Investing in Innovation fund, where outcomes had to be pre-specified and reported regardless of significance. The median effect was 0.03 standard deviations and the ninetieth percentile 0.23, with no effects above 0.23 among adolescents. Kraft's own conclusion is that effects of 0.15 or even 0.10 should be considered large and impressive when they arise from large-scale preregistered field experiments examining broad achievement measures. On that standard, 0.11 sits near the top of the distribution.

The reference points are worth more than either benchmark, because they translate the units into things a reader already understands. Roughly 0.20 standard deviations is about a year of ninth-grade classroom learning, and about the gain from having a very high-quality rather than a below-average teacher for a year. Drastically reducing elementary class sizes was 0.20. For comparison outside education: descriptive norms on energy conservation 0.03, implementation intentions on flu vaccination 0.12, and simplifying a financial aid form on college enrolment 0.16. Yeager and Dweck's summary is that in the real world single variables do not have huge effects, and nor do large, expensive, years-long reforms.

The sceptical use of the same idea is just as legitimate and reaches a different place. Carolina Gazmuri of the Durham University Centre for Evidence in Education rated 24 randomised trials by the sieve approach that underlies the Education Endowment Foundation's own security ratings, and reports that the strongest studies, by sample size, missing data and data quality, give effects from d = -0.01 to +0.065. Against the Kraft distribution those are small rather than nil. Against the Hattie distribution they are nothing at all. Both camps are reading real numbers and reaching for different rulers.

There is one translation on this subject that should be labelled as arithmetic rather than as a finding. Because over three million students enter ninth grade each year in the United States and half are lower-achieving by the study's definition, a 5.3 percentage point reduction in the share below a 2.0 grade average implies on the authors' own calculation about ninety thousand at-risk students a year kept on track. That is a multiplication, not a measurement, and it is the proponents' own.

In effect

The same number read twice

The research case is the national mindset experiment's headline. Its effect on the grades of its preregistered focal group is 0.10 grade points, a standardised 0.11. Set against 0.57 it is a rounding error; set against a median of 0.03 and a ninetieth percentile of 0.23 it is a strong result for a free, under-an-hour, self-administered online lesson. Nothing about the study changes between those two readings. Only the denominator does.

Why a school governor cannot use a bare effect size

In ordinary decision-making the benchmark question is the only question. A governor told that a programme produces 0.11 standard deviations learns nothing without being told what else is on offer at what cost. Told that a year of classroom teaching is about 0.20, that an excellent teacher rather than a poor one for a year is about the same, and that the median federally funded education trial produces 0.03, the governor can work out whether a free online lesson delivering a tenth of a grade point to the lower-achieving half of a year group is worth an hour of the timetable. The honest answer in that frame is probably yes, and the honest answer in the other frame is probably no, and the frame is the argument.

The cost line that decides it, written by a sceptic

The practical version comes from the Education Endowment Foundation's cost figures for the first English mindset trial: pupil workshops at 397 pounds per pupil per year over three years, and teacher training at sixteen pounds per pupil per year, 678 pounds per school per year, or forty-nine pounds per pupil over the single evaluation year. The expensive arm moved the needle non-significantly and the cheap arm did nothing. Gazmuri's closing sentence is the one worth quoting because she is the hardest independent sceptic in the file: if there is an opportunity to implement such interventions at a minimal or negligible cost, or as part of another objective, it might be reasonable to proceed with them, considering the potential for a small positive impact.

What it does not say

It does not say small effects are always worth having. A small effect with a large delivery cost is a bad buy, which is what the English workshop arm's cost per pupil shows.

It does not say the Kraft benchmark is the only correct one. It says the benchmark must be named, and that a benchmark built from immediate post-tests on researcher-made measures is not comparable with one built from preregistered field trials on national assessments.

It does not supply a threshold. There is no number above which an effect becomes real; there is only a distribution to compare it with.

It does not convert an effect size into people. The ninety-thousand-students figure is the proponents' arithmetic on their own result and is labelled as such everywhere this site uses it.

Who developed it

No single originator, and this page claims none. The specific education-trial benchmark is attributed to Matthew Kraft's 2020 analysis by Yeager and Dweck, whose 2020 American Psychologist review was read in full; Kraft's own paper was not read. The sceptical application is Carolina Gazmuri's 2025 structured review, read in full and open access.

Sources

  1. Yeager, D. S., & Dweck, C. S. (2020). "What can be learned from growth mindset controversies?" American Psychologist. Full accepted manuscript read at source at par.nsf.gov. The benchmark argument, the Kraft figures, the classroom-year comparisons and the three behavioural-science reference points all come from this paper's report of them.
  2. Kraft, M. A. (2020), on the Investing in Innovation distribution, and Hattie, J., Biggs, J., & Purdie, N. (1996). Neither read; both reach this page through Yeager and Dweck 2020.
  3. Gazmuri, C. (2025). "Can growth mindset interventions improve academic achievement? A structured review of the existing evidence." Review of Education, 13, e70066. doi:10.1002/rev3.70066. Full open-access text read at source.
  4. Yeager, D. S., and colleagues (2019). Nature, 573, 364-369. Full text read at source.
  5. Education Endowment Foundation (2014/2017), "Changing Mindsets" first trial, University of Portsmouth, evaluated by NIESR and published as Rienzo and colleagues 2015. Project page read in full at source; the evaluation report itself was not read. The cost figures are the EEF's.