Heterogeneous Treatment Effects

What it is

The idea is uncontroversial in statistics and medicine, where subgroup variation has been the standard worry for decades. The forceful behavioural-science statement is Christopher Bryan, Elizabeth Tipton and David Yeager's 2021 paper in Nature Human Behaviour, arguing that the field's habit of chasing average effects in convenience samples explains many small, unstable and non-replicating results. Tipton, a statistician, is also the author of the generalizability index used to establish that the mindset study's sample was nationally representative, and a co-author of that study, which is a conflict a reader should know about and which her own commentary discloses.

The mindset experiment is worth describing because it separates the two questions properly. Yeager and colleagues, writing in Nature in 2019, randomised 12,490 ninth-graders individually within 65 regular United States public high schools drawn from a frame of roughly twelve thousand, reaching a generalizability index of 0.98 against eight benchmarks where 0.90 is essentially a random sample. The intervention was two self-administered online sessions of about twenty-five minutes against an active control about brain function, with no teacher training. Median school fidelity was ninety-seven per cent of screens viewed.

On beliefs there was no heterogeneity across schools, with Q64 = 57.2 and p = 0.714, which means the lesson was delivered evenly and worked evenly as a lesson. On grades there was significant heterogeneity, with Q64 = 85.5 and p = 0.038. The two preregistered moderators locate where. Effects on lower achievers' grades were smaller in higher-achieving schools, with an interaction of B = -0.07, z = -2.76 and a standardised beta of -0.25. Behavioural challenge-seeking norms, measured by averaging control-group students' choices on a make-a-worksheet task, moderated positively, at B = 0.11, z = 2.18 and beta = 0.23, while self-reported mindset norms did not moderate at all.

Combining the two gives the shape. Within low- and medium-achieving schools the effect on core grades was 0.14 grade points in the third quartile of behavioural norms and 0.18 in the fourth, against an overall figure of 0.10 for the targeted group. In the highest-achieving quarter of schools the effect on grades was near nothing, and those schools instead gained four percentage points on advanced mathematics course-taking, where the interaction runs the opposite way. The authors' own line is that an exclusive focus on grade point average would have obscured intervention benefits among students attending higher-achieving schools.

The limit is theirs and it applies to every moderator here: the norms were measured, not manipulated, so no causal conclusion about the norm itself is available. The same constraint applies to the 2022 follow-up matching 9,167 student records with 223 mathematics teachers, which uses the same dataset and therefore inherits its strengths without being able to corroborate it.

The meta-analytic dispute in this literature is, at bottom, a disagreement about this page. Brooke Macnamara and Alexander Burgoyne aggregated effect sizes within each study before combining them and tested moderators one at a time on small subgroups, reaching d-bar = 0.05 across 63 studies and 0.02 across the six strongest. Jeni Burnette and colleagues modelled variation within studies across subgroups and outcomes using multilevel metaregression, reaching d = 0.14 in targeted high-fidelity subsamples with a ninety-five per cent prediction interval running from minus 0.08 to 0.35. Tipton and colleagues argue the heterogeneity-attuned approach is current best practice and report, in a reanalysis they themselves label exploratory, that applying it to the sceptics' dataset recovers a meaningful effect in focal groups; a reply in the same journal questions the coding, effect-size and inclusion changes that reanalysis required, arguing such decisions should be made in advance and applied consistently.

In effect

Uniform belief change, variable grades

The research case is the Q statistic pair, because it is as close as this literature comes to a clean dissociation. The same lesson, delivered to the same standard across sixty-five schools, moved beliefs by the same amount everywhere and moved grades by different amounts in different places. That rules out one tempting explanation, that the variation is just sloppy delivery, and points at the schools rather than the script. It also means a school average is the wrong unit for the question anybody actually wants answered.

The tax letter that works on everyone except the people who owe most

Outside education the standard example is a compliance letter. Telling people that most of their neighbours pay on time raises payment overall, while the largest debtors, who see themselves as exceptions, respond better to a message about what everyone loses when tax goes unpaid. A department that rolled the winning average message out to every segment would do worse with its most valuable cases. The principle is robust; the particular tax figures circulate through secondary accounts rather than a published trial report, and this site treats them that way.

Why a pooled A/B result can mislead a product team

A test that lifts a metric by one per cent on average may be lifting it four per cent for new users and depressing it two per cent for the heaviest ones. The average is real and the decision it supports may be wrong. The practical version of this page for a product organisation is to preregister the two or three subgroups that theory says should differ, as the mindset study did with exactly two moderation hypotheses, rather than hunting through segments after the result arrives. The difference between those two procedures is the difference between a finding and a story.

What it does not say

It does not say averages are worthless. An average over a genuinely homogeneous set is the right summary, and the homogeneity test is what tells you which case you are in.

It does not license post-hoc subgroup hunting. Variation is a reason to specify subgroups in advance, not a reason to look for one that reaches significance.

It does not make a moderator causal. Measured moderators, including the school norms in the mindset study, describe where an effect appeared rather than why, and the authors say so.

It does not settle the mindset dispute. Two teams applied two defensible pooling methods to overlapping data and got 0.02 and 0.14, and both published their intervals.

Who developed it

No single originator, and this page claims none. The modern behavioural-science argument is Christopher Bryan's with Elizabeth Tipton of Northwestern University and David Yeager of the University of Texas at Austin, in Nature Human Behaviour in 2021. The worked example is the National Study of Learning Mindsets, first-authored by Yeager.

---