Moderator Analysis

What it is

The procedure is standard and the discipline is not. Split too coarsely and real differences disappear into an average that describes no study in the set. Split too finely and cells shrink until a headline rests on a handful of studies. Both failures are common and the second is harder to spot, because a narrow cell with a large effect reads as a precise finding rather than a thin one.

The reward literature is the best worked example available. The 1999 meta-analysis of 128 studies reports that rewards undermine free-choice persistence. Moderated, the same data gives six different pieces of advice. Verbal reward and positive feedback enhance, at d = +0.33, significantly for college students at +0.43 and not significantly for children at +0.11. Unexpected tangible rewards do nothing, at d = +0.01 across nine homogeneous studies. Rewards not tied to task behaviour do nothing significant, at d = -0.14 across seven studies. Expected tangible rewards undermine, at d = -0.36. And uninteresting tasks were excluded from the primary analyses by design, because the authors hold that the phenomenon has always been specified as applying only to interesting tasks.

The cost of splitting finely appears in the same paper. The performance-contingent category was divided four ways by type of control group and feedback valence, and the row the authors call the real-world one carries the largest undermining effect in the entire analysis, d = -0.88, on six studies. Both facts have to travel together.

The methodological dispute deserves to be stated as a dispute. Judy Cameron and W. David Pierce argue that aggregating effect sizes within a study is standard and avoids violating the independence assumption, that entering several effect sizes from one experiment is a serious statistical violation because the errors among observations are correlated, and that properly accounting for the dependency would enlarge the error term and shrink mean effects. They also state plainly what they did about heterogeneity: they set the critical value of Q just below the .01 level on the grounds that the test too often rejects homogeneity, achieved homogeneity by excluding extreme effect sizes, and reported every analysis with extreme values both included and excluded. The 1999 authors state the opposite choice just as plainly: because performance-contingent effects were not homogeneous, they examined four categories rather than discarding outliers. Two defensible procedures, two different answers, both printed in full.

The discipline that sits above both is bias correction. Petr Cala of the Institute of Economic Studies at Charles University, with Tomas Havranek, Zuzana Irsova, Jindrich Matousek and Jiri Novak, took 44 experimental economics studies on financial incentives and performance, found evidence of publication bias by funnel asymmetry and by the under-reporting of estimates just short of significance at the five per cent level, and report that after correction the mean experimental result suggests a negligible effect. No amount of moderating fixes a biased pool. That paper is a CEPR discussion paper, is not peer-reviewed, and was not read: its content reaches this site through the authors' own summary column.

In effect

Six findings out of one headline

The clearest research case is the 1999 table itself. A reader who learns only that rewards undermine motivation has been told something that is true of one cell and false of three others in the same analysis. A reader who learns the moderator structure can act on it: praise the work, do not promise a prize in advance for an activity somebody already enjoys, and do not expect either result to say much about a salary.

Why two experts can both be honest and disagree

In everyday terms, the dispute between the two meta-analytic traditions is not a dispute about numbers. Both camps agree that praise enhances, that unexpected rewards do nothing and that expected tangible rewards on interesting tasks reduce later unpaid persistence. They disagree about whether to aggregate several measurements from one experiment into one, how finely to subdivide performance-contingent rewards, whether self-report counts as an outcome, and whether dull tasks belong in the analysis. Those are procedural choices with reasons behind them, which is why the disagreement has lasted thirty years without either side making an error anybody can point at.

The reading rule for a pooled number in a strategy deck

When a single effect size arrives in a business case, the useful question is what it was averaged over. Ji Hyun Kim of the University of Wisconsin-Madison, with Barry Gerhart and Meiyu Fang, demonstrated the value of asking it: they took the primary studies from two work-domain meta-analyses whose headline results sat about nine tenths of a standard deviation apart on incentives and interesting tasks, re-analysed them together, and found the relationship positive in both interesting tasks at delta = +.58 and dull ones at +.52, with the single exception that it is less positive where performance is measured as quality. Two opposite headlines, one reconciled answer, produced by looking at what each pool contained.

What it does not say

It does not make a pooled estimate meaningless. An average over a homogeneous set of studies is informative, and the test for homogeneity is the thing to look for.

It does not make a moderated cell trustworthy. A large effect on six studies is a large effect on six studies.

It does not substitute for bias correction. A pool skewed by what got published cannot be repaired by subdividing it.

It is not a licence to go looking. Choosing moderators after seeing the results is how a null becomes a finding, which is why preregistration and moderator analysis belong on the same page.

Who developed it

There is no single originator, and this page claims none. The procedure is standard in meta-analysis. The worked examples belong to Edward Deci, Richard Koestner and Richard Ryan on one side and Judy Cameron and W. David Pierce on the other, with the bias correction from Petr Cala and colleagues and the reconciliation from Ji Hyun Kim and colleagues. Elizabeth Tipton's commentary on heterogeneity in the growth-mindset literature makes the same argument in a different field.

---