Reading the Implicit Association Test
The implicit association test returns a difference in milliseconds, and it has long been read as a measure of hidden bias. Two meta-analyses of the same literature put its link to discriminatory behaviour at about 0.24 and about 0.15, its own defenders warn against using it on individuals, and moving a score has not clearly moved conduct.
Answer to the question
Does the implicit association test measure bias?
It measures how much faster a person sorts words under one category pairing than under its reverse. Whether that difference marks bias, and predicts discrimination, is disputed: pooled correlations with behaviour run from about 0.15 to about 0.24, depending on which studies count, and a score is less stable over time than a stated attitude.
Key takeaways
- The implicit association test returns a difference in reaction times between two category pairings. Calling that difference bias is an interpretation, tested by what the score predicts.
- Two meta-analyses disagree: Greenwald and colleagues (2009) found about 0.24 between the race version and discriminatory behaviour, against about 0.12 for self-report, while Oswald and colleagues (2013) found about 0.15.
- Most of the gap traces to inclusion rules, and Greenwald, Banaji and Nosek (2015) argue that correlations of that size can matter across whole populations or repeated encounters while cautioning against classifying individuals.
- Implicit measures were less stable over time than matched explicit measures, about 0.54 against about 0.75 (Gawronski and colleagues, 2017), and a 2019 pooled analysis of 492 studies found changes in them did not mediate changes in behaviour.
A screen shows a word and two response keys, each assigned to a pair of categories. The participant sorts as fast as possible while software records the milliseconds. The gap in speed between one pairing and its reverse is the whole output of the implicit association test.
What the score records, and why it spread so far
Anthony Greenwald and colleagues introduced the test in 1998, and its score has been read ever since as a measure of hidden bias. Whether it measures that is a separate question from whether it measures anything. The 1998 paper in the Journal of Personality and Social Psychology reported three experiments. The test separated flower names from insect names on evaluative grounds and found the expected differences between Japanese and Korean participants. In the third experiment it found evaluative biases among White participants who described themselves as unprejudiced, biases the participants consciously disavowed. That result carried the test out of the laboratory. It offered access to something that asking could not reach.
The promise had a ready audience in consumer research. Philip Graves's Consumer.ology, published in 2010 and revised in 2013, argues that the reasons people give for their behaviour are built after the fact from theories about what plausibly influences them, a position he takes from Nisbett and Wilson's 1977 review in Psychological Review. His version of the self-report problem is that asking changes the person asked, so the answer describes a research subject, not a shopper. His remedy is behavioural: watch what people do, in the real setting, unaware of what is under study. A reaction-time score looks like a third route, one that neither asks nor waits for the purchase, which is the appeal of the implicit association test.
A score is a difference in latencies. Calling it a bias is an interpretation laid over the difference.
A score is a difference in latencies. Calling it a bias, or an attitude the person does not know they hold, is an interpretation laid over the difference, and the question is one of construct validity. Predictive validity is the test of the interpretation, and so the criterion studies carry the argument. The implicit association test has travelled well beyond race research, including into a study of attitudes to creativity under uncertainty, which makes the question of what a score predicts more general than any one application.
Same literature, different totals: why two meta-analyses diverged on prediction
Two meta-analyses of the same literature disagree on the answer. Greenwald and colleagues pooled studies in 2009 and found that for racially discriminatory behaviour the race version of the implicit association test correlated with the criterion at about 0.24, against about 0.12 for self-report. Frederick Oswald and colleagues worked from a broader set of studies in 2013, found about 0.15 for the same category, and described the implicit association test as a poor predictor of who will discriminate against whom.
| Study | What was correlated | Correlation |
|---|---|---|
| Greenwald and colleagues, 2009 | Race version with discriminatory behaviour | about 0.24 |
| Greenwald and colleagues, 2009 | Self-report with discriminatory behaviour | about 0.12 |
| Oswald and colleagues, 2013 | Race version with criterion measures | about 0.15 |
| Gawronski and colleagues, 2017 | Implicit measures, stability across time | about 0.54 |
| Gawronski and colleagues, 2017 | Matched explicit measures, stability across time | about 0.75 |
Greenwald, Mahzarin Banaji and Brian Nosek went back to the gap in 2015 and traced most of it to inclusion rules. The 2009 analysis counted only correlations with a theoretical reason to expect a relationship; the 2013 analysis included many without one. Neither side disputes that the numbers from the implicit association test are small. The disagreement is about what small means. The 2015 paper argues that correlations of that size can still produce socially significant discrimination when a bias touches many people at once, or the same person repeatedly. A difference small enough to yield a correlation near 0.08 can disadvantage a meaningful share of a protected group after ten to twenty-five repetitions, across successive applications, evaluations or appointments.
Does a score hold still, and does it move anyone?
The same paper also concedes the point the sceptics press hardest. The implicit association test's modest test-retest reliability and small effects, its authors write, make it risky for classifying individuals, though those limits matter less for analyses of whole systems. Longitudinal data fit that caution. Bertram Gawronski and colleagues followed self-concept, racial attitudes and political attitudes in two studies and found implicit measures less stable over time than matched explicit measures, with a weighted average correlation of about 0.54 against about 0.75, even though internal consistency was comparable. A person's score on one afternoon is a weaker guide to the same person's score later than their stated attitude is.
Moving a score does not clearly move conduct either. Patrick Forscher and colleagues pooled 492 studies with 87,418 participants in 2019. Implicit measures could be changed, mostly by brief single-session procedures and mostly weakly, with effects under 0.30 in absolute size. Changes in behaviour were generally trivial, and changes in implicit measures did not mediate changes in explicit measures or behaviour.
So the evidence supports a narrow reading and leaves the broad one open. The implicit association test reliably produces a reaction-time difference. That difference correlates with some discriminatory behaviour at somewhere between 0.15 and 0.24 depending on which studies count, it is less stable than a stated attitude, and shifting it has not been shown to shift what people do. Whether the score marks bias, and whether bias of that kind drives discrimination, is not decided by any of these papers, and that is the finding.
Two uses are built on the one instrument. A population gauge treats a score as one reading among thousands and asks what the aggregate does across many encounters, and the 2015 reexamination makes a case for it. An individual verdict treats a score as a fact about one person, and the reliability and intervention figures all count against it, yet the same number sits behind both uses, and nothing in a score from the test, taken on its own, tells the person reading it which of those two uses the number is currently being asked to carry.
Common questions
Does the implicit association test measure bias?
It measures how much faster a person sorts words under one category pairing than under its reverse. Whether that difference marks bias is disputed. Pooled correlations with discriminatory behaviour run from about 0.15 to about 0.24, depending on which studies are counted.
Can the implicit association test predict who will discriminate?
Not for an individual, on the evidence reviewed here. Its own defenders caution that modest test-retest reliability and small effects make individual classification risky. They argue the effects can matter across whole populations or repeated encounters.
Does changing an implicit association test score change behaviour?
Not clearly. A 2019 pooled analysis of 492 studies found implicit measures could be changed, but weakly, and found that changes in them did not mediate changes in explicit measures or in behaviour.
Sources
Graves, P. (2010; revised paperback 2013). Consumer.ology: The Truth about Consumers and the Psychology of Shopping. London: Nicholas Brealey Publishing
Nisbett, R. E., & Wilson, T. D. (1977). 'Telling more than we can know: Verbal reports on mental processes.' Psychological Review, 84(3), 231-259. doi:10.1037/0033-295X.84.3.231
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). 'Measuring individual differences in implicit cognition: The Implicit Association Test.' Journal of Personality and Social Psychology, 74(6), 1464-1480. doi:10.1037/0022-3514.74.6.1464
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009). 'Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.' Journal of Personality and Social Psychology, 97(1), 17-41
Oswald, F. L., Mitchell, G., Blanton, H., Jaccard, J., & Tetlock, P. E. (2013). 'Predicting ethnic and racial discrimination: A meta-analysis of IAT criterion studies.' Journal of Personality and Social Psychology, 105(2), 171-192
Greenwald, A. G., Banaji, M. R., & Nosek, B. A. (2015). 'Statistically small effects of the Implicit Association Test can have societally large effects.' Journal of Personality and Social Psychology, 108(4), 553-561. doi:10.1037/pspa0000016
Gawronski, B., Morrison, M., Phills, C. E., & Galdi, S. (2017). 'Temporal stability of implicit and explicit measures: A longitudinal analysis.' Personality and Social Psychology Bulletin, 43(3), 300-312. doi:10.1177/0146167216684131
Forscher, P. S., Lai, C. K., Axt, J. R., Ebersole, C. R., Herman, M., Devine, P. G., & Nosek, B. A. (2019). 'A meta-analysis of procedures to change implicit measures.' Journal of Personality and Social Psychology, 117(3), 522-559. doi:10.1037/pspa0000160