Does the implicit association test work?
The implicit association test returns a difference in milliseconds. Comparisons of its reliability and pooled analyses of its link to behaviour give a mixed answer that depends on the claim.
Answer to the question
Does the implicit association test work?
Partly, and the answer depends on the claim. The implicit association test is consistent within a session, with an average alpha of .88 in a seven-measure comparison, but its test-retest correlation averaged .45. Pooled analyses link it to behaviour at widely varying strengths, and researchers disagree on how much the averages show.
Key takeaways
- In a comparison of seven indirect measures on 23,413 people, Bar-Anan and Nosek (2014) found the implicit association test led on internal consistency (.88) and averaged .45 on test-retest.
- Gawronski and colleagues (2017) found implicit measures less stable over time than matched explicit measures, at weighted averages of .54 and .75.
- Greenwald and colleagues (2009) pooled 122 reports and found an average correlation of .274 between the test and outcomes, against .361 for self-report.
- Oswald and colleagues (2013) reported weak prediction of discrimination outcomes, and Greenwald, Banaji and Nosek (2015) traced much of the gap to which correlations were counted.
- Kurdi and colleagues (2019) pooled 217 reports: well-matched designs averaged .37 and poorly matched ones .02.
Seven indirect attitude measures were put through the same battery on 23,413 people, and the implicit association test finished first on internal consistency, with an average alpha of .88. In the same comparison its test-retest correlation averaged .45, fifth among the seven. Yoav Bar-Anan and Brian Nosek ran the comparison, which covered attitudes towards politics, race and self-esteem, and reported both figures in one table. All seven measures correlated with one another and with direct, self-reported measures, and across the full set of criteria the implicit association test and its brief version performed best overall. A measure can be tight within a sitting and looser across weeks, and it can track some outcomes better than others.
Robert Sapolsky's Behave presents the test as a sorting task. Category images and words that are pleasant or unpleasant share two response buttons, and a pairing that runs against a person's associations costs a few milliseconds. In his account the accumulated delay records an automatic bias, which he separates from what a person consciously believes. Anthony Greenwald and two colleagues introduced the implicit association test in 1998, with early demonstrations that sorted flower names and insect names against pleasant and unpleasant words.
Stability of the score
The output of the implicit association test is the difference between a person's average response time under one pairing and under the reverse pairing. Everything beyond that difference is interpretation, and the implicit association test is no exception. Reading it as a hidden attitude, or as a bias the person does not know they hold, is a claim about what the number means, and a claim of that kind can be checked in only two ways: by asking whether the number is stable, and by asking whether it goes on to predict something.
The seven-measure comparison, published in Behavior Research Methods in 2014, draws the same distinction. It judged the measures on their validity as attitude measures, and its authors state that the results do not settle whether the attitudes measured are implicit.
On internal consistency, the extent to which the trials within one session agree with each other, the implicit association test averaged .88 in Bar-Anan and Nosek's comparison and led the field. On test-retest correlation it averaged .45, behind the brief version of the test at .63. The authors note that the retest estimates rest on about 100 participants per topic and were not their primary criterion.
A longitudinal paper in 2017 followed self-concept, racial attitudes and political attitudes in two studies. Bertram Gawronski and colleagues found implicit measures less stable across time than matched explicit measures, with weighted average correlations of .54 and .75, while their internal consistency was comparable.
Self-report versus the score
Across topics, the implicit association test correlated with self-reported attitudes at .35 on average in Bar-Anan and Nosek's seven-measure comparison. The figure moved a good deal by topic. For political attitudes, where agreement was closest of the three topics, the correlations were .64 with the self-reported preference measure and .60 with the self-reported thermometer measure. For race attitudes, where agreement was moderate, both were .32. For self-esteem, where agreement was weakest, they were .11 and .13.
An overall figure for how far the test agrees with what people say therefore depends on which attitude is under study. How steady direct questioning is as an instrument is a separate matter, taken up in Why Explaining Your Choice Spoils It.
A 2009 meta-analysis by Greenwald and colleagues, covering 122 research reports, 184 samples and 14,900 participants, found a similarly modest link. Across all the samples, the correlation between the implicit association test and self-report averaged about .21.
What the score predicts
The same 2009 meta-analysis also asked what the test predicts. Implicit association test measures predicted behavioural, judgement and physiological outcomes with an average correlation of .274, against .361 for parallel self-report measures. The self-report figures varied more from study to study and were weaker on socially sensitive topics. In the 32 samples with interracial criterion measures the test did better than self-report, at .236 against .118. Each kind of measure added predictive value beyond the other, and the closer the test and self-report agreed, the better each predicted.
A 2013 meta-analysis of studies on ethnic and racial discrimination reached a different reading. Its authors, Frederick L. Oswald and colleagues, report that the tests were weak predictors across interpersonal behaviour, person perception, policy preference, microbehaviour and response time, with brain activity the exception, and that they did no better than simple explicit measures.
The two analyses were revisited in 2015 by Greenwald, Mahzarin R. Banaji and Nosek. Their paper traces most of the difference to which correlations each analysis counted, and argues that effects statistically small in one encounter can still be socially large when they apply to many people or recur for the same person. The same paper states that modest test-retest reliability and small effects make the test risky for classifying individuals. The criterion literature is weighed further in Reading the Implicit Association Test.
The 2019 pooling of reports
The most recent pooling, in 2019, covers 217 research reports with 36,071 participants and concerns the test's relation to intergroup behaviour. Its lead author is Benedek Kurdi. Both the implicit and the explicit measures related significantly to the criteria. In a structural equation model each contributed unique variance, with coefficients of .14 for the implicit measures and .11 for the explicit ones.
The correlations varied widely between studies. Basic study features and conceptual variables did not account for the spread, and methodological features did. In Kurdi and colleagues' analysis, studies with a declared focus on the implicit-criterion correlation, a standard version of the test, high-polarity attributes, relative behaviour measures and high correspondence between test and criterion had a mean correlation of .37 across 13 studies. Studies low on those features averaged .02 across six. The authors also report that most of the studies were badly underpowered and that analyses often ignored measurement error.
Kurdi and colleagues released online tools for calculating statistical power and measurement consistency.
Three questions follow from these papers for anyone handed a single score.
1. Is the score being used about one person or about many? The 2015 paper by Greenwald, Banaji and Nosek calls the test risky for classifying individuals.
2. Was the test taken once? Scores from one sitting agree closely with each other, and scores taken weeks apart agree less.
3. How closely does the test match the behaviour in question? In the 2019 pooling, studies in which the test and the behaviour measure were closely matched, among other design features, showed a mean correlation many times stronger than studies low on those features, which averaged close to zero.
Common questions
Does the implicit association test work?
Partly, and it depends on the claim. It is consistent within a session, with an average alpha of .88 in a seven-measure comparison, and less stable across weeks, with a test-retest correlation of .45. Its link to behaviour varies widely with study design.
Is the implicit association test reliable?
Within a session, yes: internal consistency averaged .88 in a comparison of seven indirect measures. Across weeks it is lower, with test-retest correlations averaging .45 in that comparison and weighted averages of .54 in a later longitudinal study, against .75 for explicit measures.
Does the implicit association test predict behaviour?
Weakly on average, and more strongly under some designs. A 2009 meta-analysis found an average correlation of .274 with outcomes, and a 2019 meta-analysis found .37 for well-matched studies against .02 for poorly matched ones.
Do researchers agree on what the numbers show?
Not fully. A 2013 meta-analysis reported weak prediction of discrimination outcomes, and a 2015 paper attributed much of the gap with an earlier analysis to which correlations were counted and argued that small effects can matter across many people.
Sources
Sapolsky, R. M. (2017). Behave: The Biology of Humans at Our Best and Worst. Seed book; vault hub Books/Behave.
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). “Measuring individual differences in implicit cognition: The Implicit Association Test.” Journal of Personality and Social Psychology, 74(6), 1464-1480. doi:10.1037/0022-3514.74.6.1464 https://doi.org/10.1037/0022-3514.74.6.1464
Bar-Anan, Y., & Nosek, B. A. (2014). “A comparative investigation of seven indirect attitude measures.” Behavior Research Methods, 46(3), 668-688. doi:10.3758/s13428-013-0410-6 https://doi.org/10.3758/s13428-013-0410-6
Gawronski, B., Morrison, M., Phills, C. E., & Galdi, S. (2017). “Temporal stability of implicit and explicit measures: A longitudinal analysis.” Personality and Social Psychology Bulletin, 43(3), 300-312. doi:10.1177/0146167216684131 https://doi.org/10.1177/0146167216684131
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009). “Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.” Journal of Personality and Social Psychology, 97(1), 17-41. doi:10.1037/a0015575 https://doi.org/10.1037/a0015575
Oswald, F. L., Mitchell, G., Blanton, H., Jaccard, J., & Tetlock, P. E. (2013). “Predicting ethnic and racial discrimination: A meta-analysis of IAT criterion studies.” Journal of Personality and Social Psychology, 105(2), 171-192. doi:10.1037/a0032734 https://doi.org/10.1037/a0032734
Greenwald, A. G., Banaji, M. R., & Nosek, B. A. (2015). “Statistically small effects of the Implicit Association Test can have societally large effects.” Journal of Personality and Social Psychology, 108(4), 553-561. doi:10.1037/pspa0000016 https://doi.org/10.1037/pspa0000016
Kurdi, B., Seitchik, A. E., Axt, J. R., Carroll, T. J., Karapetyan, A., Kaushik, N., Tomezsko, D., Greenwald, A. G., & Banaji, M. R. (2019). “Relationship between the Implicit Association Test and intergroup behavior: A meta-analysis.” American Psychologist, 74(5), 569-586. doi:10.1037/amp0000364 https://doi.org/10.1037/amp0000364