What is confirmatory hypothesis testing?
Confirmatory hypothesis testing is a way of searching, not a belief. Four studies show where it misleads and where it is sound.
Answer to the question
What is confirmatory hypothesis testing?
Confirmatory hypothesis testing is checking an idea by looking for cases that fit it instead of cases that could contradict it. It describes a search procedure, not a belief. Research from Wason in 1960 to Klayman and Ha in 1987 shows it can mislead in puzzles, interviews and shopping, yet is often a sound strategy.
Key takeaways
- Confirmatory hypothesis testing is checking an idea by looking for cases that fit it; it describes a search procedure, which is narrower than confirmation bias.
- Wason (1960) gave 29 participants a number-rule task built to trap confirming-only tests: 6 reached the correct rule without first announcing a wrong one.
- Snyder and Swann (1978) found that people told an interviewee was an extravert or an introvert chose confirming questions, and those questions shaped the interviewee's behaviour to match.
- Hoch and Ha (1986) framed an advertisement as a tentative hypothesis that consumers test through product experience, with ambiguous experience giving it room to steer.
- Klayman and Ha (1987) argued that looking for positive cases is often a good strategy, and that its fit depends on the task.
Confirmatory hypothesis testing is the practice of checking an idea by collecting cases that fit it while leaving out the cases that could break it. It is a way of searching, not a belief, and it turns up in number puzzles, interviews and shopping alike. Most of the time it is a reasonable shortcut. The trouble starts when every test on offer can only return a yes.
The term is narrower than confirmation bias, which covers what people read, remember and believe. Confirmatory hypothesis testing follows one procedure: what a person does once they set out to test an idea, and what that procedure is able to return. The consumer side of it has a longer account in We Define First, Then We See.
A Number Puzzle That Rewards Wrong Guesses
The first controlled demonstration came from Peter Wason, in the Quarterly Journal of Experimental Psychology in 1960. His question was whether intelligent young adults test a hypothesis only by collecting instances that agree with it, which he called enumerative behaviour, or also by looking for instances that would rule it out. The task asked participants to discover a rule that generated sets of numbers, and it is now known as the 2-4-6 task.
Wason built the puzzle so that confirming-only reasoning was likely to fail. The correct rule was broader than several more obvious rules, and the supply of possible number sets was unlimited. A participant who tried only sets that fitted a personal guess would hear yes every time and still be wrong.
Of 29 participants, 6 reached the correct rule without first announcing a wrong one. Thirteen announced one wrong rule, nine announced two or more, and one reached no conclusion. Wason read the people with two or more wrong announcements as unable or unwilling to test their hypotheses, and he discussed what that might mean for scientific thinking.
The design matters more than the headcount. A yes from a test that would also have said yes to many other rules says little about which rule is true. Seen this way, confirmatory hypothesis testing is less a matter of stubbornness than of choosing a test whose answer was never in doubt.
A yes from a test that would also have said yes to many other rules says little about which rule is true.
The Interview That Writes Its Own Answers
Interviews gave the idea a social setting. In four investigations reported by Mark Snyder and William Swann in 1978, female undergraduates were told that a person they were about to interview was an extravert or an introvert, and they chose the questions they would put. The abstract reports that they planned to test the label by preferentially searching for behaviour that would confirm it.
The questions did more than collect evidence. The abstract also reports that the question-selection strategies shaped the interactions so that the people being interviewed displayed behaviour in line with the label they had been given. The authors add a theoretical account of the processes behind the confirmation-seeking search and its interpersonal consequences.
That result changes what confirmatory hypothesis testing can do. In the number puzzle the rule stays fixed while the participant guesses at it. In a conversation the questions belong to the situation, so a test can help produce its own evidence, and the record that results looks like clean confirmation.
Daniel Kahneman places the habit inside effortful thinking as well. In Thinking, Fast and Slow he describes the slower, effortful system as testing a hypothesis in the same way, by hunting for evidence that fits. On his account, slowing down does not remove confirmatory hypothesis testing. It changes who is doing it.
Advertising Supplies the Question
Shopping supplied the next setting. In the framing Stephen Hoch and Young-Won Ha proposed in 1986, consumers treat an advertisement as a tentative hypothesis about a product and then test it through using the product. The title of the paper names the condition that matters, the ambiguity of product experience: when what a person experiences could honestly be read more than one way, a hypothesis supplied beforehand has room to steer the reading.
The order of events is the point. An advertisement arrives before the trial, so it decides what the trial is a test of. For someone who buys, a pleasant first use after a persuasive advertisement is therefore weaker evidence than it feels, because an ambiguous experience can be read as agreement. For someone who sells, the test takes place in the customer's own head, before the product is touched. What changes once a shopper recognises the pitch is covered in What Spotting a Sales Pitch Changes.
Hoch and Ha's account of confirmatory hypothesis testing concerns how experience is read. It is not a forecast that any particular advertisement will lift sales, and a laboratory result of this kind should not be carried into a commercial plan without a trial of its own.
When Looking for Yes Is Sensible
The analysis that keeps the subject honest came in 1987, from Joshua Klayman and Young-Won Ha. Ha is the second author of the shopping study and of this one, so the two papers are one line of work on confirmatory hypothesis testing seen from two sides. They described a positive test strategy, which means examining cases one expects to have the property in question, and argued that under realistic conditions it is often a good way to find out whether a hypothesis is true.
The same strategy can also produce systematic errors and wasted effort. Their conclusion was that whether a strategy is appropriate has to be judged by how it interacts with the task in hand, so confirmatory hypothesis testing is a question of fit between method and task. The number puzzle was built to trap the strategy. A task in which the property is hard to find is a different case, and there asking about the cases you expect to match may be the efficient move. That is why a blanket instruction to doubt everything does not follow from the research on confirmatory hypothesis testing. The costs of teaching suspicion as a rule are taken up in Teaching People to Doubt Everything.
A workable check follows from the four studies. Before asking a question or running a trial, write down which answer would count against the idea, and ask for that answer first. If no reply the question could return would count against the idea, the exercise is a rehearsal of the idea and has not yet become a test of it, however thorough it looks.
Common questions
What is confirmatory hypothesis testing?
Confirmatory hypothesis testing is checking an idea by looking for cases that fit it rather than cases that could contradict it. It describes a search procedure, not a belief. Research from Wason in 1960 to Klayman and Ha in 1987 shows it can mislead in puzzles, interviews and shopping, yet is often a sound strategy.
Is confirmatory hypothesis testing the same as confirmation bias?
No. Confirmation bias is the broader family, covering what people notice, remember and believe. Confirmatory hypothesis testing names one procedure within it: the way a person tests an idea once they hold it. Klayman and Ha argued that much of what gets labelled bias is a positive test strategy.
Who first studied confirmatory hypothesis testing?
Peter Wason published the first controlled demonstration in 1960, using a number-rule task. Snyder and Swann extended it to interviews in 1978, and Hoch and Ha applied it to advertising and product experience in 1986.
Is looking for confirming evidence always a mistake?
No. Klayman and Ha argued in 1987 that a positive test strategy is often an effective way to learn whether a hypothesis is true. It can also cause systematic errors, so whether it fits depends on the task. It is a problem when the test cannot return a no.
Sources
Kahneman, D. (2011). Thinking, Fast and Slow. Seed book; vault hub Books/Thinking Fast and Slow.
Wason, P. C. (1960). “On the failure to eliminate hypotheses in a conceptual task.” Quarterly Journal of Experimental Psychology, 12(3), 129-140. doi:10.1080/17470216008416717 https://doi.org/10.1080/17470216008416717
Snyder, M., & Swann, W. B. (1978). “Hypothesis-testing processes in social interaction.” Journal of Personality and Social Psychology, 36(11), 1202-1212. doi:10.1037/0022-3514.36.11.1202 https://doi.org/10.1037/0022-3514.36.11.1202
Hoch, S. J., & Ha, Y.-W. (1986). “Consumer learning: Advertising and the ambiguity of product experience.” Journal of Consumer Research, 13(2), 221-233. doi:10.1086/209062 https://doi.org/10.1086/209062
Klayman, J., & Ha, Y.-W. (1987). “Confirmation, disconfirmation, and information in hypothesis testing.” Psychological Review, 94(2), 211-228. doi:10.1037/0033-295X.94.2.211 https://doi.org/10.1037/0033-295X.94.2.211