Two Readings of the Same Case
The subject is variance, not bias, and the strongest measurements of it are medical rather than financial. The insurance figure everyone quotes was published in a practitioner magazine with proprietary data and no method a reader can inspect.
Answer to the question
How large is noise in judgement between two experts?
On hard calls, very large. The same pathologist re-reading the same breast biopsy slide nine months later matched his own earlier call on atypia fifty-three per cent of the time, against forty-three per cent between two different pathologists. On straightforward calls the same study found agreement near ninety per cent.
Key takeaways
- The same pathologist, re-reading the same breast biopsy slide at least nine months later, agreed with his own earlier call on atypia fifty-three per cent of the time, against forty-three per cent between different pathologists.
- The famous fifty-five per cent insurance figure was published in a practitioner magazine in 2016 with proprietary data, an unnamed client and no participant counts, so no method is publicly available.
- Among 286 inspectors rating ten identical vignettes, agreement ran from almost perfect on one case to barely better than chance on another, and groups were more reliable than individuals.
- The book's own review found the same-person-different-day component the smallest of the three it decomposes, so the gap between two readers is wider than the gap within one.
On the difficult call of atypia, a pathologist handed the same breast biopsy slide at least nine months after first reading it, filed it under the same heading 53 per cent of the time. Give that slide to a different pathologist and the agreement was 43 per cent. Those figures, from Sara Jackson and colleagues in 2017, come from a design measurements of noise in judgement rarely manage: the same case put twice to the same expert and once to another. A second look by the same expert is worth having. A second opinion from a different expert is worth more. On the calls professionals find easy, neither is worth much, because there is almost nothing left to resolve.
So the move comes first. Ask which kind of judgement is being made, because the agreement rate is a property of the question: invasive cancer was called the same way nine times in ten, atypia about half the time. Ask for a second reader before asking for a second look. And where the thing arriving is a number rather than a diagnosis, collect more than one. Two statisticians reviewing the book that made noise in judgement famous put that point in nine words:
> "If you are a customer, your optimal strategy is to get multiple quotes."
Everything below is the justification.
What Sara Jackson's pathologists did twice
Jackson and colleagues recruited pathologists across eight American states and had them interpret 60 breast specimens, one glass slide per case, on two occasions separated by at least nine months, and 49 of them read the same slides in both phases. The separation matters: re-presenting a case within a session measures memory, as the authors of Noise say themselves.
Same pathologist, nine months apart: invasive breast cancer 92 per cent, ductal carcinoma in situ 84, atypia 53, benign without atypia 84. Different pathologists on the same material: 89, 79, 43 and 77. Their conclusion is the hinge, that self-agreement on atypia was "similar to observed rates of agreement for atypia between different pathologists".
Their conclusion is the hinge, that self-agreement on atypia was "similar to observed rates of agreement for atypia between different pathologists".
So the value of your own earlier mind is 10 percentage points on the hardest call in the set and 3 to 7 points elsewhere: 3 on invasive cancer, 5 on ductal carcinoma in situ, 7 on benign without atypia. The figure that travels, 10 points, is the atypia figure, not the general one.
Where else the same reading was repeated
A 2017 melanoma study led by Joann Elmore, corrected by the journal eight weeks later in a notice we did not read, had pathologists re-read the same slides at least eight months on and found agreement of about 35 per cent on moderately dysplastic nevi. Carole Ridge and colleagues, studying six thoracic radiologists averaging 21 years in the specialty, found readers matching their own earlier classification of the same lung nodule in 77 per cent of cases, and one of them in 45.
The between-reader arm of the same breast programme is larger and points the same way: 115 pathologists agreed with a three-expert reference standard on 75.3 per cent of 6,900 interpretations, and on atypia on 48 per cent; the three experts, reading independently before conferring, were unanimous on 75 per cent of their cases.
What the error equation settles
This is where noise in judgement stops being a version of bias. Overall error, measured as the average of the squared errors, is the sum of bias squared and noise squared, where bias is the average error and noise is the scatter around it. The two weigh the same. That is arithmetic, inherited from Gauss, with nothing in it to replicate.
Two scope limits travel with it. Statistical bias here means average error and nothing about discrimination. And the identity covers judgements aiming at a true value, which, as Kahneman, Sibony and Sunstein note, excludes the insurance pricing their own flagship chapter is built on.
The second idea is better still, and it is a definition. Turn the targets round so the bull's-eye cannot be seen, and the biased team becomes indistinguishable from the accurate one while the scattered teams stay visibly scattered.
> "All we have to do to measure noise is look at the back of the target."
Which answers the objection that the spread might be real signal, the one [How Far the Halo Effect Spreads](https://thewhyweproject.com/articles/how-far-the-halo-effect-spreads) fights out in another form. Where the case is held identical, nothing that moves can be merit.
What the inspectors and valuers found
The design the subject calls for has been run on consumers, best by Alan Boyd and colleagues in 2016, who had 286 inspectors each rate ten vignettes of hospital service drawn from real inspection reports, on a four-point scale. Agreement held or collapsed depending on the case. On one vignette 93 per cent chose the same rating; on another, half chose one rating and the rest another.
Groups of inspectors were more reliable than individuals, which is what the research on [how much weight people put on a second opinion](https://thewhyweproject.com/articles/how-much-advice-people-actually-take) would predict. This was a vignette survey run while the inspection model was still being piloted, and its authors call for replication.
Money behaves no differently.
Alastair Adair and colleagues in 1996 asked qualified valuers to value the same commercial properties on the same stated assumptions. More than 80 per cent of valuations of rack-rented interests came within 20 per cent of the mean, so nearly one in five did not, against a profession claiming 5 to 10 per cent.
Insurance is the odd one out.
The book's most quoted evidence is an insurance audit. Executives at a large insurer were asked how far apart two randomly paired qualified underwriters would be on the same case, as a share of their average; 828 of them gave 10 per cent as both the median and the most common answer. The measured median difference was 55 per cent for underwriters and 43 for claims adjusters, about five times what anybody expected. And median means half the pairs were further apart.
The audit was published in 2016 in Harvard Business Review by Daniel Kahneman with Andrew Rosenfield, Linnea Gandhi and Tom Blaser, two of whom the publisher's page describes as principals of the consulting firm that ran the engagement, and which classifies the item as a magazine article. It carries no peer review and no digital object identifier, the client is named only as a global financial services firm, the data are proprietary, and no headcount of underwriters or adjusters appears anywhere. Its body stayed behind a paywall we could not pass, and the book, the public source, supplies no method either.
Sibony and Sunstein are not its authors, correcting our earlier record.
This is a provenance limit, not a replication failure. Nobody has tried and failed to reproduce 55 per cent. Nobody can try. And the judgement the book made famous, variability between underwriters inside one firm pricing the same risk, is the one measurement of noise in judgement we could not find made independently, across two search passes with no keyword web search available to either.
Where Szreder parts from the authors
The published criticism lands on the arithmetic's consequences. Miroslaw Szreder, writing in Przeglad Statystyczny in 2022, calls it an excellent book, then concedes the identity and attacks what follows from it. Shrink the scatter while leaving bias in place and overall error falls exactly as advertised, but the chance that any single judgement lands near the true value falls too, because the tightened distribution is centred on the wrong spot.
"A reduction of noise, if bias is present in judgments, makes it less likely that the true value is going to be discovered," he writes. His close is the best one-line case anybody has made against the programme: if everyone says the same thing and everyone turns out to be wrong, more dispersed views would have hinted that the truth lay elsewhere.
He adds a second objection, to the structure. The equation separating the stable half of a judge's idiosyncrasy from the part varying by occasion is, he says, intuitively meaningful but not analytically justified. The authors half concede it, writing of their own sentencing analysis that they "effectively ignored occasion noise" and that the assumption "is certainly optimistic".
Gaurav Sood and Andrew Gelman, reviewing the book in CHANCE, make a sharper charge: that a book warning against reading too much into noisy evidence cites very small studies without qualification. Its software-developer figure, they report, rests on seven developers.
Where we part from the authors
Our own objection is to the remedies. The six decision-hygiene principles carry no measured effect size, and the authors say as much, calling the likely benefit of each strategy an open question research could address. That is a reasonable protocol, not an intervention with evidence behind it.
The theorem holds and the reversed-target argument holds, because both are definitions. Szreder is right that reducing scatter around a biased centre is not automatically a gain, and right that the stable-against-occasion split is the weakest joint in the chain. On the size of noise in judgement the medical literature settles it in the authors' favour, more convincingly than their insurance audit.
The honest ceiling comes from the book itself. In every case its authors could measure, the same-person-different-day component was the smallest of the three. Michael Kahana's team, as the book reports, ran 79 subjects through 23 sessions of word recall on separate days, and found alertness, sleep, time of day, practice and fatigue together explained 11 per cent of one person's variation. The remainder, they said, "should give pause to anyone who thinks occasion noise can be eliminated".
So the checks on noise in judgement are checks on the judgement, not on whoever is making it:
- Whether the call is one professionals find easy or one they split on. The 92 per cent and the 53 per cent came from the same 49 pathologists.
- Whether a second opinion is a different reader or the same reader again. On the hardest call that difference measured 10 percentage points, and 3 to 7 points on the easier ones.
- Whether a quote comes from inside one organisation, where the measurement applies, or from competing firms, where the spread is a market instead.
- Whether the striking figure has a method a reader can inspect. The famous one does not; the medical ones do.
Averaging is the remedy the arithmetic recommends and the inspectors' groups demonstrate, a reversal of how averages usually appear here, where [a star average failing to track quality](https://thewhyweproject.com/articles/why-we-trust-online-ratings-anyway) is the problem. Two readings of the same case will differ, and the gap between two readers is wider still, which is why the second opinion worth paying for is the one that arrives from somebody else.
Common questions
How large is noise in judgement between two experts?
On hard calls, very large. Jackson and colleagues found different pathologists agreeing on atypia 43 per cent of the time and the same pathologist agreeing with his own earlier reading 53 per cent of the time, nine months apart. On straightforward calls the same study found agreement near 90 per cent. The spread is a property of the category being judged.
Is noise in judgement the same thing as bias?
No, and the difference is arithmetic. Bias is the average error and noise is the scatter around that average, and in the error equation they carry identical weight. Bias cannot be measured without knowing the right answer; scatter can be measured by comparing judgements of an identical case, with the true value unknown throughout.
Can a person reduce noise in judgement by thinking twice?
A little. The gain from judging the same question twice, after a delay, is real but modest beside the gain from a genuinely independent second opinion, and the book's own review found the same-person component the smallest of the three it decomposes. Kahana's recall study suggests the day-to-day variation left over resists explanation.
Which noise in judgement figures can a reader check?
The medical ones. Jackson's breast-biopsy agreement rates and the melanoma and radiology repeat-read studies are peer reviewed, with published methods and denominators. The 55 per cent insurance figure is not: it appeared in 2016 in a practitioner magazine without peer review, reporting proprietary consulting data on an unnamed financial services client, with no participant counts and no published method.
Sources
Jackson, S. L., Frederick, P. D., Pepe, M. S., Nelson, H. D., Weaver, D. L., Allison, K. H., Carney, P. A., Geller, B. M., Tosteson, A. N. A., Onega, T., & Elmore, J. G. (2017). 'Diagnostic Reproducibility: What Happens When the Same Pathologist Interprets the Same Breast Biopsy Specimen at Two Points in Time?' Annals of Surgical Oncology, 24(5), 1234-1241. doi:10.1245/s10434-016-5695-0
Elmore, J. G., Barnhill, R. L., Elder, D. E., Longton, G. M., Pepe, M. S., Reisch, L. M., Carney, P. A., Titus, L. J., Nelson, H. D., Onega, T., Tosteson, A. N. A., Weinstock, M. A., Knezevich, S. R., & Piepkorn, M. W. (2017). 'Pathologists' diagnosis of invasive melanoma and melanocytic proliferations: observer accuracy and reproducibility study.' BMJ, 357, j2813. doi:10.1136/bmj.j2813
Ridge, C. A., Yildirim, A., Boiselle, P. M., Franquet, T., Schaefer-Prokop, C. M., Tack, D., Gevenois, P. A., & Bankier, A. A. (2016). 'Differentiating between Subsolid and Solid Pulmonary Nodules at CT: Inter- and Intraobserver Agreement between Experienced Thoracic Radiologists.' Radiology, 278(3), 888-896. doi:10.1148/radiol.2015150714
Kahneman, D., Rosenfield, A. M., Gandhi, L., & Blaser, T. (2016). 'Noise: How to Overcome the High, Hidden Cost of Inconsistent Decision Making.' Harvard Business Review, 94(10), 38-46
Kahneman, D., Sibony, O., & Sunstein, C. R. (2021). Noise: A Flaw in Human Judgment. London: William Collins
Boyd, A., Addicott, R., Robertson, R., Ross, S., & Walshe, K. (2016). 'Are inspectors' assessments reliable? Ratings of NHS acute hospital trust services in England.' Journal of Health Services Research & Policy, 22(1), 28-36. doi:10.1177/1355819616669736
Adair, A., Hutchison, N., MacGregor, B., McGreal, S., & Nanthakumaran, N. (1996). 'An analysis of valuation variation in the UK commercial property market: Hager and Lord revisited.' Journal of Property Valuation and Investment, 14(5), 34-47. doi:10.1108/14635789610154271
Szreder, M. (2022). 'Noise and bias, some controversies raised by the book "Noise: A Flaw in Human Judgment".' Przeglad Statystyczny, 69(1), 39-49. doi:10.5604/01.3001.0015.8792
Sood, G., & Gelman, A. (2024). 'Review of Noise: A Flaw in Human Judgment.' CHANCE, 37(3), 70-72. doi:10.1080/09332480.2024.2416879
Gilhooly, K. J., & Sleeman, D. H. (2022). 'To differ is human: A reflective commentary on "Noise. A Flaw in Human Judgment".' Applied Cognitive Psychology, 36(3), 724-730. doi:10.1002/acp.3941