Occasion Noise

What it is

If the first lottery decides which judge you get, occasion noise is the product of a second one: the moment the judgement is made, the judge's mood, the sequence of cases fresh in mind, and countless other features of the occasion.

Its size is the useful thing about it, and it is deflationary. In every case the authors of Noise could measure, occasion noise was the smallest of the three components, smaller than the stable differences between individuals. Their own rule is the one to remember: you are more similar to yourself yesterday than you are to another person today.

The design decides whether a figure means anything, and most of the published literature fails the test. Much of it asked experts to judge the same case twice within the same session, where, as the book puts it, they tended to agree with themselves. That is measuring memory, not stability. The studies this publication trusts separated the sittings by months.

What the properly separated studies found. Sara L. Jackson and colleagues separated the sittings by at least nine months. Forty-nine pathologists, sixty breast specimens, 2,940 interpretations. Same pathologist nine months apart: invasive breast cancer 92 per cent, ductal carcinoma in situ 84, atypia 53, benign without atypia 84. Different pathologists on the same material: 89, 79, 43 and 77. Their conclusion is the hinge: self-agreement on atypia was "similar to observed rates of agreement for atypia between different pathologists".

Ten percentage points on the hard call. That is the measured value of your own earlier mind, and it is the sharpest available answer to whether a second opinion should come from the same person or somebody else.

And the residue that resists explanation. Michael Kahana's team, as the book reports, ran seventy-nine subjects through twenty-three sessions of word recall on separate days and found alertness, sleep, time of day, practice and fatigue together explaining eleven per cent of one person's variation. The remainder, they said, should give pause to anyone who thinks occasion noise can be eliminated.

In effect

In medicine: two companions with their sittings properly separated

Joann G. Elmore and colleagues had pathologists re-read the same melanoma slides at least eight months later and found agreement of about 35 per cent on moderately dysplastic nevi. A correction was published eight weeks after that paper and this publication has not read it. Carole A. Ridge and colleagues found six thoracic radiologists averaging twenty-one years in the specialty matching their own earlier classification of the same lung nodule in 77 per cent of cases, one of them in 45, with at least three weeks between sittings. That paper does not use the Landis and Koch bands.

In everyday life: the crowd within

Asking yourself the same quantity question twice, unwarned, and averaging the two answers gains about one tenth as much as a second opinion from somebody else; with three weeks between the guesses the gain rises to about one third. Deliberately assuming the first estimate was wrong, listing reasons why, then re-estimating gets about half the value of a second opinion from two consecutive estimates. The practical rule in the book is to get other people's judgements where you can, and to judge twice and average where you cannot.

In commerce: the quote you already have

Where the output is a number rather than a category, the arithmetic recommends collecting more than one. A second look at your own earlier estimate is worth something. A second estimate from a colleague is worth more, and the measured difference between the two is small enough that both are worth having.

What it does not say

It does not say that mood and weather move expert judgements by much. This publication's grade on the concept is mixed precisely because the chapter that popularises it is where the weakest studies in the book live, including mood inductions, weather effects and several results this publication quarantines on independent grounds.

The concept survives the removal of those illustrations entirely, because the authors' own analysis puts occasion noise smallest of the three, which deflates their own chapter before anybody else has to.

It does not say the within-person gap is negligible. Ten percentage points on a diagnostic call is not negligible.

And it does not say thinking twice is a substitute for asking somebody else. On the figures above it is worth roughly a tenth as much, rising to a third with a long delay.

---