System Noise

What it is

Judgement error has two components. Bias is the average error, the same direction every time. Noise is the scatter around that average. System noise is the second, measured among people rather than within one person, and the vault holds its three components separately: level noise, the durable differences in how severe or generous individual judges are; pattern noise, the differences in how judges rank cases against one another; and occasion noise, the transient share of pattern noise, where the same judge differs from himself.

The definition is inseparable from the design that detects it. Many equally qualified judges, one identical case, one scale, with the judges working independently and not told that variability is what is being measured. That design is the noise audit.

Why that design matters more than any single figure. Bias cannot be measured without knowing the truth. System noise can, today, inside any organisation whose people judge written material. The book's image for this is the reversed target: turn the targets round so the bull's-eye cannot be seen, and the biased team becomes indistinguishable from the accurate one while the scattered teams stay visibly scattered. All you have to do to measure noise is look at the back of the target.

Which answers the standing objection that the spread might be real signal. Where the case is held identical, nothing that moves can be merit.

And the arithmetic that makes the measurement actionable. Overall error, measured as the average of the squared errors, is the sum of bias squared and noise squared. The two weigh exactly the same. That is an identity inherited from Gauss, with no sample size in it and nothing to replicate. Two scope limits travel with it: statistical bias here means average error and nothing about discrimination, and the identity covers judgements aiming at a true value, which excludes insurance pricing.

The quantity is a property of the question rather than of the people, and this is the most practical thing on the page. In the measurement this publication trusts most, forty-nine pathologists reading the same sixty breast specimens agreed with one another on invasive cancer 89 per cent of the time and on atypia 43 per cent. Same readers, same slides, same design. So a noise figure quoted without the category it was measured on carries almost no information.

In effect

In medicine: the cleanest cases

Sara L. Jackson and colleagues had forty-nine pathologists interpret sixty breast specimens twice, at least nine months apart, producing 2,940 interpretations: self-agreement on atypia 53 per cent, agreement between different pathologists 43 per cent. Joann G. Elmore and colleagues had 187 pathologists interpret 240 skin biopsy cases and record that an average of ten different diagnostic terms was applied to each case. Carole A. Ridge and colleagues found six thoracic radiologists averaging twenty-one years in the specialty matching their own earlier classification of the same lung nodule in 77 per cent of cases, one of them in 45 per cent. None of these measured accuracy or any outcome for a patient, and none is clinical advice.

In regulation: the design run on consumers

Alan Boyd and colleagues had 286 hospital inspectors assign a domain and a four-point rating to ten identical vignettes drawn from real inspection reports. Agreement was almost perfect on one vignette and barely better than chance on another: on one, 93 per cent chose the same rating, and on another half chose one rating and the rest another. Groups of inspectors were more reliable than individuals. It was a vignette survey run during the piloting of the inspection model in 2013 and 2014, and its authors call for replication, so it is not a measurement of current ratings.

In commerce: the figure to handle carefully

The most quoted evidence in this field is an insurance audit in which the median difference between two randomly paired underwriters was 55 per cent of their average, and 43 per cent for claims adjusters, against the 10 per cent that 828 senior executives gave as both the median and the most common prediction. It was published in a practitioner magazine without peer review, reporting proprietary consulting data on a client named only as a global financial services firm, with no participant counts and no published method. It is uncheckable rather than wrong. Nobody has tried and failed to reproduce it, because nobody can try, and the medical literature carries the size claim better.

What it does not say

It does not say that reducing scatter is automatically a gain. Miroslaw Szreder's published objection, in a peer-reviewed statistics journal, is that shrinking noise around a biased centre makes the true value less likely to be found, because the tightened distribution is centred on the wrong spot. He calls the book excellent and praises its clarity, and both halves of that travel together.

It does not say the decomposition into stable and occasion components is established. Szreder argues it is intuitively meaningful rather than analytically justified, and the authors half concede it, writing of their own sentencing analysis that they effectively ignored occasion noise and that the assumption is certainly optimistic.

It does not say the remedies are measured. No effect size is attached to any of the six decision-hygiene principles anywhere in the book, and the authors say so.

And it does not say that agreement means accuracy. Two readers agreeing does not make either right.

---