Decision Hygiene

What it is

The six principles, as the authors state them, are the operational payload of the book.

1. The goal of judgement is accuracy, not individual expression. Stable pattern noise follows from judgement personalities, so individuality belongs in choosing goals, framing problems and generating options rather than in evaluating them.
2. Think statistically, and take the outside view of the case. Treat the case as a member of a reference class, which entails regressive, moderate predictions and calibrated confidence.
3. Structure judgements into several independent tasks. Divide and conquer, made necessary by excessive coherence, with independence protected by assigning assessments to different teams.
4. Resist premature intuitions. Intuition need not be banned but should be informed, disciplined and delayed, with information sequenced so that judges are not given what they do not need.
5. Obtain independent judgements from multiple judges, then consider aggregating them. Averaging independent judgements is guaranteed to reduce system noise, though not bias, and group discussion often increases noise.
6. Favour relative judgements and relative scales, because our ability to make pairwise comparisons is much better than our ability to categorise objects on a scale.

Why this publication grades it `mixed`, and the reason must travel with the list. The authors state at p. 224 that it would be valuable to quantify the likely benefits of each strategy and that, because noise has attracted little attention, these remain open questions. On the decision-observer checklist specifically, at p. 243: "we are not aware of any systematic exploration of the effects of this approach." Individual components have independent support, aggregation is a theorem and structured interviews have a real literature, but the package is well reasoned and largely untested as a package.

And the published criticism lands on the direction of the remedy rather than on its size. Miroslaw Szreder, writing in a peer-reviewed statistics journal in 2022, calls the book excellent and praises its clarity, concedes the error identity, and then argues: "A reduction of noise, if bias is present in judgments, makes it less likely that the true value is going to be discovered." Shrink the scatter while leaving bias in place and overall error falls exactly as advertised, while the chance that any single judgement lands near the true value falls too, because the tightened distribution has been narrowed around a displaced centre. His close is the best one-line case anybody has made against the programme: if everyone says the same thing and everyone turns out to be wrong, more dispersed views would have hinted that the truth lay elsewhere.

His second objection is structural. The identity splitting pattern noise into stable and occasion components is, on his reading, intuitively meaningful but not analytically derived. The authors half concede it, writing of their own sentencing analysis that they effectively ignored occasion noise and that the assumption is certainly optimistic.

In effect

In research: a second published criticism, aimed at the book's own standard

Gaurav Sood and Andrew Gelman, reviewing the book in CHANCE, restate the insurance figure with the median caveat intact and then charge that a book warning against reading too much into noisy evidence cites very small studies without qualification, its software-developer figure resting on seven developers. If individual judgements are noisy, they argue, general insights should not be drawn from seven or eight people in one place. Ken Gilhooly and Derek Sleeman, in Applied Cognitive Psychology, credit the book with raising awareness, supplying a taxonomy and outlining remedies, and what is absent from that list is any claim that the remedies were measured.

In regulation: aggregation demonstrated rather than asserted

Alan Boyd and colleagues put ten identical hospital-service vignettes to 286 inspectors and found groups of inspectors more reliable than individuals. That is principle five with a measurement attached, in a real regulatory system, and it is the strongest independent support any of the six has in this publication's files. It was a vignette survey run during a pilot in 2013 and 2014, and its authors call for replication.

In everyday life: the handwashing problem

The authors press the analogy's uncomfortable half themselves: handwashing is tedious, its benefits are invisible, and compliance is hard to enforce even among clinicians who know it matters. Noise is an invisible enemy, and preventing the assault of an invisible enemy can yield only an invisible victory. Which is a prediction about adoption, and a good one: a procedure whose success is unobservable is a procedure people stop following.

What it does not say

It does not say the six principles do not work. It says nobody has measured them as a package, which is a statement about the evidence rather than about the procedures.

It does not say reducing noise is pointless. The error identity guarantees that reducing scatter reduces overall error. The criticism is that this is not the same as making any individual judgement more likely to be right, and that distinction is Szreder's.

It does not claim the authors oversold it. They state the absence of effect sizes themselves, and their own closing concessions bound the prescription: noise may be too costly to reduce, some may be a necessary side effect of due process, some may even be desirable, and the practices risk bureaucratising decisions and demoralising professionals.

And the two published criticisms here are arguments about statistical method, between researchers who have each printed their reasoning.

---