The Evidence Ladder

What it is

The three rungs, as they set them out at pages 184 to 185. Prescriptive means large randomised trials, placebo controlled and blinded, with real-world outcomes, replicated by other people. Signal means careful laboratory work that establishes feasibility, mechanism and proof of principle, and no more. Reasonable hypothesis means an anecdote, a few studies, and sound logic.

They add that evidence is not the only thing a reader should weigh, and list cost, time, size of the likely gain and side effects alongside it. They describe the scale as their impressions, which is the right label and is recorded here as printed.

In effect

The grading, as they gave it in the summary at pages 209 to 211. Physical exercise was the only intervention placed on the top rung. Cognitive exercises, video-game training and meditation were graded strong signals. Drugs, brain stimulation and neurofeedback were graded weaker signals. Traditional education and nature exposure were graded reasonable hypotheses.

The interesting result is where their own work landed. Their video-game training research sits on the second rung rather than the first, and the top rung went to an intervention in which neither author has a stake. A book written to sell a game would not have published the rubric first and then graded itself down with it. The scale also produced two admissions most authors would have avoided: that there is no convincing evidence traditional education improves cognitive control, and that their own category is not prescriptive.

Two extensions are worth noting because the three rungs do not cover them. The ladder grades interventions, and a reader also needs a way to grade claims, sorting a cited empirical finding from a reinterpretation of somebody else's finding from an outright conjecture. Applied to the best-sourced book in this vault, that sort put roughly a third of its substantive claims in each of the three boxes, meaning about two claims in three were not evidence for its thesis, with no typographic signal of which was which. The unit to grade is the inferential step, not the citation. The ladder also lacks a rung for a framework, an organising scheme that no single study could confirm or refute, which is what several of the large theories in this vault are. Filing those on the bottom rung misdescribes them and filing them as findings is worse; they are judged on coherence instead.

What it does not say

It does not say the scale is authoritative. It is the authors' own scale, their own grading, and their own product is among the things being graded. That combination is exactly the situation a conflict of interest disclosure exists to annotate, and there is none.

It does not say the rubric was applied consistently. The chapter breaks it once in the same breath. Nature exposure is graded a reasonable hypothesis and then, in the next sentence at page 201, highly encouraged, on health benefits the section never cited.

It does not stand in for the authors having applied it everywhere. The instrument is only worth anything when it is turned on the pages a reader will act on, and it was not used on two other chapters of the same book. Steal the instrument, and do not take the grades on trust.

---