Inter-Rater Reliability

What it is

The question is simple and the reporting is not.

Raw agreement is the share of cases on which two readers filed the same answer. A reader can interpret it without training, and it is what the pathology and radiology studies in this publication's work report.

An agreement coefficient discounts the agreement that chance alone would produce. Kappa is the common one, and it is why a 77 per cent raw agreement and a moderate kappa can describe the same data: the coefficient depends on how common each category is, so two studies with identical raw agreement can report very different coefficients.

Test-retest reliability is the same question asked of one person twice, and it is reported alongside the between-rater figure throughout this literature. The design constraint is the one most studies get wrong: re-presenting a memorable case to the same judge within a session measures memory rather than stability. The studies this publication trusts separated the sittings by months.

And the sentence the whole vocabulary rests on. J. Richard Landis and Gary G. Koch set out the bands at pp. 164 to 165 of their 1977 paper: below 0.00 poor, 0.00 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, 0.81 to 1.00 almost perfect. Immediately below the table, verbatim:

"Although these divisions are clearly arbitrary, they do provide useful 'benchmarks' for the discussion of the specific example in Table 1."

Arbitrary, and scoped to one data set. That is not an argument against using them. It is an argument for never treating a band label as a finding, for asking what the raw agreement was whenever a paper reports only a coefficient, and for never describing a paper as using these bands when it does not.

In effect

In pathology: the hardest call, and the best measured

Sara L. Jackson and colleagues report that forty-nine pathologists reading the same sixty breast specimens agreed with their own earlier reading on atypia 53 per cent of the time, and with a different pathologist 43 per cent of the time, against 92 and 89 per cent for invasive cancer. The agreement rate is a property of the diagnostic category and not of the profession. Joann G. Elmore and colleagues record an average of ten different diagnostic terms applied to each of 240 skin biopsy cases across 187 pathologists and 8,976 interpretations.

In psychiatry: the lowest figures and the most candid design

Darrel Regier and colleagues ran the DSM-5 field trials across eleven academic centres in the United States and Canada, with consecutive patients screened, stratified and randomly assigned to two clinicians who each conducted an independent usual-practice interview on separate occasions, blind to any previous diagnosis, and with no comorbidities or unclear presentations excluded. Their headline, verbatim: "Overall, five diagnoses were in the very good range (kappa=0.60-0.79), nine in the good range (kappa=0.40-0.59), six in the questionable range". The design is the reason to trust it: they declined to make the task easier than clinical practice.

In regulation: the number a consumer actually meets

Alan Boyd and colleagues put ten identical hospital-service vignettes to 286 inspectors and found agreement ranging from almost perfect on one case to barely better than chance on another, with groups of inspectors more reliable than individuals. A one-word rating on a public register is the output of a process whose reliability varies this much by case, which is worth knowing before reading a rating as a measurement.

What it does not say

It does not say agreement is accuracy. Two raters filing the same answer does not make either answer right, and none of the studies above measured accuracy against a criterion or any outcome for a patient. Nothing on this page is clinical advice.

It does not say a low coefficient means a bad clinician or a bad instrument. A coefficient falls when a category is rare, when the scale is fine-grained, and when the cases are genuinely borderline, and only the first two are measurement problems.

It does not supply thresholds. The bands in universal use were called arbitrary by the people who wrote them, and scoped to a single example.

And it does not transfer across papers. A study reporting raw agreement and a study reporting kappa are not comparable without recomputation, and the figures on this page are reported as each paper printed them.

---