External Validation

What it is

A model can look accurate on its developers' own data and still perform poorly elsewhere, because patients, equipment and record-keeping differ between places. External validation checks for that. The best-known recent example is Andrew Wong and colleagues' 2021 study in JAMA Internal Medicine. They tested the Epic Sepsis Model, a proprietary tool in use at hundreds of US hospitals, on 38,455 hospital stays at Michigan Medicine. Sepsis occurred in 2,552. The model failed to identify 1,709 of them, 67 per cent, while generating alerts for 18 per cent of all hospitalised patients. The authors noted that the model's ability to identify sepsis had not been adequately evaluated despite its widespread use.

In effect

For patients and hospitals, external validation is the evidence that a tool works for people like them. Roxana Daneshjou and colleagues' 2022 Stanford study is another form of it: three published dermatology AI models were tested on a new, carefully built set of 656 images and all three performed below what their developers had reported, and worse on dark skin.

External validation also differs from regulatory clearance. In the United States most AI-enabled devices reach the market through the FDA's 510(k) route, which asks a maker to show a device is substantially equivalent to one already legally sold. That comparison is a different question from whether the device performs well on a particular hospital's patients.

What it does not say

A successful external validation shows that a model's accuracy holds in a new setting. It does not show that using the model improves patients' outcomes; that needs a trial comparing care with and without the tool, such as the Swedish MASAI trial of AI-supported breast screening. And a single validation applies to one place and one period. Performance can change as patient populations, practices or the software itself change.

---