Algorithmic Bias

What it is

In 2022 Roxana Daneshjou and colleagues at Stanford published the Diverse Dermatology Images dataset, 656 pathologically confirmed skin images matched so that light skin (Fitzpatrick types I and II) and dark skin (types V and VI) could be compared directly. They tested three published dermatology AI models on it. All three performed below the accuracy reported on their own test data, and worse on dark skin than on light. One model, DeepDerm, detected 69 per cent of malignancies on light skin and 23 per cent on dark skin. Dermatologists also performed worse on dark skin and on uncommon diseases.

The study pointed to the source of the gap. When two of the models were retrained on the more varied images, the difference between light and dark skin closed.

In effect

For patients, algorithmic bias means that a tool's headline accuracy may not apply to them. It is one reason external validation, testing a model on new patients in new settings, matters before a tool is used widely. Andrew Wong and colleagues' 2021 test of the Epic Sepsis Model at Michigan Medicine is a related case: a widely used tool performed much worse on outside patients than its spread implied, missing 67 per cent of sepsis cases.

Urs Gasser and Viktor Mayer-Schönberger's Guardrails (2024) argues in chapter 9 that bias cannot be fixed by technical tools alone, and that society first has to choose which kinds of bias to eliminate.

What it does not say

Algorithmic bias does not mean an algorithm is worse than the people it assists. In the Stanford study the dermatologists also struggled with dark skin, and after retraining the models performed as well as or better than the dermatologists on that subset. Nor is the bias fixed: the same study showed it could be reduced with better data. The lesson is that performance has to be checked group by group, not assumed from an overall figure.

---