Stereotypical Dataset Bias

What it is

There are two different holes a training set can have, and they are usually discussed as one.

Representational bias. A group is thin everywhere: fewer images of them, across every label. This is what people mean when they say a dataset is unrepresentative, and it is what a headline demographic breakdown measures.

Stereotypical bias. A group is present overall but thin in one particular label, so the model's idea of what that label looks like is built almost entirely from other people.

The experiment is what makes the distinction useful rather than merely plausible. Dominguez-Catena and colleagues took a large facial expression dataset as a baseline and engineered training subsets carrying varying strengths of each kind of skew, then computed per-class recall separately by group on the full test set.

Their result, in their words: "representational bias has a weaker impact than expected. Models exhibit a good generalization ability even in the absence of one gender in the training dataset. Conversely, stereotypical bias has a significantly stronger impact, primarily concentrated on the biased class, although it can also influence predictions for unbiased classes."

The magnitude, in their words: in over-represented groups there is "a weak increase in recall (under 0.1)", while in under-represented groups "the impact is more pronounced ... with a total drop of up to 0.4 in the extreme cases compared to the balanced reference".

So a group missing entirely from the training set barely hurts, and a group missing from one emotion label moves recall in that label by as much as four tenths. The quantity to interrogate is not how many images of a group a dataset holds. It is how those images are distributed across the labels.

One limit stated in the same breath. It is one research group, on one benchmark dataset, manipulating gender rather than skin tone. The same first author's twenty-dataset audit of demographic bias metrics flags that its own protected-attribute labels come from another classifier, which is a measurement limit worth carrying wherever these figures travel.

In effect

In research: the skew is in the data of the study usually cited

Tian Xu and colleagues' seven-way expression task is the best-powered demographic study of expression recognition itself, and its own training data carries exactly this pattern: ten of 164 anger images were African-American, and five of 79 fear images were. Its reported gaps include mean class-wise accuracy of 51.3 per cent for people aged 70 and over against 65.9 per cent for people aged 20 to 39, with African-American accuracy at 68.1 per cent, above the 65.9 per cent for Caucasian faces, and Asian faces lowest at 60.0 per cent. The authors warn that their smallest subgroups are unstable, and the paper is an arXiv preprint and is cited as one.

In the public conversation: the audit that measured a different task

The work usually quoted on demographic gaps in emotion recognition is Gender Shades, by Joy Buolamwini and Timnit Gebru, with error rates from 20.8 to 34.7 per cent on darker-skinned women against 0.0 to 0.8 per cent on lighter-skinned men. It measures commercial gender classification, audited by gender and by Fitzpatrick skin type, not emotion, and its authors state in the paper that whether such gaps persist in other vision tasks needs further work. Importing its figures into an emotion claim gets the task wrong.

In commerce: the breakdown worth requesting

A buyer auditing a classifier usually asks for a demographic breakdown of the training set. On this evidence the more diagnostic request is a breakdown by group and by label at once, because that is the cell where recall collapses. Nobody we could find has published the comparison a buyer would most want: the commercial products, on the same stimuli, broken down by skin tone and age.

What it does not say

It does not say representational balance is pointless, only that its effect was weaker than expected in this experiment, on this dataset, with gender as the manipulated attribute.

It does not license any claim about any particular deployed product. The experiment manipulated training sets; it did not audit a vendor.

It does not quantify skin-tone effects, which are measured in a thinner and newer literature. A 2025 peer-reviewed conference paper reports that audited models were more likely to read a negative emotion from people with dark skin tones even when they were smiling, and an unrefereed 2024 preprint reports one specific confusion at 2.1 times the rate in Black women as in White women. Those are a conference paper and a preprint and are labelled as such.

And no accuracy gap, in either direction, bears on whether a facial configuration indicates an emotion at all, which is the prior question and the one the four criteria on reverse inference are built to test.

---