Affective Computing

What it is

The commercial products in this field output a small set of emotion labels from an image or video of a face, usually the six categories of the twentieth-century research tradition, sometimes with a valence score alongside.

The best available measurement of how well they do it is Damien Dupre and colleagues' 2020 comparison, which ran eight commercially available classifiers over the same 937 videos. Pooled across both stimulus sets, their mean true positive rates ran from 48 to 62 per cent. Fourteen human observers, judging the same videos in a seven-option forced choice, reached 72.48 per cent. Fourteen is a number that belongs in the sentence rather than in a footnote. The paper also records that independent peer-reviewed validation of these classifiers is scarce and mostly limited to posed displays, and it reports no demographic breakdown of its own.

The obvious reading of a gap like that is that the software is a weak detector of a well-defined thing. The 2019 consensus review suggests the problem sits further back, and its sentence is the one to keep: all face-emotion associations observed in an experiment reflect human consensus, the degree of agreement between self-judgments, expert judgments and other observers' judgments, and "these types of agreement are often referred to as accuracy, but this may or not be valid". In the same section: "Currently, there are no objective measures, either singly or as a pattern, that reliably, uniquely, and replicably identify an instance of one emotion category compared with an instance of another."

So a 98 per cent figure against expert coders on images from one database is agreement with those coders. It is not a measurement of what anybody felt.

What the systems can do, on a positive result that deserves its full space. Louisa Kulke and colleagues ran a preregistered comparison in which twenty students imitated happy, angry and neutral portraits while electrodes recorded cheek and brow muscles, and the software's smile-minus-frown score tracked the muscle score closely; their conclusion was that the software's results are comparable to the muscle findings. Twenty people, seventeen of them women, three categories of deliberate imitation. Set beside the collapse off standardised stimuli, that draws the line precisely: these systems can tell you, in agreement with a muscle sensor, that a face has moved into a smile. Two of the same authors report elsewhere that machine classification approached human performance on spontaneous displays, which conflicts with the head to head above, and that disagreement is unresolved.

And the demographic literature, with the task named correctly. The work usually cited here, Gender Shades by Joy Buolamwini and Timnit Gebru, measures commercial gender classification by gender and Fitzpatrick skin type, not emotion. On expression recognition itself the best-powered study is Tian Xu and colleagues' 2020 preprint, whose clearest gap is age. The mechanism behind such gaps is at stereotypical dataset bias.

In effect

In research: three stimulus sets, one engine

Sabrina Stockli and colleagues recorded 97 per cent on standardised posed photographs, 63 per cent on real people imitating those photographs, and 10 per cent on real people imitating fear, with a second engine at 73, 55 and 1 per cent. A system drops from 97 to 10 without getting any worse at seeing.

In the official record: a precedent for buying this kind of inference

The United States Government Accountability Office reported in 2013 that four meta-analyses covering more than 400 studies over sixty years put human accuracy at identifying deceptive behaviour from behavioural cues at 54 per cent, the same as or slightly better than chance. The Transportation Security Administration had spent about 900 million dollars on behaviour detection by then and ended the programme in 2017. When it later offered 178 sources to justify a revised list of thirty-six indicators, the office found 175 of them did not provide valid evidence, and that the remaining three supported eight indicators.

In law: the doubt written into statute

The EU AI Act defines an emotion recognition system as one for identifying or inferring emotions or intentions from biometric data. Inferring emotions in workplaces and education institutions has been prohibited since 2 February 2025, with an exception for systems placed on the market strictly for medical or safety reasons. Where permitted, emotion recognition is classified as high-risk, with those obligations applying from 2 December 2027. Nothing in the Act restricts retail, advertising or market research, and none of this makes any particular system lawful to deploy.

What it does not say

It does not say these systems measure nothing. They agree with a muscle sensor about facial movement, and in a preregistered comparison they did so closely.

It does not say the accuracy figures are false. They are measurements of agreement with human coders, which is a real quantity, and the criticism is that it is not the quantity a buyer assumes.

It does not say a better classifier would settle the question. The open question is what a benchmark would have to be made of before a score against it could mean that a person felt something, and a higher score against the same benchmark does not address it.

What nobody has published, and this publication looked, is the comparison a buyer would most want: the commercial products, on the same stimuli, broken down by skin tone and age.

---