What a Camera Can Read on a Face

Answer to the question

Can emotion recognition software tell how you feel?

Not in the sense the products imply. The consensus review finds the predicted facial configurations occur above chance and are not specific or reliable enough to count as diagnostic displays of any emotional state. A camera plus a classifier can tell you a face has moved into a smile.

Key takeaways

• The commissioned consensus review finds the predicted facial configurations occur with the predicted emotion above chance, and are not reliable or specific enough across contexts, individuals and cultures to count as diagnostic displays of any emotional state.
• The strongest opposing case is no longer the basic six: twenty-eight categories were reliably identified in 92 per cent of 1,500 naturalistic photographs, and sixteen expression types recurred in six million videos from 144 countries.
• One commercial classifier matched 97 per cent of standardised posed photographs, 63 per cent of real people imitating those same photographs, and 10 per cent of real people imitating fear.
• Inferring emotion in workplaces and education institutions has been prohibited across the European Union since 2 February 2025, with an exception for medical and safety purposes.

- The consensus review that emotion recognition answers to finds the predicted facial configurations do occur above chance, and are not reliable or specific enough across contexts, individuals and cultures to count as diagnostic displays of any emotional state.
- The live alternative is no longer the basic six: twenty-eight categories were reliably identified in 92 per cent of 1,500 photographs, and sixteen expression types recurred across 144 countries.
- One classifier matched the labels on 97 per cent of posed photographs across all categories, and on imitated fear alone its figure was 10 per cent.
- Inferring emotion in workplaces and education has been prohibited across the European Union since 2 February 2025, with a medical and safety exception.

The question everyone asks about emotion recognition is how accurate it is. That is the wrong question, because an accuracy figure needs something to be accurate against. The better question is what these systems are agreeing with.

Ask what the classifiers are scored against

In 2020 Damien Dupre and colleagues published the only peer-reviewed head to head of eight commercially available emotion recognition classifiers, running all of them over the same 937 videos. Pooled across both stimulus sets, their mean true positive rates ran from 48 to 62 per cent. Fourteen human observers, judging the same videos in a seven-option forced choice, reached 72.48 per cent, on a panel of fourteen.

The obvious reading of a gap like that is that the software is a weak detector of a well-defined thing. The consensus review published the year before, written in part by computer scientists who build these systems, suggests the problem sits further back:

> "All face-emotion associations that are observed in an experiment reflect human consensus, that is, the degree of agreement between self-judgments (from the participants), expert judgments (from the scientist), and/or judgments from other observers... These types of agreement are often referred to as accuracy, but this may or not be valid."

The same section states the position in one line: "Currently, there are no objective measures, either singly or as a pattern, that reliably, uniquely, and replicably identify an instance of one emotion category compared with an instance of another." A score of 98 per cent against expert coders on images from one database is agreement with those coders. It is not a measurement of what anyone felt.

Start with the review's four criteria

That review is the document the question turns on. Lisa Feldman Barrett is its first author, with a cognitive neuroscientist, two computer scientists and a developmental psychologist, and it was commissioned by a journal whose purpose is to tell non-specialists what a literature supports. Its stated audience is "consumers of emotion research", which is anyone being sold emotion recognition. Of its sixty-eight published pages, the twenty-four holding the executive summary, the criteria and the production evidence were read here, and its recommendations were not, so nothing below describes them.

Barrett set the argument out for a general audience first, in How Emotions Are Made, which is where this article started. We cite the book for that framing and take no finding from it.

Its finding, in its own words from the executive summary, is narrow and double-edged. The six categories are each expressed with a particular configuration of facial movements "more reliably than would be expected by mere chance", and instances of those categories are not expressed with movements "sufficiently reliable and specific across contexts, individuals, and cultures to be considered diagnostic displays of any emotional state".

Reading a smile as happiness is what the review calls a reverse inference, and it names the application directly: every time an app identifies a frowning configuration and proclaims the person sad, it has made one. Four criteria have to hold before a configuration earns the name of an emotional expression.

1. Reliability. Given that someone is happy, a smile should appear more often than chance.
2. Specificity. A scowl must express anger and not also sadness, confusion, indigestion or an attempt to influence somebody, at better than chance.
3. Generalisability. The pattern should replicate in infants, in congenitally blind people and in remote cultures.
4. Validity. The configuration should track measures of emotion that do not depend on anybody's subjective report.

The scoring bands the review applies are taken from its opponents' own published thresholds, 70 to 90 per cent counting as strong support and 20 to 39 per cent as weak. Validity, it says, is almost never addressed at all. Its summary scorecard rates reliability weak for spontaneous adult expression in the lab and in the field, with specificity unknown in both, and rates recognition moderate to strong when people pick from a short list of emotion words and weak to moderate when they say what they see in their own words.

Weigh the many-categories evidence

What is defended now is not the six prototypes. Paul Ekman, who died in 2025, surveyed 248 emotion scientists in 2014 and got a 60 per cent response; 88 per cent endorsed compelling evidence for universals in some aspect of emotion, 80 per cent endorsed universal signals in face or voice, and five categories were endorsed as empirically established by between 76 and 91 per cent of those taking a discrete or mixed view. Surprise was not among the five. He asked James Russell, his best-known critic, to vet the selection process and the questions, and Russell contributed one.

The modern version is quantitative and large. Alan Cowen, with Dacher Keltner, had 1,794 raters judge 1,500 naturalistic photographs and found 92 per cent of them reliably identified with at least one of twenty-eight categories, at agreement levels the authors show are comparable with the meta-analytic figures for prototypical posed expressions. Erasing scenery, objects, clothing and jewellery left those judgements closely correlated with judgements of the originals across all twenty-eight categories. In 2021 Cowen and colleagues reported sixteen expression types across six million videos from 144 countries, with each type's associations to context 70 per cent preserved across twelve world regions.

The authors of that study state two limits themselves: their classifier could not handle twelve other documented expressions, disgust and fear among them, and the algorithms are proprietary.

Their companion paper in the same journal issue as the review concedes the central point in print: "we agree with Barrett et al. that emotions are not expressed in six prototypical patterns of facial-muscle movement." What they defend is a larger space of facial and bodily configurations, and their objection is that a test built on six categories, which they calculate capture at most about 30 per cent of the systematic variance, understates what a face conveys. The review for its part says it is "not a referendum on the 'basic emotion' view". On how many categories the face carries, and how far they travel, the science is open.

Follow one system across three stimulus sets

What both camps have stopped defending is exactly what the products ship. They output the six basic emotions from a face, and the clearest demonstration of what that buys is a 2018 validation study led by Sabrina Stockli, which put two widely licensed engines through the same three tasks.

What the system saw	FACET	AFFDEX
Standardised posed photographs from three databases	97 per cent	73 per cent
Real people imitating those same photographs	63 per cent	55 per cent
Real people imitating fear	10 per cent	1 per cent

One further figure from the same study: classifying the valence of real facial responses to pleasant pictures, AFFDEX was correct 17 per cent of the time and 97 per cent on responses to unpleasant ones. The second study's participants were 110 students, and the authors state that only Caucasian participants without glasses, beards or scarves were included.

Louisa Kulke and colleagues, in a preregistered comparison, had twenty students imitate happy, angry and neutral portraits while electrodes recorded the cheek and brow muscles, and the software's smile-minus-frown score tracked the muscle score closely. Their conclusion was that the software's results are comparable to the muscle findings. The sample was twenty people, seventeen of them women, judged on three categories of deliberate imitation.

Set beside the Stockli figures that study draws a precise line. These systems can tell you, in agreement with a muscle sensor, that a face has moved into a smile. Two of the same authors also report elsewhere that machine classification approached human performance on spontaneous displays, which conflicts with the head to head above, and that disagreement is unresolved.

Check which demographic gap was actually measured

The work usually cited here is Gender Shades, by Joy Buolamwini and Timnit Gebru, and it measures commercial gender classification, audited by gender and by Fitzpatrick skin type, not emotion. Its error rates ran from 20.8 to 34.7 per cent on darker-skinned women against 0.0 to 0.8 per cent on lighter-skinned men, and its authors say in the paper that whether such gaps persist in other vision tasks needs further work.

The mechanism is the part that rarely gets reported, and it decides how any figure that follows should be read. Iris Dominguez-Catena and colleagues engineered training sets carrying two different skews and found that a group missing across all classes barely hurt, while a group under-represented in one specific emotion label moved recall in that label by as much as four tenths. The skew that matters is between a group and a label, not between a group and the data set as a whole.

Ask the question of expression recognition itself and the best-powered answer is a 2020 Cambridge preprint led by Tian Xu, on a seven-way task, whose own training data carry that kind of skew: ten of 164 anger images were African-American and five of 79 fear images were. Its clearest gap is age, at mean class-wise accuracy of 51.3 per cent for people aged 70 and over against 65.9 per cent for people aged 20 to 39. Accuracy for African-American faces came out at 68.1 per cent, above the 65.9 per cent for Caucasian faces, with Asian faces lowest at 60.0 per cent.

That ordering should be held loosely. The preprint is unrefereed, and its authors warn that their smallest subgroups are unstable, which is where a label-level skew of ten images does its damage.

Skin tone in emotion recognition is a thinner, newer and more weakly venued literature than the gender-classification work cited in its place, and it is not empty. A 2025 peer-reviewed conference paper by Rina Khan and Catherine Stinson reports that the audited models were more likely to read a negative emotion such as anger or sadness from people with dark skin tones even when they were smiling. An unrefereed 2024 preprint led by Kaylee Chhua reports one specific confusion, anger scored as disgust, at 2.1 times the rate in Black women as in White women.

What nobody we could find has published is the comparison a buyer would want: the commercial products, on the same stimuli, broken down by skin tone and age.

Set the official record beside the law

There is a precedent for buying this kind of inference at scale. The United States Government Accountability Office reported in 2013 that four meta-analyses covering more than 400 studies over sixty years put human accuracy at identifying deceptive behaviour from behavioural cues at 54 per cent, the same as or slightly better than chance. The Transportation Security Administration had spent about 900 million dollars on behaviour detection by then; it ended the programme in 2017. When it later offered 178 sources to justify a revised list of thirty-six indicators, the office found 175 of them did not provide valid evidence, and that the remaining three supported eight indicators.

The European Union has since written the measurement doubt into statute. The AI Act defines an emotion recognition system as one for identifying or inferring emotions or intentions from biometric data. Inferring emotions in workplaces and education institutions has been prohibited since 2 February 2025, with an exception for systems placed on the market strictly for medical or safety reasons. Where it is permitted at all, emotion recognition is classified as high-risk, and the obligations attaching to that classification apply from 2 December 2027.

Outside those listed high-risk uses the Act imposes no conformity duties, and that is the whole of the freedom retail, advertising and market research enjoy. Article 50(3) reaches them anyway. It requires a deployer of an emotion recognition system to inform the natural persons exposed to it that the system is operating, and the only carve-out is for uses the law permits for detecting, preventing or investigating criminal offences. It has applied since 2 August 2026. Shoppers and research subjects are natural persons exposed to such a system, so they have to be told.

Shoppers and research subjects are natural persons exposed to such a system, so they have to be told.

Retail is also not a safe harbour for a camera pointed at the staff. Staff are at work, and inferring their emotions is prohibited outright, whatever the same shop is doing with the same camera on the customer side of the till. None of this makes any particular system lawful to deploy either, because a disclosure made is not a finding that the inference underneath it works. The legislature's stated reason sits closer to the review than legislative reasoning usually does:

> "Among the key shortcomings of such systems are the limited reliability, the lack of specificity and the limited generalisability."

Three of the review's four criteria, in the same order, in the legislature's own words. On the labelling and disclosure side of the same Act, [what an AI label does to a product](https://thewhyweproject.com/articles/what-an-ai-label-does-to-a-product) is a separate story, and [what people say about being watched](https://thewhyweproject.com/articles/why-we-value-privacy-and-share-anyway) is a third. Human oversight as the standard remedy is covered where we looked at [what healthcare AI can be checked against](https://thewhyweproject.com/articles/what-ai-in-healthcare-can-and-cannot-do), and the contrast with this case is sharp: a diagnostic model can be wrong because a biopsy exists to contradict it.

Ask what the camera is accurate about

Go back to the Stockli rows with that in mind. The 97 per cent is an all-category score against the labels on posed photographs, and the 10 per cent is a single-category score on real people imitating fear, so the two are not one measurement degrading; they are different questions, and the system did not get worse at seeing between them. The posed photographs were made by actors instructed to produce a configuration the research tradition had already specified, and the review's own warning about that paradigm is that if people's facial poses are shaped by the common view, a relationship between those poses and the hypothesised categories is evidence of the belief.

So the accuracy figures in the brochures are measurements of agreement, and the Dupre comparison scores eight products against fourteen people with a word list in front of them. That is a real quantity, and it is not the quantity a buyer of emotion recognition thinks they are getting.

Whether a face has moved into a configuration is a camera question, and cameras are good at it. Whether the person behind it is angry remains, after two decades of measurement, a question about what anger is. So the open question is what a benchmark would have to be made of before a score against it could mean that a person felt something, and on that the two camps in the science are still, in print and in good faith, apart, which is why the next classifier scoring higher would settle nothing.

Common questions

Can emotion recognition software tell how you feel?

Not on this evidence. The consensus review finds the predicted facial configurations occur above chance and are not specific or reliable enough to count as diagnostic displays of any emotional state. Commercial engines agree well with muscle sensors that a face has moved into a smile. In the only peer-reviewed comparison of eight of them, pooled true positive rates ran from 48 to 62 per cent.

Is emotion recognition banned under the EU AI Act?

Not generally. The Act prohibits inferring emotions in workplaces and education institutions, in force since 2 February 2025, with a medical and safety exception. Elsewhere it is classified high-risk, those obligations applying from 2 December 2027. Outside those uses no conformity duties apply, but Article 50(3) has required deployers, retail and market research included, to tell people exposed to a system that it is running since 2 August 2026. None of this makes any system lawful to deploy.

Is emotion recognition less accurate on darker skin?

The widely cited Gender Shades figures measure gender classification, not emotion. On expression recognition itself the best-powered numbers come from an unrefereed 2020 preprint, where age is the clearest gap, at 51.3 per cent for the over-seventies against 65.9 per cent for people in their twenties and thirties, and African-American accuracy sits above Caucasian; its authors warn their smallest subgroups are unstable. A 2025 conference paper and another preprint do report skin-tone effects.

Sources

Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). 'Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements.' Psychological Science in the Public Interest, 20(1), 1-68. doi:10.1177/1529100619832930

Cowen, A., Sauter, D., Tracy, J. L., & Keltner, D. (2019). 'Mapping the Passions: Toward a High-Dimensional Taxonomy of Emotional Experience and Expression.' Psychological Science in the Public Interest, 20(1), 69-90. doi:10.1177/1529100619850176

Cowen, A. S., & Keltner, D. (2020). 'What the face displays: Mapping 28 emotions conveyed by naturalistic expression.' American Psychologist, 75(3), 349-364. doi:10.1037/amp0000488

Cowen, A. S., Keltner, D., Schroff, F., Jou, B., Adam, H., & Prasad, G. (2021). 'Sixteen facial expressions occur in similar contexts worldwide.' Nature, 589(7841), 251-257. doi:10.1038/s41586-020-3037-7

Ekman, P. (2016). 'What scientists who study emotion agree about.' Perspectives on Psychological Science, 11(1), 31-34. doi:10.1177/1745691615596992

Dupre, D., Krumhuber, E. G., Kuster, D., & McKeown, G. J. (2020). 'A performance comparison of eight commercially available automatic classifiers for facial affect recognition.' PLOS ONE, 15(4), e0231968. doi:10.1371/journal.pone.0231968

Stockli, S., Schulte-Mecklenbeck, M., Borer, S., & Samson, A. C. (2018). 'Facial expression analysis with AFFDEX and FACET: A validation study.' Behavior Research Methods, 50(4), 1446-1460. doi:10.3758/s13428-017-0996-1

Kulke, L., Feyerabend, D., & Schacht, A. (2020). 'A Comparison of the Affectiva iMotions Facial Expression Analysis Software With EMG for Identifying Facial Expressions of Emotion.' Frontiers in Psychology, 11, 329. doi:10.3389/fpsyg.2020.00329

Buolamwini, J., & Gebru, T. (2018). 'Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification.' Proceedings of Machine Learning Research, 81, 77-91.

Xu, T., White, J., Kalkan, S., & Gunes, H. (2020). 'Investigating Bias and Fairness in Facial Expression Recognition.' arXiv:2007.10075v3.

Dominguez-Catena, I., Paternain, D., Jurio, A., & Galar, M. (2024). 'Less can be more: representational vs. stereotypical gender bias in facial expression recognition.' Progress in Artificial Intelligence. doi:10.1007/s13748-024-00345-w

Khan, R., & Stinson, C. (2025). 'Auditing Facial Emotion Recognition Datasets for Posed Expressions and Racial Bias.' 2025 3rd International Conference on Foundation and Large Language Models. doi:10.1109/fllm67465.2025.11391216

Chhua, K., Wen, Z., Hathalia, V., Zhu, K., & O'Brien, S. (2024). 'From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models.' arXiv:2408.14842v1.

United States Government Accountability Office (2013). 'Aviation Security: TSA Should Limit Future Funding for Behavior Detection Activities.' GAO-14-159.

European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence, as amended. Articles 3, 5 and 50, Annex III and Recital 44.

Barrett, L. F. (2017). How Emotions Are Made: The Secret Life of the Brain. Houghton Mifflin Harcourt.

Related concepts

• Stereotypical Dataset Bias
• Emotion Fingerprints
• Reverse Inference
• The Basic Emotion Method
• Affective Computing
• Construct Validity
• The Population to Individual Gap