Read the Count Before the Score
Across 1,272 products in 120 categories, average user ratings did not converge with objective quality measures. People weight them heavily anyway, and fail to discount them when the sample is too small to mean anything.
Key takeaways
- Across 1,272 products in 120 categories, average user ratings did not converge with Consumer Reports scores, the standard objective quality measure in this literature.
- Average ratings are often based on sample sizes too small to be informative, and do not predict resale prices in the used-product marketplace.
- Ratings run higher for more expensive products and premium brands even after controlling for objective quality, so price is partly feeding back into the score.
- People weight the average heavily against price and review count, and fail to moderate their reliance on it according to sample size. The authors call it an illusion of validity.
- Rating system design shapes the numbers too: multidimensional systems produce different overall ratings from single-dimensional ones.
1,272 products, across 120 categories, all of them things where quality can actually be measured. That is the dataset behind the most uncomfortable finding in consumer research, and the discomfort is not that ratings are gamed. Honest ratings, collected in good faith, turn out not to tell you what you think they tell you.
What the ratings do not track
Bart de Langhe, Philip Fernbach and Donald Lichtenstein compared average user ratings against Consumer Reports scores, the standard measure of objective quality in this literature. The two lack convergence, in the authors' phrase. That is weaker than saying they are unrelated and stronger than most shoppers would guess.
That is the headline and it has three supports underneath it. Average [ratings](https://thewhyweproject.com/articles/why-does-one-bad-review-hurt-so-much) are often based on sample sizes too small to be informative. In a small sample one angry reviewer is a large share of the score, and shoppers read that share as a warning. They do not predict resale prices in the used-product market, which is a real-money test of whether a product turned out to be any good. And they are higher for more expensive products and premium brands, even after controlling for the Consumer Reports score, which means part of what you read as quality is price feeding back into perception.
Worth pausing on the method before going further, because this is the sort of claim that has been retracted elsewhere. This is not a priming experiment on forty undergraduates. It is an archival comparison of two independent datasets across a large sample of real products, with the psychological part, how people weight the average, tested separately. It is the kind of finding that tends to survive re-examination, and we would not be leaning on it this hard if it were not.