Why We Trust Online Ratings Anyway
In 1927 a book diagnosed the problem correctly: the buyer's competence had been outrun by what was being sold, and the cure was impartial published test results. We got the cure, and it turned out not to measure the thing.
Answer to the question
Can you trust online ratings?
Not for quality you cannot check yourself. A 2016 study led by Bart de Langhe examined 1,272 products across 120 categories and found average user ratings did not converge with Consumer Reports scores, did not predict resale prices, and ran higher for premium brands once quality was controlled for. Buyers weight the average heavily anyway, without discounting a thin sample.
Key takeaways
- Across 1,272 products in 120 categories, average user ratings showed no convergence with Consumer Reports quality scores and did not predict resale prices.
- An extra half-star on Yelp caused restaurants to sell out 19 percentage points more often, an increase of about half, in a peer-reviewed 2012 study.
- The much-quoted finding that a one-star rise lifts restaurant revenue 5 to 9 per cent sits in a Harvard Business School working paper that has never been peer reviewed.
- Roughly 16 per cent of Yelp restaurant reviews were filtered by the platform's own algorithm, used by researchers as a proxy for fraud rather than a count of it.
Online ratings arrived as the answer to a complaint almost a century old: that an ordinary buyer can no longer judge what is being sold. Stuart Chase and F. J. Schlink made that complaint in Your Money's Worth, and their remedy was to publish impartial comparative tests so people could buy to specification instead of buying to advertising.
The wish has since been granted in a form they never imagined, free to read and attached to almost everything on sale. Their diagnosis has held up. Their cure has not.
Start with the 1927 diagnosis
The argument opens with a loss of competence the authors insist is dated rather than innate. Before mass production, they write, adulteration and slack workmanship "were readily detectible", and the housewife "had the standards of her own competent cottage industry to judge them by" (p. 27). Maker and buyer were then separated, and "the whole system for judging sound values went into decay" (p. 28). What was left was trial and error, at a bitter price.
They close the escape route too. Education will not restore the lost skill, because "the time has gone, possibly forever, when it is possible for each of us to become informed on all the things we have to buy" (p. 256).
Then comes the sentence everything here rests on, stated as a law. "The less chance the consumer has of knowing the technical facts underlying a product, the easier it is to describe its merits. Such descriptions are difficult to verify" (p. 259). Two lists follow it. Cars, paints, vacuum cleaners, radios and refrigerators are complex and heavily advertised. Cement, structural steel, fence wire, sugar, salt and tincture of iodine are simple and barely advertised at all.
The deficit named is technical, not intellectual. One patent tonic, the authors write, taught the advertising trade that it "no longer needed to confine his activities to the ignorant; he had learned the technique for capsizing the intelligent, provided they knew no chemistry" (p. 129). The book does not hold that line, since eighty pages earlier it asks whether "the pitiful ignorance and gullibility of the consumer" could be better illustrated (p. 48). We read the wobble as authors who had the right theory and kept sliding back into the wrong one.
Credit Cook for the only experiment
For a book about judgement, Your Money's Worth contains almost no psychology. No psychologist is named in its 267 pages, and its roughly 130 references lean on Federal Trade Commission dockets and medical journals rather than on laboratories. One piece of evidence survives the vault's audit, and the credit belongs to Rosamond C. Cook of Teachers College, Columbia.
Cook took nine makes of cotton sheeting and ranked them by laboratory analysis, following Bureau of Standards procedure. She then asked ordinary shoppers, and trained shop assistants, to rank the same nine by looking and touching. The book reports that "the consumer had only one chance in nine of determining relative quality in the light of her own shopping experience. The worst quality alone she recognized" (p. 85). The salespeople did no better.
Cook's own qualifier travels with the result: shoppers judge accurately only where quality varies widely. Our copy of it is second-hand, because her original publication has not been located.
Economics took four more decades to name what p. 259 describes. Phillip Nelson's 1970 paper in the Journal of Political Economy split product qualities in two, into search qualities you can inspect before buying and experience qualities that only use reveals.
The third category arrived in 1973, from Michael Darby and Edi Karni. They called them credence qualities, the ones a buyer cannot evaluate even after using the thing unless further cost is incurred. Online ratings are the mass-market attempt to handle that category by delegation.
Check what the average actually tracks
A signal is worth reading only if it moves with the thing it claims to measure. In 2016 a team led by Bart de Langhe put online ratings to that test in the Journal of Consumer Research, across 1,272 products in 120 categories where quality can be ranked at all.
| What the average rating was tested against | What they found |
|---|---|
| Consumer Reports quality scores | No convergence |
| Resale prices in the used-product market | No prediction |
| Price and premium brands, quality held constant | Ratings run higher |
| The sample behind the average | Often too small to inform |
None of those four is a small shortfall. The authors' term for what buyers bring to the number is an "illusion of validity". Their scope limit matters as much: the benchmark is one expert panel, and it says nothing about restaurants or hotels, where quality is partly a matter of taste.
The second half of the paper explains why the gap survives. Asked to judge quality and say what they would buy, people weight the average heavily against other cues, including price and the number of ratings, and do not moderate their reliance on it when the sample behind it is thin.
A 2017 study led by Derek Powell in Psychological Science found the neglect running the other way. The authors modelled how a sensible observer should read review scores, using an empirical prior drawn from millions of reviews posted on Amazon.com, then set its predictions against what people chose. Under some conditions participants preferred a heavily reviewed product with a higher average to a lightly reviewed one the model judged likely to be better. Volume was read as quality, and the authors concluded that participants had failed to make meaningful statistical inferences.
An early study of online word of mouth, by Judith Chevalier and Dina Mayzlin in 2006, compared book reviews at two retailers and found them "overwhelmingly positive" at both. A scale almost everyone scores near the top of cannot separate much.
Ask what the money figure rests on
The case for online ratings was never that they are accurate. It is that they work, and the strongest version deserves stating properly. In 2012 Michael Anderson and Jeremy Magruder, in The Economic Journal, exploited the rounding in Yelp.com's displayed average, comparing restaurants landing just above a rounding threshold with those just below. An extra half-star caused restaurants to sell out 19 percentage points more often, an increase of about half on the base rate, with larger effects where other information was scarce.
That last clause is the 1927 law reappearing as an interaction term. The outcome, though, is reservation availability on one booking platform, not revenue and not repeat custom.
The revenue figure everyone quotes comes from elsewhere, and its status is worth saying out loud. Michael Luca's Yelp paper reports that a one-star increase in rating leads to a 5 to 9 per cent rise in restaurant revenue, driven by independent restaurants with no effect on those with chain affiliation. It has never been peer reviewed. It was posted as a Harvard Business School working paper, dated 2011 and revised 2016, and that is still what it is. The most-cited evidence that online ratings move money is unpublished, and no equivalent peer-reviewed revenue result exists for any other category.
Watch who gains and who stays silent
A number that moves money is a number worth producing. In 2016 Michael Luca and Georgios Zervas, in Management Science, took the restaurant reviews Yelp's filtering algorithm had flagged as suspicious and used them as a proxy for fraud, alongside businesses caught soliciting fake reviews in a sting the platform ran. Roughly 16 per cent of restaurant reviews were filtered, those reviews ran more extreme in either direction, and their share grew over time.
The pattern in who does it is the useful part. Fraud was likelier where a reputation was weak, meaning few reviews or recent bad ones, and less likely among chain restaurants, which gain less from reviews in the first place. Where competition increased, restaurants became more likely to receive unfavourable fake reviews.
The cleanest demonstration identifies no individual fake review at all. In 2014 Dina Mayzlin and colleagues, in the American Economic Review, compared the same hotels across two platforms. Expedia.com accepted a review only from someone who had booked through the site; TripAdvisor.com accepted anyone. Hotels with most to gain from promotional reviewing had more positive reviews on the open platform relative to the verified one, and their neighbours more negative ones.
A 2021 field experiment led by Andrey Fradkin on Airbnb changed one rule: feedback stayed hidden until both guest and host had submitted theirs, instead of appearing on submission. More people reviewed, retaliation and reciprocation fell, and ratings came down as a result. The stays did not change. The rule did.
Chris Nosko and Steven Tadelis built a better number instead. From transaction-level eBay data they constructed a measure of seller quality using the share of transactions generating positive feedback, counting the transactions where buyers left no feedback at all. It predicted poor outcomes and later purchasing even after the visible reputation scores were accounted for. When the platform put the measure into its search ranking in a field experiment, buyers shifted towards higher-quality sellers and bought more afterwards, with no fall in conversion at the time. The silence is the information, and the public star rating throws it away.
The silence is the information, and the public star rating throws it away.
What this evidence cannot settle
Every outcome above is a sale, a booking, a revenue line or a later purchase. None of it measures whether buyers ended up better off. We looked for a study showing that a rating system improved consumer welfare or decision quality rather than shifting demand, and found none.
Two other limits. None of these papers gave a participant count we could verify, which is why no sample sizes appear above. The eBay paper circulated as a working paper from 2015 and reached a journal in 2026, and it records an eBay Research Labs affiliation, a data-access fact worth knowing when reading it.
Certification can work, with care. A field experiment led by Jens Hainmueller in a US grocery chain lifted sales of the two most popular coffees by almost 10 per cent under a Fair Trade label compared with a meaningless placebo label, though sales of the cheaper coffee fell about 30 per cent when its price rose 9 per cent. A label has to carry content, not just adhesive.
Where we come down is that the 1927 diagnosis survives and the 1927 prescription does not. Published comparative testing was the right cure for a buyer who could no longer judge. Online ratings are not that. They are an average of other buyers who could not judge either, produced by a mechanism carrying incentives of its own.
An average of other buyers who could not judge either, produced by a mechanism carrying incentives of its own.
### Four checks before you trust a score
- How many ratings is this average built on, and would you accept the same opinion from that many strangers in a shop?
- Could you have judged this quality yourself after a month of use, or would it take a laboratory?
- Does the platform require proof of purchase before a review can be posted?
- Does this seller have more to gain from a flattering review than most, meaning few reviews, a recent bad run, or a crowded market?
None of that restores the competence Chase and Schlink said we had lost, and it is a good deal more use than the number printed above the reviews.
Common questions
Can you trust online ratings?
Not for quality you cannot check yourself. In 2016 a team led by Bart de Langhe examined 1,272 products in 120 categories and found average user ratings did not converge with Consumer Reports scores, did not predict resale prices, and ran higher for premium brands once quality scores were controlled for. The authors call buyers' trust in that average an illusion of validity.
Do online ratings actually change what people buy?
Yes, demonstrably. Michael Anderson and Jeremy Magruder found in 2012 that an extra half-star on Yelp caused restaurants to sell out 19 percentage points more often, an increase of about half. The much-quoted finding that a one-star rise lifts revenue 5 to 9 per cent comes from a Harvard Business School working paper that has never been peer reviewed.
How many online reviews are fake?
Nobody knows, and the honest figure is narrower than the headline. Michael Luca and Georgios Zervas reported in 2016 that roughly 16 per cent of restaurant reviews on Yelp were filtered by the platform's own algorithm, which they used as a proxy for fraud rather than a count of it. Fraud concentrated where reputations were weakest.
What should I look at instead of the star average?
Look at the count, the posting rules and the text. A thin sample carries little information, and people reliably fail to discount it. Platforms requiring proof of purchase are harder to game than open ones. A measure built from the share of transactions going silently well outperformed the visible star rating in eBay's own field experiment.
Sources
Chase, S., & Schlink, F. J. (1927). Your Money's Worth: A Study in the Waste of the Consumer's Dollar. The Macmillan Company, New York, pp. 1 to 267. Read in full from the Internet Archive OCR of the first edition, item yourmoneysworths00chas; vault hub audited 20 September 2026.
Nelson, P. (1970). 'Information and Consumer Behavior.' Journal of Political Economy, 78(2), 311-329. doi:10.1086/259630. Bibliographic record read at OpenAlex and cross-checked at Crossref; no legitimately reachable full text found, so metadata only and nothing quoted.
Darby, M. R., & Karni, E. (1973). 'Free Competition and the Optimal Amount of Fraud.' The Journal of Law and Economics, 16(1), 67-88. doi:10.1086/466756. Item record read at Crossref and OpenAlex, author order confirmed; full text not available in ChicagoUnbound, so the taxonomy is named and not quoted.
de Langhe, B., Fernbach, P. M., & Lichtenstein, D. R. (2016). 'Navigating by the Stars: Investigating the Actual and Perceived Validity of Online User Ratings.' Journal of Consumer Research, 42(6), 817-833. doi:10.1093/jcr/ucv047. Abstract read in full, reconstructed from the OpenAlex inverted index; published online 2015, print issue April 2016.
Powell, D., Yu, J., DeWolf, M., & Holyoak, K. J. (2017). 'The Love of Large Numbers: A Popularity Bias in Consumer Choice.' Psychological Science, 28(10), 1432-1442. doi:10.1177/0956797617711291. Abstract read in full, reconstructed from the OpenAlex inverted index; no sample sizes verified.
Chevalier, J. A., & Mayzlin, D. (2006). 'The Effect of Word of Mouth on Sales: Online Book Reviews.' Journal of Marketing Research, 43(3), 345-354. doi:10.1509/jmkr.43.3.345. Abstract read in full, reconstructed from the OpenAlex inverted index.
Anderson, M., & Magruder, J. (2012). 'Learning from the Crowd: Regression Discontinuity Estimates of the Effects of an Online Review Database.' The Economic Journal, 122(563), 957-989. doi:10.1111/j.1468-0297.2012.02512.x. Abstract read in full, reconstructed from the OpenAlex inverted index; sample size and city coverage not in the abstract.
Luca, M. (2016). 'Reviews, Reputation, and Revenue: The Case of Yelp.com.' Harvard Business School Working Paper 12-016, 41 pages, dated September 2011 and revised March 2016. doi:10.2139/ssrn.1928601. Abstract read in full at the EconPapers RePEc record RePEc:hbs:wpaper:12-016; never published in a peer-reviewed journal, and the HBS PDF returns 403.
Luca, M., & Zervas, G. (2016). 'Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud.' Management Science, 62(12), 3412-3427. doi:10.1287/mnsc.2015.2304. Abstract read in full, reconstructed from the OpenAlex inverted index; the fraud measure is Yelp's own filtering flag used as a proxy.
Mayzlin, D., Dover, Y., & Chevalier, J. (2014). 'Promotional Reviews: An Empirical Investigation of Online Review Manipulation.' American Economic Review, 104(8), 2421-2455. doi:10.1257/aer.104.8.2421. Abstract read in full, reconstructed from the OpenAlex inverted index; author order checked at Crossref and OpenAlex.
Fradkin, A., Grewal, E., & Holtz, D. (2021). 'Reciprocity and Unveiling in Two-Sided Reputation Systems: Evidence from an Experiment on Airbnb.' Marketing Science, 40(6), 1013-1029. doi:10.1287/mksc.2021.1311. Abstract read in full, reconstructed from the OpenAlex inverted index; the size of the ratings change is not reported in the abstract and is not quantified here.
Nosko, C., & Tadelis, S. (2026). 'The limits of reputation in platform markets: An empirical analysis and field experiment.' Quantitative Marketing and Economics, 24(1). doi:10.1007/s11129-026-09313-1, published online 4 August 2026. Earlier circulated as NBER Working Paper 20830, January 2015, doi:10.3386/w20830. Both abstracts read in full; the NBER record carries an eBay Research Labs affiliation, recorded as a data-access disclosure.
Hainmueller, J., Hiscox, M. J., & Sequeira, S. (2015). 'Consumer Demand for Fair Trade: Evidence from a Multistore Field Experiment.' The Review of Economics and Statistics, 97(2), 242-256. doi:10.1162/REST_a_00467. Abstract read in full, reconstructed from the OpenAlex inverted index.
Dranove, D., & Jin, G. Z. (2010). 'Quality Disclosure and Certification: Theory and Practice.' Journal of Economic Literature, 48(4), 935-963. doi:10.1257/jel.48.4.935. Abstract read in full; a literature review, cited for framing and not as a result.