How Much Recommendations Really Sell
Recommendation engines are sold as mind readers, yet the published field tests put their average sales lift at one to five per cent. They also shape where you look, what everyone else ends up buying, and how watched you feel.
Key takeaways
- A 2019 review by Dietmar Jannach and Michael Jugovac found that where sales were measured directly, recommendations raised them by one to five per cent on average.
- In one news-site test reported in that review, moving and enlarging the suggestion box raised clicks by about 100 per cent, against about 35 per cent for a better algorithm.
- A 2019 field experiment covering 1,138,238 users found that recommendations helped each shopper explore while concentrating overall sales on popular products.
- Elizabeth Aguirre's 2015 study found personalised ads raised click intentions when data was collected openly, and made no difference when it was collected covertly.
Published field tests put the average sales lift from recommendations at between one and five per cent, according to a 2019 review by Dietmar Jannach and Michael Jugovac. That is serious money for a large retailer, and it also means that most baskets would have looked much the same without the suggestions.
Count what the recommendations are compared against
Jannach, at the University of Klagenfurt, and Jugovac, at TU Dortmund, set out to gather the published tests of what recommender systems do for a business, as opposed to how accurate they are in a laboratory. Their review in ACM Transactions on Management Information Systems found that where sales were measured directly, the typical increase reported was one to five per cent. Larger figures exist, such as a rise of more than 26 per cent for one category in an online grocery store, but they describe a slice of a shop rather than the whole of it.
The size of any reported gain depends on what the new system replaced. Sometimes the comparison was a site with no suggestions at all. Sometimes it was a simple list of bestsellers that treated every visitor alike. Only in a few cases were two sophisticated methods tested against each other. A system that beats nothing will always look better than one that beats a decent bestseller list, so a large number means little until you know what it was measured against.
The review is also clear about where the most repeated figures come from. The best-known claims about how much of Netflix's viewing and how much of Amazon's sales come from recommendations trace back, respectively, to a company blog post and to a chief executive's remark reported in online media. Neither comes with a method, so neither belongs in a sentence that claims to measure anything.
Netflix did later publish its own account. In a 2015 paper in the same journal, Carlos Gomez-Uribe and Neil Hunt, both writing from Netflix, said their recommender influences choice for about 80 per cent of hours streamed, and that personalisation and recommendations together save the company more than a billion dollars a year by reducing cancellations. The paper introduces the billion-dollar figure with "We think" and gives no calculation behind it. That share counts hours that began from a recommended row, including programmes a subscriber might have found anyway.
Look at where the box sits
The most useful single result in the review comes from a news website. A more sophisticated recommendation method raised the click-through rate, the share of visitors who click a suggestion, by about 35 per cent. The same team then left the method alone and changed only the position and size of the box the suggestions appeared in. Clicks rose by about 100 per cent.
| What changed | What happened | Where it was reported |
|---|---|---|
| A better news recommendation method | About 35 per cent more clicks | Jannach and Jugovac, 2019 review |
| Same method, box moved and enlarged | About 100 per cent more clicks | Jannach and Jugovac, 2019 review |
| Recommender added, across field tests | Sales up one to five per cent on average | Jannach and Jugovac, 2019 review |
The furniture moved the number almost three times as far as the intelligence did. For a shopper, that changes what a suggestion box is evidence of. A product you clicked because it sat large and high on the page tells you about the page, and only a little about whether the algorithm understood you.
The furniture moved the number almost three times as far as the intelligence did.
Clicks are also a poor guide to value. The review states that click-through rates, outside pay-per-click advertising, are in most cases not a true measure of business value, and it reports live trials in which a plain popularity list or even random suggestions did as well as clever methods on clicks. A good habit, then, is to treat a click on a suggestion as attention rather than as proof that the suggestion was right for you.
Assume the bestsellers will get a little bigger
What recommendations do to the range of things people buy is the question where the evidence is most interesting, because it pulls in two directions. In 2009 Daniel Fleder and Kartik Hosanagar used models and simulations in Management Science to show that some well-known recommenders can push each person towards new products while pushing everyone towards the same ones. Individual variety can rise while the market as a whole concentrates.
A decade later Dokyun Lee and Hosanagar tested this in the field. Their randomised experiment on a large North American retailer's website, reported in Information Systems Research in 2019, covered 1,138,238 users and 82,290 products. Recommendations of the "customers who bought this also bought" kind lowered overall sales diversity compared with no recommendations. Individual shoppers explored more, but similar shoppers explored the same items, and while niche products gained sales, popular ones gained more. According to Carnegie Mellon's summary, the bottom 80 per cent of products took 37 per cent of sales without a recommender and 27 per cent with one.
The other side comes from Netflix. Gomez-Uribe and Hunt report that their recommender spreads viewing across many more titles, and much more evenly, than an unpersonalised system would, by a factor of roughly four at one point on their own measure. Netflix withheld the values on its chart's axes for competitive reasons, so the claim cannot be checked, but it is consistent with Fleder and Hosanagar's own conclusion that design choices decide which way the effect runs.
A subscription service that pays for its catalogue in advance has a reason to spread viewing out. A retailer selling stock does not. Our lean is towards the field experiment for ordinary shopping sites, because it was randomised, independent and large, and because it describes the kind of box most shoppers actually see. Expect the suggestions to send you, slightly more than you would otherwise go, to what everyone else is already buying.
Ask the shop how it knows you
The Future of Shopping by Jörg Snoeck and Pauline Neerman, a 2022 trade forecast of where retail is heading, comes at recommendations from the retailer's side. Its third chapter, on technology, frames Amazon, Alibaba and Tencent as companies racing to own a customer's data from birth to death. Its sixth chapter, on retail marketing after third-party cookies, cites a 2018 Harvard Business Review article on the personalisation paradox, the fine line between a suggestion that delights and one that unsettles. The book does not put a number on what recommendation adds to sales, which is the gap the review above fills.
The research on that fine line supports the book's instinct. Elizabeth Aguirre and colleagues, in the Journal of Retailing in 2015, ran Facebook ads for a financial services brand and a dog food brand, both sold in the Netherlands. For both, an ad personalised by age, gender and location drew fewer clicks than a moderately personalised one. In follow-up experiments, more personalised ads raised people's intention to click when the firm had collected their data openly, and made no difference when it had collected it covertly. The authors trace the difference to a feeling of vulnerability, which signals of trust could offset.
An earlier study by Tiffany Barnett White and colleagues in Marketing Letters in 2008 found the same pattern in email. Highly personalised offers provoked reactance, a push back against a perceived attempt to steer, when the firm did not explain why the offer fitted the person. The effect was strongest among people who saw little value in the service.
Prefer the shop that explains its suggestions
On this evidence, recommendations are a modest convenience that gets described as a mind-reading machine. The best estimate of what they add to a shop's sales is one to five per cent. The presentation of the box can matter more than the algorithm behind it, and on a typical retail site the suggestions nudge the whole market slightly towards what is already popular, even as they help each shopper find a few things they would have missed.
Recommendations are a modest convenience that gets described as a mind-reading machine.
An average can hide the cases that matter. In a catalogue of tens of thousands of products, a shopper with no suggestions at all is not a realistic comparison, and a category that rises by a quarter is a real effect for the people who shop in it. We accept that, and it is why we would describe recommendations as useful for finding things in large catalogues rather than as useless.
What we would change is what the shopper does with them. Treat a suggestion as a pointer to what similar people bought, and give more trust to a shop that says why it is showing you something, since the work by Aguirre and by White suggests an explained suggestion is more welcome. A shop that will not say how it knows what it knows about you is asking for a kind of trust that an average effect of one to five per cent has not yet earned.
Common questions
How much do recommendations increase sales?
In their 2019 review of published field tests, Dietmar Jannach and Michael Jugovac found that where sales were measured directly, recommendations raised them by between one and five per cent on average. Larger figures usually come from comparisons with no recommender at all, from single product categories, or from company statements with no stated method.
Do recommendations make everyone buy the same things?
On a typical retail site, slightly. In a 2019 field experiment covering 1,138,238 users, Dokyun Lee and Kartik Hosanagar found that collaborative filtering lowered overall sales diversity, even though individual shoppers explored more products. Netflix reports the opposite for its own system, but has not published the figures needed to check it.
Why do personalised ads sometimes feel creepy?
A 2015 study by Elizabeth Aguirre and colleagues found that more personalised ads raised click intentions only when the data had been collected openly. Covert collection left people feeling vulnerable. A 2008 study by Tiffany Barnett White found reactance when firms did not explain why an offer fitted the recipient.
Sources
Jannach, D., & Jugovac, M. (2019). “Measuring the business value of recommender systems.” ACM Transactions on Management Information Systems, 10(4), 1-23. doi:10.1145/3370082. Preprint arXiv:1908.08328. Full text read (preprint): sections 2 to 4; sales increases of one to five per cent on average where business effects are measured directly (sections 3.2 and 4.1); over 26 per cent for one category of an online grocery store; baselines of no recommender, a popularity list or, rarely, another sophisticated method; a news-site test (Garcin et al.) with about 35 per cent higher click-through from a new method and about 100 per cent from changing the widget's position and size; a 17 per cent revenue fall at an online bookshop when recommenders were removed (the review gives one month in section 2.2.3 and a week in section 3.2); click-through 'in most cases not a true measure of business value'; Netflix figure traced to a 2012 blog post and Amazon figure to a 2006 chief executive statement in online media.
Gomez-Uribe, C. A., & Hunt, N. (2015). “The Netflix recommender system: algorithms, business value, and innovation.” ACM Transactions on Management Information Systems, 6(4), Article 13. doi:10.1145/2843948. Issue dated December 2015, often cited as 2016. Full text read: recommender 'influences choice for about 80% of hours streamed'; 'We think the combined effect of personalization and recommendations save us more than $1B per year'; churn reduced 'by several percentage points'; effective catalogue size roughly four times that of an unpersonalised ranking at the median play rank, with axis values withheld for competitive reasons.
Fleder, D., & Hosanagar, K. (2009). “Blockbuster culture’s next rise or fall: the impact of recommender systems on sales diversity.” Management Science, 55(5), 697-712. doi:10.1287/mnsc.1080.0974. Abstract read: analytical model and simulation; recommenders can push each person to new products but often push users toward the same products; individual diversity can rise while aggregate diversity falls; design choices shape the outcome.
Lee, D., & Hosanagar, K. (2019). “How do recommender systems affect sales diversity? A cross-category investigation via randomized field experiment.” Information Systems Research, 30(1), 239-259. doi:10.1287/isre.2018.0800. Abstract read: randomised field experiment on a major North American retailer's website, 82,290 products and 1,138,238 users; collaborative filtering associated with lower aggregate sales diversity than no recommendations; individual diversity rises; niche items gain in absolute terms but lose share. Bottom 80 per cent of products at 37 per cent of sales in the control group against 27 per cent with a recommender, from Carnegie Mellon University's news release of November 2019 (read).
Aguirre, E., Mahr, D., Grewal, D., de Ruyter, K., & Wetzels, M. (2015). “Unraveling the personalization paradox: the effect of information collection and trust-building strategies on online advertisement effectiveness.” Journal of Retailing, 91(1), 34-49. doi:10.1016/j.jretai.2014.09.005. Full text read (City, University of London accepted manuscript): Facebook field study for a financial services brand and a dog food brand sold in the Netherlands, highly personalised ads (age, gender, location) drew lower click-through than moderately personalised ones for both; in experiments, more personalised ads raised click-through intentions under overt collection and made no difference under covert collection; effect attributed to feelings of vulnerability, offset by trust-building strategies.
White, T. B., Zahay, D. L., Thorbjørnsen, H., & Shavitt, S. (2008). “Getting too personal: reactance to highly personalized email solicitations.” Marketing Letters, 19(1), 39-50. doi:10.1007/s11002-007-9027-9. Abstract read: personalisation reactance to highly personalised messages when the fit between offer and personal characteristics is not explicitly justified; effect primarily among consumers who perceive the service's utility as relatively low.
Snoeck, J., & Neerman, P. (2022). The Future of Shopping: Re-set Re-made Re-tail. LannooCampus. Read in full by this publication, with a reliability audit. Ch. 3, The Future of Technology (p.63), on the platforms' race for data; Ch. 6, The Future of Retail (p.179), citing John, Kim and Barasz (2018), Harvard Business Review, on the personalisation paradox. Page numbers are not readable in our photographs of this book, so the locus is given by chapter and photograph. The book states no effect size for recommendation.