What Engagement Metrics Really Measure
Engagement metrics count clicks, minutes and returns. Studies of search logs, paid ads and Google's own experiments show what those counts leave out, and why a rising number is not the same as a better product.
Answer to the question
What do engagement metrics actually measure?
Engagement metrics measure observable behaviour, such as clicks, time on page and return visits, not whether the person got what they wanted. Studies of search logs, eBay ads and Google experiments show the same count can mean success, waste or learning to ignore the product.
Key takeaways
- Engagement metrics record behaviour a product can count (clicks, minutes, returns) and strip out the user's reasons, as Mounia Lalmas, Heather O'Brien and Elad Yom-Tov describe web analytics in Measuring User Engagement (2014).
- James Williams (2018) says analytics built to measure advertising made clicks, impressions and time on site the default metrics of websites. It is a philosophical argument, not a measurement.
- Jane Li, Scott Huffman and Akihito Tokuda (2009) showed that many abandoned searches were answered by the results page, so a search with no click is not always a failure.
- Tom Blake, Chris Nosko and Steven Tadelis (2015) found that eBay's paid search returns, measured by experiment, were a fraction of conventional estimates, and brand keyword ads showed no measurable short-term benefit.
- Henning Hohnhold, Deirdre O'Brien and Diane Tang (2015) found that short-term test results do not always predict long-term effects, because users learn from the quality of the ads they have seen.
Engagement metrics count what a product can observe: clicks, minutes on a page, visits per week. They do not record whether the person was served. A click shows that something happened, and a minute shows that the person was still there. Neither says why, and the gap between the count and the reason is where a measurement turns into a bet. The count came out of advertising, and in published experiments it has scored a better answer as a worse result and pointed away from the long-term outcome.
Counted for advertisers first
James Williams, who says in chapter 2 of Stand Out of Our Light (2018) that he once worked at Google, gives the history in chapter 5. Analytics platforms built to measure advertising, he writes, "helped establish the 'engagement' metrics of advertising (e.g. number of clicks, impressions, or time on site) as default operational metrics for websites themselves." The book is a philosophical essay, and its preface calls it "more exploration than argument", so the history is a claim to weigh, not a measurement. It does fit the vocabulary. Clicks, impressions and time on site were units of ad sales before engagement metrics made them units of product success, which is the inversion examined in [Native Advertising and the Inversion](https://thewhyweproject.com/articles/native-advertising-and-the-inversion).
The people who measure user engagement for a living describe a wider field. Mounia Lalmas, Heather O'Brien and Elad Yom-Tov, in Measuring User Engagement (2014), treat engagement as the quality of interacting with an online application and the desire to keep using it. They survey four families of method: self-report questionnaires, observation, neuro-physiological signals and web analytics.
Web analytics records behaviour with the reasons removed.
Each has a trade-off. Small laboratory studies are rich but hard to generalise. Web analytics is the family that fills a dashboard, and in the book's own account it is powerful at scale and strips away the user's motivation and context. The authors treat engagement as a multifaceted phenomenon that no single count captures. A dashboard of engagement metrics treats one count as the whole of it.
So engagement metrics are web analytics, and web analytics records behaviour with the reasons removed. A reader of a dashboard sees the behaviour and supplies the reasons from hope. The studies below show what happens when the reasons turn out to be different from the hope.
What a click can hide
A search that ends without a click looks like a failure in a log. A 2009 paper by Jane Li, Scott Huffman and Akihito Tokuda gave the exception a name, good abandonment: a query dropped without a click or a rewrite because the results page already answered the need. They analysed abandoned queries on PC and mobile in three locales and found that good abandonment made up a large share of them, and was more common on mobile than on PC. Their conclusion was that abandonment should not be treated uniformly as a negative signal. A team that rewarded clicks would score the better answer as the worse result, and it would work to bring the click back.
Clicks mislead from the other side too, because a click is also the unit in which advertising is sold and judged. Tom Blake, Chris Nosko and Steven Tadelis ran field experiments on paid search at eBay, published in Econometrica in 2015. Clicks and purchases are correlated, and the returns measured by experiment came out as a fraction of the conventional non-experimental estimates. Ads on brand keywords showed no measurable short-term benefit, which means the clicks they collected were a poor guide to the sales they caused. For other keywords, new and infrequent users responded, but frequent users, whose purchases the ads did not influence, accounted for most of the spending, so the average return was negative.
Engagement metrics do not fail evenly across their forms, so the table sets out what each common count leaves out.
A team judged on this week's clicks cannot see a user learning to look away.
The table is our reading, not a finding from either study.
| Metric | What it records | What it leaves out |
|---|---|---|
| Click | An action on an item | Whether the click was wanted, or caused the purchase |
| Time on page | The person's presence | Whether the time was attention or delay |
| Query abandonment | No further action | Whether the need was met or missed |
| Return visits | Repetition | Whether the return was chosen or prompted |
Users learn after the test ends
The strongest case against reading a dashboard as a verdict comes from inside the industry. Henning Hohnhold, Deirdre O'Brien and Diane Tang, writing about Google search ads at the 2015 Knowledge Discovery and Data Mining conference, argued that short-term A/B test results do not always predict long-term effects, because users change their behaviour after a product launches. They measured that learning directly, and named two directions of it. Users can become less likely to click on ads, which the paper calls ads blindness, or more likely, ads sightedness. The change was driven by the quality of the ads previously seen or clicked, including relevance and landing page quality.
The paper reports two applications, a change to the ads auction that raised the weight on ad quality and a 50 per cent cut in ad load on mobile search. The authors summarise what they learned in one line: "user satisfaction is paramount."
Our reading is narrower than the paper's. For engagement metrics, the lesson is about timing. A team judged on this week's clicks cannot see a user learning to look away over the next few months, because the learning shows up in the count only after the decision has been made. The authors built a model that uses short-term metrics to predict the long-term outcome, which concedes that the raw short-term number is not the answer. It does not concede that the metric is useless. It treats the count as an input to a prediction about long-term outcomes.
Questions for any dashboard
The studies agree on a practical point about engagement metrics, whatever platform produces them. Before engagement metrics are trusted as a measure of value, three questions apply: what exactly was counted, what else would produce the same count, and what a satisfied person would look like in it. A search that was answered, a purchase that would have happened anyway and a user who is quietly learning to ignore the product all produce counts that look like their opposites. Each is a question the dashboard cannot answer from its own numbers.
The same arithmetic runs through [What You Actually Pay for Free Apps](https://thewhyweproject.com/articles/what-you-actually-pay-for-free-apps) and [Why You Cannot Stop Scrolling](https://thewhyweproject.com/articles/why-you-cannot-stop-scrolling), where time spent is read as a sign of value when it can equally be a sign of capture. A team that wants its counts to carry meaning has to check its counts against cause and intent, and keep checking for longer than the experiment runs; the Google authors went the same way when they treated the early count as a forecast of the result and not as the result itself.
Common questions
What do engagement metrics actually measure?
Observable behaviour such as clicks, time on page and return visits. They do not measure whether the person got what they wanted, which needs self-report, causal tests or long-run follow-up.
Where did engagement metrics come from?
According to James Williams, they were advertising measures, clicks, impressions and time on site, that analytics platforms then carried into the design of websites themselves.
Can a drop in clicks be good news?
Sometimes. Google researchers found that many abandoned searches were answered by the results page, a pattern they called good abandonment, so a missing click can be a success.
Do short-term results predict long-term effects?
Not reliably. A 2015 Google paper found users change behaviour after launch, and the quality of ads they had seen shaped whether they clicked later.
Sources
Williams, J. (2018). Stand Out of Our Light: Freedom and Resistance in the Attention Economy. Cambridge: Cambridge University Press. Open access under CC BY-NC 4.0. doi:10.1017/9781108453004
Lalmas, M., O'Brien, H., & Yom-Tov, E. (2014). Measuring User Engagement. Synthesis Lectures on Information Concepts, Retrieval, and Services. San Rafael, CA: Morgan & Claypool. Publisher description read. doi:10.1007/978-3-031-02289-0 (Springer edition, dated 2015)
Li, J., Huffman, S., & Tokuda, A. (2009). "Good abandonment in mobile and PC internet search." Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 43-50. doi:10.1145/1571941.1571951. research.google/pubs/pub35486/
Blake, T., Nosko, C., & Tadelis, S. (2015). "Consumer heterogeneity and paid search effectiveness: A large-scale field experiment." Econometrica, 83(1), 155-174. doi:10.3982/ECTA12423. NBER Working Paper 20171
Hohnhold, H., O'Brien, D., & Tang, D. (2015). "Focus on the long-term: It's better for users and business." Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Sydney, 1849-1858. doi:10.1145/2783258.2788583 (the ACM record titles it "Focusing on the Long-term: It's Good for Users and Business"). research.google/pubs/pub43887/