What Does an AB Test Actually Prove
An AB test shows that a change caused a difference in one chosen outcome, among the people enrolled, over the days it ran. A Bing bug, a Facebook comparison and a Google study of ad load show how much of the rest has to be argued.
Answer to the question
What does an AB test actually prove?
It proves that, for the users enrolled and the days it ran, a change caused the measured difference in the chosen outcome, on average. It does not show why people responded, whether the outcome was the right one to watch, or that the effect will last once users adapt.
Key takeaways
- Kohavi and colleagues (2012) report a Bing bug that showed users very poor results: distinct queries per user rose more than 10 per cent and revenue per user more than 30 per cent. The test measured both correctly, and neither was a gain in long-term value.
- A 2019 comparison of 15 Facebook advertising experiments, covering about 500 million user-experiment observations, found that common observational methods usually overstated the effect of ads and sometimes understated it.
- Google researchers (2015) found that short-term test readings do not always predict what happens once users adapt: ad clicks shift with the quality of the ads people saw before.
- Lewis and Rao (2015) ran 25 field experiments on advertising and found a median confidence interval on return on investment more than 100 percentage points wide.
A Bing experiment once showed users very poor search results by mistake. Distinct queries per user rose by more than a tenth and revenue per user by more than a third, as Ron Kohavi and his colleagues at Microsoft reported in 2012.
The episode appears in Adam Kucharski's Proof (2025), and it shows what an AB test certifies. The test measured those increases correctly, and they were not good news: people who cannot find an answer search again and click more ads. An AB test proves that a change caused a difference in one chosen outcome, among the people enrolled, over the days it ran.
Random Allocation Isolates the Change From Everything Around It
The mechanism is random allocation. Visitors are sent by chance to the current page or the changed one, so the two groups differ in nothing systematic except the change, and any gap in outcomes beyond what chance produces can be put down to it. The lineage is the statistical tradition Kucharski follows through his chapter on testing, from Ronald Fisher's The Design of Experiments (1935) to the randomised trial. He notes that such a trial estimates an average effect across a group, which is a weaker thing than a verdict on any one person.
An AB test proves that a change caused a difference in one chosen outcome, among the people enrolled, over the days it ran.
The value of the design shows when it is removed. Researchers at Northwestern University and Facebook took 15 US advertising experiments, covering about 500 million user-experiment observations and 1.6 billion ad impressions, and asked whether common observational methods, applied to the same data, could recover the effect each experiment had measured. They often could not. The observational estimates usually overstated what the ads had done and sometimes understated it. [What Scent Does in a Shop](https://thewhyweproject.com/articles/what-scent-does-in-a-shop) shows the same logic at shop scale, in a randomised trial run in a bookstore.
The same restriction governs what an AB test can carry to other settings. Kucharski's account of randomised trials stresses that they cannot always be run, and that their results do not always transfer. An AB test run on one site's visitors over one fortnight supports a causal claim about those visitors in that fortnight. A launch decision adds the assumption that later visitors will behave in the same way.
The Metric You Chose Is Part of the Result
Degraded organic results, Kohavi and colleagues explain in their 2012 paper, push users to issue more queries and click more ads. They liken the gain to a shop raising its prices, where customers go along for a while and then leave, so lifetime value falls. Distinct queries per user is a figure a search engine should want to reduce when people finish their tasks. The authors recommend an overall evaluation criterion built to stand in for long-term value, and they constrain revenue per user with engagement measures such as sessions per user. [What Engagement Metrics Really Measure](https://thewhyweproject.com/articles/what-engagement-metrics-really-measure) covers the same trap outside search.
Google researchers measured the gap directly. A 2015 paper presented at the KDD conference developed a method for measuring long-term user learning and applied it to ads on Google Search. A test readily measures short-term effects, the authors write, but those do not always predict what happens once users adapt. Clicks on ads shift with the quality of the ads people saw before. One application looked at a 50 per cent cut in ad load on Google's mobile search, and the authors argue that attention to user satisfaction can support lower ad loads without hurting long-term business results.
When the Test Is Allowed to End
An A/A test, in which both groups see the identical page, can look like a result early on. In one that Kohavi and colleagues show, the cumulative difference appeared to trend strongly and then settled near zero as data accumulated. A new feature can also draw repeated tries and then lose its appeal. They describe opening Hotmail in a new window as a strong novelty effect that settled into a positive long-term result.
Standard p-values assume that the sample size was chosen before the data arrived. Someone who checks every day and stops when the result first looks significant has broken that assumption, and a preprint on "always valid" inference says that inferences from samples chosen by continuous monitoring are wholly unreliable. Its authors propose p-values and confidence intervals that stay valid at any stopping point, and report that the method was built into a large commercial AB test platform and used on hundreds of thousands of experiments.
The Microsoft team's own practice is to run a test for at least a week to capture day-of-week effects, and longer where novelty or primacy effects are a concern. For a metric such as sessions per user, they add, power comes mainly from more users per day and hardly at all from more days, because the metric's variability rises as the test runs.
Most Ideas Lose, and the Effects Are Small
About a third of the ideas tested at Microsoft were positive and statistically significant, a third were flat and a third were negative and significant, according to slides Kohavi presented in 2012. The same slide cites Jim Manzi's book Uncontrolled for a figure on Google, approximately 12,000 randomised experiments in 2009 with about 10 per cent leading to business changes. That figure is reported by Kohavi, second-hand, and the primary source was not checked here. The slide's own verdict is that intuition is poor: 60 to 90 per cent of ideas do not improve the metric they were designed to improve, depending on the domain, and a randomised test is how a team finds out which ones do.
A winner can still be very large. A change to how Bing displayed ad headlines sat as a low-priority idea for more than six months, took an engineer a few days to build, and raised revenue by 12 per cent, a gain that a Harvard Business Review article co-written by Kohavi put at more than 100 million dollars a year in the United States. The revenue jump was high enough to trigger a too-good-to-be-true alert, and an analysis confirmed it was real.
Advertising is the hard case for measurement. Two researchers ran 25 field experiments with large US retailers and brokerages, most reaching millions of customers. Individual sales vary so much against the cost of advertising per person that the median confidence interval on return on investment was over 100 percentage points wide, and an informative experiment could need more than 10 million person-weeks. A firm that expects an AB test to show whether its advertising pays should work out the width of the interval it can expect before the test starts, because the width of that interval depends largely on how much sales vary from one customer to the next and on how many customers can be reached.
Common questions
What does an AB test actually prove?
That, for the users enrolled and the period it ran, the change caused the measured difference in the chosen outcome, on average. It does not show why people responded, whether the outcome was the right one to watch, or that the effect will last.
How long should an AB test run?
Kohavi and colleagues run tests for at least a week to capture day-of-week effects, and longer where novelty or primacy effects are a concern. The stopping rule should be fixed before the test starts, because stopping at the first significant reading makes the p-value unreliable.
Can observational data replace an AB test?
Often not. In the Facebook comparison of 15 advertising experiments, common observational methods usually overstated the effect of the ads.
Sources
Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & Xu, Y. (2012). 'Trustworthy online controlled experiments: five puzzling outcomes explained.' Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '12). doi:10.1145/2339530.2339653
Kohavi, R., & Thomke, S. (2017). 'The surprising power of online experiments.' Harvard Business Review, 95(5). https://hbr.org/2017/09/the-surprising-power-of-online-experiments
Kohavi, R. (2012). 'Slides presented at the ACM Conference on Recommender Systems, 9 September 2012, slide 17.' https://ai.stanford.edu/~ronnyk/2012-09ACMRecSysNR.pdf
Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). 'A comparison of approaches to advertising measurement: evidence from big field experiments at Facebook.' Marketing Science. doi:10.1287/mksc.2018.1135
Hohnhold, H., O'Brien, D., & Tang, D. (2015). "Focus on the Long-Term: It's better for Users and Business." Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Sydney. https://research.google/pubs/pub43887/
Johari, R., Pekelis, L., & Walsh, D. J. (2015). 'Always valid inference: bringing sequential analysis to A/B testing.' arXiv:1512.04922. https://arxiv.org/abs/1512.04922
Lewis, R. A., & Rao, J. M. (2015). 'The unfavorable economics of measuring the returns to advertising.' Quarterly Journal of Economics, 130(4), 1941-1973. https://ideas.repec.org/a/oup/qjecon/v130y2015i4p1941-1973.html
McGrath, M. C., Aronow, P. M., & Shotwell, V. (2016). 'Chocolate scents and product sales: a randomized controlled trial in a Canadian bookstore and cafe.' SpringerPlus, 5, article 670. doi:10.1186/s40064-016-2303-5
Fisher, R. A. (1935). The Design of Experiments. Edinburgh: Oliver and Boyd.
Kucharski, A. (2025). Proof: The Uncertain Science of Certainty. London: Profile Books, chapter 4.