AB Testing

What it is

In the words of the 2007 KDD paper by Ron Kohavi, Randal Henne and Dan Sommerfield, "in the simplest controlled experiment, often referred to as an A/B test, users are randomly exposed to one of two variants." The existing version is the control and the new one is the treatment. The same paper lists other names for the approach: controlled experiments, randomised experiments, split tests, control and treatment tests, and parallel flights. After enough users have passed through, the difference in the chosen outcome is tested for statistical significance.

The method is a randomised controlled trial moved online. Ronald Fisher's The Design of Experiments (1935) pioneered the application of statistical procedures to the design of scientific experiments, and the 1948 Medical Research Council trial of streptomycin allocated patients by a series of random numbers. An online test applies it to page views, accounts or sessions, where allocation is a line of code and the sample can run to millions. Adam Kucharski, in the chapter of Proof (2025) that follows the history of statistical testing, stresses that a randomised trial estimates an average effect across a group. That is a weaker thing than a statement about any one person, and the same holds for an AB test.

The chosen outcome matters as much as the random split. Kohavi and colleagues call the quantity a test is judged on the overall evaluation criterion, and they say a good one "should not be short-term focused (e.g., clicks)." It should include predictors of long-term goals such as lifetime value and repeat visits. See Overall Evaluation Criterion.

Large technology firms run such tests as a matter of routine. Kohavi's own page records that he joined Microsoft in 2005 and founded its Experimentation Platform team in 2006, after earlier work at Amazon. The 2007 paper describes Amazon's shopping-cart recommendations, proposed by Greg Linden, which won a controlled experiment by a wide margin despite the objection of a senior executive. The authors present the episode as an argument for letting data outweigh the opinion of the highest-paid person in the room, a figure they call the HiPPO.

What a test certifies depends on conditions that the random split does not protect. The outcome can be the wrong one, the run can be too short or stopped too early, the sample can be too small to show a real effect, and the users enrolled may differ from the users the change is meant for. Each of these has a documented case, set out below.

In effect

A bug that looked like a win

In an experiment at Bing, a bug caused very poor search results to be shown to users. Distinct queries per user rose by more than 10 per cent and revenue per user by more than 30 per cent, according to Kohavi, Deng, Frasca, Longbotham, Walker and Xu (2012). The authors explain the rise as the effect of degraded results: users search again and click more ads. They compare it to a shop raising its prices, where customers go along for a while and then leave, and they argue that lifetime value, not revenue per user alone, should guide the choice of criterion. The random split worked and the outcome was wrong.

Fifteen Facebook experiments

Gordon, Zettelmeyer, Bhargava and Chapsky (2019) used 15 US advertising experiments at Facebook, covering about 500 million user-experiment observations and 1.6 billion ad impressions, as a benchmark. They asked whether common observational methods, applied to the same data, could recover the effect each experiment had measured. The methods often failed. They usually overstated the effect of the ads and in some cases understated it. This is the strongest argument for running a randomised test at all.

Chocolate scent in a bookshop

McGrath, Aronow and Shotwell (2016) tested the claim that chocolate scent raises bookshop sales. They ran the test for 31 days in an independent Canadian bookshop with an adjoining cafe, with the scent condition assigned by a coin flip day by day and diffused out of sight. Staff and customers did not know which condition was running. It is an AB test at the scale of one shop, and it shows the design needs no software, only a rule for allocation that no one can bend.

A headline worth a hundred million dollars

A change to how Bing displayed ad headlines sat for more than six months as a low-priority idea. An engineer then built it in a few days, and revenue rose 12 per cent. A Harvard Business Review article co-written by Kohavi put the gain at more than 100 million dollars a year in the United States. The result was high enough to trigger a too-good-to-be-true alert, and an analysis confirmed it was real.

What it does not say

An AB test does not say why users responded. It reports a difference in a number, and the explanation has to come from elsewhere. The Bing bug shows the cost of reading a difference as a verdict.

It does not say that the outcome was the right one to watch. The 2007 paper's own example is an Office Help test in which a five-star rating widget replaced a yes-or-no widget, and ratings fell by an order of magnitude, a result the team did not want.

It does not say the effect will last. Kohavi and colleagues (2012) describe novelty effects, in which a new feature draws repeated tries and then loses its appeal, and primacy effects, in which experienced users adapt slowly. Their example is opening Hotmail in a new window, which showed a strong novelty effect and settled into a positive long-term result. Hohnhold, O'Brien and Tang (2015) report that short-term effects do not always predict what happens once users adapt, and that ad clicks shift with the quality of the ads users saw before.

It does not say the result is sound if the test was stopped when it first looked good. A preprint on "always valid" inference by Johari, Pekelis and Walsh states that inferences drawn from samples chosen by continuous monitoring are wholly unreliable, and offers p-values and confidence intervals that remain valid at any stopping point. Early data also deceive without any stopping rule: in an A/A test shown in the 2012 paper, the cumulative difference between two identical groups appeared to trend strongly and settled near zero.

It does not say that most ideas win. Kohavi's 2012 slides report that about a third of ideas tested at Microsoft were positive and significant, a third flat and a third negative and significant. The same slide cites a book for the figure that Google ran roughly 12,000 randomised experiments in 2009 and about 10 per cent led to business changes.

It does not say that a test is affordable for every question. Lewis and Rao (2015) ran 25 field experiments on advertising and found that the median confidence interval on return on investment was more than 100 percentage points wide, because individual sales are so variable against the cost of advertising. They estimate an informative experiment could need more than 10 million person-weeks. Finally, in the same chapter where Kucharski discusses randomised trials, results are described as not always transferring to other settings.

Who developed it

There is no single originator. The statistical design of experiments owes most to Ronald Fisher, whose The Design of Experiments (1935) includes the tea-tasting example in which cups are served in random order. Online practice was formalised by Ron Kohavi and colleagues at Microsoft, in papers from 2007 and 2012 and a 2017 Harvard Business Review article. Kohavi's book with Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, is a practitioner text. The vault's source for the commercial reading is Adam Kucharski's Proof.

---