Trustworthy online controlled experiments: five puzzling outcomes explained
Kohavi and colleagues explain that companies such as Amazon, Microsoft and Google rely on online controlled experiments for data-driven decisions. The theory is simple and dates to Ronald Fisher's agricultural trials of the 1920s, but running thousands of experiments a year has produced practical lessons. The authors recount surprising outcomes investigated over weeks or months, often traced to root causes that generalise, and cover the overall evaluation criterion, click tracking, effect trends, length bias, statistical power and carryover effects. They argue that at Bing, where a wrong decision can affect annual revenue by millions of dollars, understanding anomalies is worthwhile.
Kohavi and colleagues explain that companies such as Amazon, Microsoft and Google rely on online controlled experiments for data-driven decisions. The theory is simple and dates to Ronald Fisher's agricultural trials of the 1920s, but running thousands of experiments a year has produced practical lessons. The authors recount surprising outcomes investigated over weeks or months, often traced to root causes that generalise, and cover the overall evaluation criterion, click tracking, effect trends, length bias, statistical power and carryover effects. They argue that at Bing, where a wrong decision can affect annual revenue by millions of dollars, understanding anomalies is worthwhile.
Written from the abstract read 2026-10-26 via OpenAlex.
What our sources record
Recorded when this source was checked, figures included, with page references and quotation marks removed.
Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '12). doi:10.1145/2339530.2339653