Testing

Treat the idea as a guess, assign people to versions at random and run them at the same time, decide the main measure, size and length in advance, run long enough to cover at least a full week, and log every result including those that show no difference. At Microsoft, only about a third of tested ideas improved results.

What the report found

1. Test, don't guess. At one large technology firm, about a third of tested ideas improved results, a third made no difference and a third made things worse. Even experts struggle to predict which is which.
2. Compare fairly. When advertising was measured without a randomised comparison, the usual methods often overstated what the ads achieved.
3. Decide in advance what counts. With four flexible choices about how to analyse a study, the chance of a false finding rose from 5 per cent to about 61 per cent.
4. Make it big enough and long enough. Small tests miss real effects and exaggerate the ones they find, and early results often fade.
5. Learn from every result. A test that shows no difference is still an answer, and a record of every test is worth more than a few striking wins.

Who it is for

For marketers, product managers, founders, analysts and anyone who decides what to launch.

How we work

Field Guides turn behavioural science into practical principles. Every claim is graded Established, Emerging or Untested. Each principle comes with the evidence, a worked example from the UAE and what to try and watch out for, and the guide ends with a worksheet and a checklist.

Cite this report

The Why We Project (2026) Testing: Running small experiments. Field Guide No. 10. Dubai: Why We Tech Ltd. Available at: thewhyweproject.com