Reference Class Forecasting

What it is

Bent Flyvbjerg set the method out in a 2006 paper in the Project Management Journal, "From Nobel Prize to Project Management: Getting Risks Right". It has three steps. First, identify a relevant set of past, similar projects, large enough to be statistically meaningful and close enough to be comparable. Second, build a probability distribution of outcomes for that set, which requires credible data on enough projects. Third, compare the project in hand with that distribution to estimate its most likely outcome.

The first step contains a trade-off that no formula settles. A narrow class is comparable and small, a wide class is large and loosely comparable, and the person choosing the class is making a judgement. The paper presents the method as resting on the work of Daniel Kahneman and Amos Tversky on decision making under uncertainty, for which Kahneman received the 2002 Nobel prize in economics, and as a way of correcting the systematic, predictable bias that work identified. Its abstract says the approach sidesteps optimism bias and strategic misrepresentation, because the forecast starts from what happened to other projects and not from the project's own account of itself.

The idea behind it is older than the name. Kahneman and Dan Lovallo called the two ways of forecasting the inside view and the outside view in 1993. The inside view builds the estimate from the specifics of the case. The outside view treats the case as one member of a class and starts from the class. Reference class forecasting turns that second habit into something a public body can require.

In effect

Flyvbjerg's 2006 paper gives example distributions for transport projects in the United Kingdom, in constant prices. Average cost inaccuracy was 44.7 per cent for rail (standard deviation 38.4), 33.8 per cent for bridges and tunnels (62.4) and 20.4 per cent for roads (29.9). The samples were 46 rail projects, 34 bridges and tunnels and 172 road projects. Rail passenger forecasts averaged an inaccuracy of minus 51.4 per cent, and 84 per cent of rail projects were off by more than 20 per cent in one direction or the other. The wide standard deviations matter: a class average says where a project is likely to land, and the spread says how far it can land from there.

The method became policy in steps. After HM Treasury issued guidance on optimism bias in 2003, the Department for Transport commissioned Flyvbjerg and the consultancy Cowi to develop uplifts from reference class data. From August 2004, local authorities seeking transport funding had to apply those uplifts. The first practical use, according to the 2006 paper, was a 2004 review of the Edinburgh Tram Line 2 by the engineering firm Ove Arup, which judged the promoter's estimate of roughly 320 million pounds optimistic. In April 2005 the American Planning Association endorsed the method alongside traditional techniques, stating that planners should never rely solely on civil engineering technology as a way to generate project forecasts.

The Department for Transport has since revisited the numbers. A 2020 update of the evidence behind its optimism bias uplifts notes that the 2004 guidance covered capital cost estimates only, and that the 2004 rail class held 46 projects, three of them British. The update recalculated the rail figures from international cost data, because the British-only rail sample, at 18 projects, fell below the report's own threshold of 20. For a rail project at the full business case stage it recommends an uplift of 19 per cent at the 50th percentile and 60 per cent at the 80th, with 121 per cent and 77 per cent at the 80th percentile for the strategic outline and outline business case stages. An estimate of 100 at the full business case stage is therefore carried forward as somewhere between 119 and 160, depending on how much risk the decision maker accepts.

What it does not say

Reference class forecasting corrects the number. It does not say why the original number was low. The same uplift applies whether an estimate was low through honest bias, as in the experiments behind the planning fallacy, or low through deliberate choice, which Flyvbjerg and his colleagues argue in their 2002 study of 258 transport projects. That dispute is set out under strategic misrepresentation.

The method also depends on data that may not exist. A project without precedent has no class, and a class with few members gives an unstable distribution, which is why the Department for Transport's 2020 update sets a minimum of 20 projects and turns to international data when a national sample falls short. The uplifts it describes adjust capital cost only, and the update suggests the 50th percentile as the low bound, the 65th as a midpoint and the 80th as the high bound.

Finally, this page cites no controlled comparison of forecasts made with and without the method. The evidence set out above is evidence of two kinds: the record of overruns on which the uplifts are built, and the fact of adoption by a regulator and endorsement by a professional body. Adoption is not the same as a demonstrated improvement in accuracy.

---