The Psychological Solution First
The most quoted case in the argument for buying a perception instead of a capability is not in the book it is credited to. It is a 2009 conference line, the supermodels in it are explicitly a joke, and the railway it uses as a foil is the one place where the metric it attacks came out ahead of the concrete.
Answer to the question
Does behavioural economics actually work?
The evidence is split. A 2022 meta-analysis of 447 effect sizes in behavioural economics reported a moderate average, while a bias-corrected reanalysis of the same corrected data put it near zero, with a ceiling around fourteen hundredths of a standard deviation. The nudge units' own census of 126 trials found 1.4 percentage points of take-up against 8.7 in the journals.
Key takeaways
- The Eurostar comparison is not in Alchemy. It is from a 2009 conference talk, quoted here from a broadcaster's verbatim transcript, in which the supermodels are explicitly a joke and the real proposal is wireless internet at about one per cent of the cost.
- High Speed 1 cost 6,163 million pounds in outturn terms and the scheme saved about forty minutes, from a base nearer three hours than the three and a half the talk assumes.
- The journey time savings are the part that paid, valued at 7,000 million pounds to 2070, while demand came in at a third of the 1995 forecast and net taxpayer support may reach 10,200 million.
- Transport economics already backs the substantive point: positive utility of travel time across 26,221 passengers, and 208 crowding valuations pricing standing time at roughly double seated time on average.
- Of the book's testable cases only Save More Tomorrow survives, and the method question behind all of them is open rather than settled.
The most quoted argument in popular behavioural economics is not in the book it is usually attributed to, and the metric it attacks is the one the auditors actually used. The Eurostar comparison belongs to a conference talk rather than to the book. The scheme behind it has been audited, and the audit reads against the argument.
The case that is not in the book
Rory Sutherland's Alchemy was published by WH Allen in 2019 and read here in the 2020 paperback. Its operative claim sits at page 69, under the heading "Psychological moonshots":
> "Making a train journey 20 per cent faster might cost hundreds of millions, but making it 20 per cent more enjoyable may cost almost nothing."
The Eurostar illustration that every retelling attaches to that sentence is not on the page. It comes from one of his TED talks, and the passage below reaches us through NPR's verbatim transcript of a TED Radio Hour segment carrying it. That segment splices clips from more than one of his talks and never says which, so no talk is named here.
> "They came up with a very good engineering solution, which was to spend 6 billion pounds building completely new tracks from London to the coast and knocking about 40 minutes off a three and a half hour journey time."
Then come the supermodels, the Chateau Petrus and the three billion pounds of change, which is the part that travels. He offers it as the naive advertising man's suggestion, and the transcript marks laughter at that point.
Later in the same transcript, speaking to the presenter in a 2014 interview about the talk rather than from the stage, he says his more serious suggestion had been wireless internet at not quite one per cent of that money.
The quote travels; the source does not.
Everybody quotes the supermodels. The supermodels are not in the book, and they were never the proposal.
The book's equivalent rail case is a counter-proposal to High Speed 2 at pages 137 to 138, a different scheme with different numbers. Anybody who puts the supermodels inside the best-selling popular book on behavioural economics has the wrong source, and anybody who presents them as the recommendation has the wrong proposal.
What six billion pounds actually bought
The scheme is High Speed 1, formerly the Channel Tunnel Rail Link, and the National Audit Office's closing report on it, The completion and sale of High Speed 1, published on 28 March 2012, is the authority. We read its executive summary in full.
His headline figure is right. Construction came in at 6,163 million pounds in outturn terms, including scope added and funded separately, eighteen per cent above the contractor's combined targets. The auditor notes that the West Coast Main Line upgrade rose by over six billion pounds and ran four years late, so High Speed 1 "compares well with other railway projects".
What the report records the money as buying:
| Item | As audited |
|---|---|
| Construction, outturn terms | 6,163 million pounds |
| Theoretical capacity | up to 20 trains an hour in each direction |
| Full opening | November 2007, Eurostar moved to St Pancras |
| Kent domestic high-speed services | from December 2009 |
| Promoter's regeneration case | at least 70,000 jobs |
On the clock, forty minutes is defensible for the two-stage scheme as a whole, which took London to Paris from about three hours to roughly two and a quarter. Three and a half hours is not defensible: the base was nearer three. We read those times at news reports and a railway society account rather than at a timetable, which is why the figure here is "about three hours" and not a booked time.
Wireless internet at one per cent of six billion pounds is sixty million pounds. Sixty million pounds does not buy twenty trains an hour, a domestic service into Kent, or the regeneration case at King's Cross, Stratford and Ebbsfleet. As an illustration of where imagination goes missing in procurement, the comparison is a good one. As a costing of two options against each other, it is not one.
What the journey time savings were worth
Here the arithmetic matters. The auditor, using the Department for Transport's own appraisal method, valued the journey time saving benefits at 7,000 million pounds in present value to 2070, inside a range of 6,100 to 7,700 million. The comparison it draws is not with construction but with net taxpayer costs of about 10,200 million in present value, and on that comparison it concludes the project is not value for money. Its summary states that project costs exceed the value of journey time saving benefits. Construction outturn of 6,163 million is a subset of project cost, and a cash outturn is not commensurable with a present value.
The report also records that transport benefits for international passengers, journey time savings and capacity improvements together, made up over half of the expected project benefits in the Department's 1998 assessment. Even on the Department's own accounting, time was never the whole case, and the rest of the case was not a comfort problem.
What went wrong was the passengers. There were 9.7 million Eurostar journeys in 2011. Actual numbers between 2007 and 2011 averaged one third of the level forecast in the 1995 bid and ran about thirty per cent below the 1998 forecast, the one against which the government guaranteed the debt. That verdict accepts that no clear conclusion is possible once unmeasured benefits are counted.
The 1998 business case claimed a benefit to cost ratio of one and a half to one. Reaching it would take at least 8,300 million pounds of benefits nobody has measured.
The appraisal did not fail because it counted minutes. It failed because it counted passengers wrong.
The appraisal did not fail because it counted minutes. It failed because it counted passengers wrong.
What the auditor could not check
And then the finding that makes this a piece about a method rather than about a railway. By 2012 the Department had not evaluated the project at all. It had not reassessed costs and benefits since 2001, despite a commitment to the Public Accounts Committee that it would. The auditor warns that it may not be able to measure the project's impact with any confidence, because it cannot show that it collected the baseline data or established a counterfactual.
That word is the hinge.
The engineers' case was never properly checked, which is a different failure from the one the talk diagnoses, and the report's first recommendation says so: departments should ensure that demand forecasts are subject to rigorous scrutiny and scepticism. Over-modelling is not the charge. Optimism is.
The symmetry is awkward for anyone using this scheme as a foil. The talk prices one option against another without building a counterfactual either. Sixty million pounds of wireless internet is costed against nothing, and no pilot is reported. The complaint the auditor makes of the Department is the complaint this piece makes of the talk, and behavioural economics has partly earned its reputation for cheapness because the cheap option is rarely asked to produce a business case at all.
The literature he does not cite
The substantive claim underneath the joke is that appraisal undervalues journey quality. On that, transport economics agrees with him, and has agreed for twenty years.
The British synthesis of the value of travel time is Pedro Abrantes and Mark Wardman's 2011 meta-analysis, covering 226 studies run between 1960 and 2008 and 1,749 separate valuations, with an elasticity of the value of time to income per head of about nine tenths. Credit the first author; the paper is habitually cited as Wardman. We read the abstract and the bibliographic record rather than the full text.
The weak joint in that framework was found by the field itself. Glenn Lyons, with Juliet Jain and David Holley, surveyed 26,221 British rail passengers by post in autumn 2004 and asked what they did with the time. They report a substantial if not overwhelming incidence of positive utility of travel time use, strongest among business travellers and present among commuters, and argue that this raises questions about the benefits assumed from policies that cut journey times. The abstract is as far as our reading went.
That is his argument, in a transport journal, twelve years before the book, with 26,221 respondents and a method.
He does not cite it.
Comfort already has a price too. Mark Wardman and Gerard Whelan pooled 208 valuations from seventeen British crowding studies spanning about twenty years. The seating multiplier averages a little under one and a fifth. The standing multiplier averages about two and a third, which means an hour standing in a crowded carriage is valued roughly like two and a third hours seated in an empty one. That is an average across studies and conditions, and newer measurement at observed peak densities runs considerably lower, so it is not the price of standing.
What that does to the argument is specific. In this literature comfort and speed are commensurable rather than rival paradigms, because a comfort improvement is quantified as a reduction in the time multiplier. His complaint about ten people having to stand a tenth of the time is, in appraisal terms, a request for a non-linear crowding function, which is the thing Wardman and Whelan spend their paper recommending. He is pushing at a door the field opened before he wrote.
The real complaint survives in a smaller form. Appraisal prices comfort inside a framework whose benefit side is still dominated by time, and the sixty-million-pound option has no promoter.
The effect size at full strength
Behind the rail case sits behavioural economics' own empirical question: how large are these interventions on average, once you stop selecting on what got published.
Stephanie Mertens, with Mario Herberz, Ulf Hahnel and Tobias Brosch, pooled 447 effect sizes in a 2022 meta-analysis and described choice architecture as an effective and widely applicable behaviour change tool. The random-effects average was forty-three hundredths of a standard deviation, with a confidence interval running from thirty-eight to forty-eight hundredths. By category, information came in at a quarter, structure at a little under three fifths, assistance at just over a fifth. A correction to the paper followed, and the corrected dataset is what the critics worked on. We did not read that paper in full; its figures reach us through the letter that tabulates them.
Maximilian Maier, with Frantisek Bartos, T. D. Stanley, David Shanks, Adam Harris and Eric-Jan Wagenmakers, wrote that letter, and we read it in full including its table and both footnotes. Their opening move is that the original authors' own finding of moderate publication bias is the real headline, and that corrected for it, no evidence for the effectiveness of nudges remains.
Their bias-adjusted Bayesian estimate falls to near zero, with the credible interval reaching only about fourteen hundredths at its top, and to roughly a ninth with a ceiling near a quarter when only the most precise estimate from each study is used. They report evidence against an effect for information interventions and for assistance interventions, and leave structure, the category that holds defaults, undecided. The asymmetry statistic in the original sits in the severe band of a standard benchmark, and publication bias shows up strongly in every subdomain except food.
Then the concession, which is the strongest card the defence holds. Every category and domain apart from finance shows heterogeneity, so some nudges may well work even where the mean effect is answered in the negative.
Two further letters attacked the same paper, from Barnabas Szaszi and from Jonathan Bakdash, and we have those from the search record alone, for their existence and their line of attack. Mertens and colleagues replied in the same journal; we did not read the reply and will not characterise it.
The cleanest number depends on no correction model at all. Stefano DellaVigna and Elizabeth Linos assembled 126 randomised controlled trials covering 23 million individuals, including every trial run by two of the largest nudge units in the United States, and set them against the academic literature. Take-up rose by 8.7 percentage points in the journal trials and by 1.4 percentage points in the units. About seventy per cent of that gap is selective publication, worsened by low statistical power. The units publish everything they run, so this is a census against a literature that selected for what worked, which is the problem [a whole generation of findings ran into](https://thewhyweproject.com/articles/the-psychology-you-learned-didnt-replicate).
Cheapness and its conditions
Small is not the same as poor value, and that is the defence worth taking at its strongest. Shlomo Benartzi, with John Beshears, Katherine Milkman, Richard Thaler and four colleagues, calculated ratios of impact to cost for behavioural economics interventions against traditional policy tools such as tax incentives and other financial inducements, and found that nudge interventions often compare favourably. They also wrote that more calculations are needed to settle relative effectiveness. We read the abstract; the cost tables sat behind a challenge page we could not pass, so no cost figure from them appears here.
This is Sutherland's own argument in different clothing. His formula at page 56, ninety per cent of the emotional and perceptual improvement at one per cent of the cost, is a cost-effectiveness claim. A letter costing pennies a household is a different proposition from a programme costing millions, whatever the two of them move.
Avishalom Tor and Jonathan Klick went back to those calculations. They argue that the original data show key cost elements variously excluded and included in a way that favours behavioural instruments, and that cost-effectiveness analysis is oversold as a way of choosing between policies. Corrected, they find that nudges do not consistently outperform traditional interventions under either cost-effectiveness or the cost-benefit analysis they say is methodologically required, and they recommend running cost-benefit analysis across every option with nudges among them. We read the abstract and the record.
Which leaves cheapness as a real argument with a condition attached. What makes a cheap intervention good value is a measured effect, and the measured effects shrink by a factor of about six once publication stops doing the selecting.
The cases, and the open question
The sharpest published criticism of the individual-first doctrine inside behavioural economics is not about effect sizes at all. Nick Chater and George Loewenstein argue that the field has concentrated on the individual frame, changing behaviour one person at a time, rather than the system frame that shapes it; that individual-level measures are dwarfed by system-level ones, a carbon tax against a nudge; and that individual-frame work can deflect attention and political support away from system reform. We read the abstract and repository records rather than the full text or the open commentaries published alongside it, and the summary is written at that width.
The pairing is not an ambush. Sutherland names Chater in that transcript as one of his great friends and a professor of decision sciences in London, and the criticism lands directly on the book's own policy claim at page 136, that governments are confined to legal compulsion and economic incentive while ignoring options that may be cheaper or less coercive. The reply, in effect, is that the twin levers are where the large effects live.
Held to the standard the book sets at page 293, where Sutherland writes that some parts of it are undoubtedly wrong and that his argument is simply that we should not recoil from testing alchemical solutions, one case survives.
| The book's testable case | Standing after the audit |
|---|---|
| Save More Tomorrow, pages 328 to 329 | Survives. A field intervention with a control group, in a named journal, which he cites. |
| High Speed 2 counter-proposal, pages 137 to 138 | No measurement offered. 250,000 pounds for a national ticketing app, with no modelling and no pilot. |
| The ride-hailing map, page 69 | No measurement offered. Ninety per cent less frustrating is a figure with nothing behind it. |
| The dot-matrix displays, page 232 | No measurement offered. A superlative with no source attached. |
Richard Thaler and Shlomo Benartzi's 2004 paper is the book's one real citation, in a footnote giving journal and date, and the footnote checks out. The doubling of participation and of average contributions after seven years are as Sutherland reports them; our reading stopped at the abstract and the bibliographic record. There is a tension worth stating rather than smoothing: the bias-corrected reanalysis finds its strongest evidence against an effect in the finance domain, where this case sits, while leaving undecided the structure category it actually belongs to.
The uncertainty claim underneath the ride-hailing map is well supported in work he does not cite, and this publication has covered [what a provider's visible effort does to a customer's valuation](https://thewhyweproject.com/articles/what-showing-the-work-actually-does) and [what a crowded checkout costs a buyer](https://thewhyweproject.com/articles/what-cognitive-load-costs-at-checkout) already.
The method question is open, and saying that is the position rather than a way of dodging one. Pooled behavioural economics looks moderate from one honest analysis and near zero from another, the census of unpublished trials looks small and cheap, and cost-effectiveness is contested in both directions by people who have read each other's spreadsheets. Nobody holding those three results at once can tell you the average size of a psychological solution. Asking for one first costs almost nothing, which is the defensible version of the claim, and it is the version at page 356: a fifth of the modelling time spent looking for psycho-logical answers instead.
That question picks between two worlds for whoever is buying. In the first, the psychological option is a cheap substitute for a capability, and the money not spent is real money saved. In the second, it is a perception bought instead of a capability, and the thing it displaced was the part that paid for itself. Those are different products. The comparison that tells them apart is the one the auditor asked for and the talk never built.
Common questions
Does behavioural economics actually work?
The evidence is split, and honestly so. A 2022 meta-analysis of 447 effect sizes reported a moderate average, while a bias-corrected reanalysis of the same corrected data put it near zero, with a ceiling around fourteen hundredths of a standard deviation. The nudge units' own census of 126 trials found 1.4 percentage points of take-up against 8.7 in the journals. Heterogeneity means some interventions still work.
Is the Eurostar supermodels argument in Alchemy?
No. It comes from one of Rory Sutherland's TED talks, reaching us through NPR's verbatim transcript, and the book's equivalent rail case is a counter-proposal to High Speed 2 at pages 137 to 138. In a 2014 interview about the talk he said his more serious suggestion had been wireless internet at not quite one per cent of the money.
Did High Speed 1 earn back its journey time savings?
No. The National Audit Office valued those savings at 7,000 million pounds in present value to 2070 and set them against net taxpayer costs of about 10,200 million, concluding that the project is not value for money. Its summary says project costs exceed the value of journey time saving benefits. Construction outturn of 6,163 million is only one part of project cost.
Does transport appraisal ignore comfort?
No. British crowding research pooled 208 valuations and prices standing time at about two and a third times seated time on average across studies, and a 2004 survey of 26,221 rail passengers found positive utility in travel time. Behavioural economics and appraisal sit closer together than the argument assumes, because comfort is priced in time's own currency rather than against it.
Sources
Sutherland, R. (2009). 'Life lessons from an ad man.' TEDGlobal 2009, July 2009
National Audit Office (2012). 'The completion and sale of High Speed 1.' HC 1834, Session 2010-2012, 28 March 2012
Abrantes, P. A. L., & Wardman, M. R. (2011). 'Meta-analysis of UK values of travel time: An update.' Transportation Research Part A, 45(1), 1-17. doi:10.1016/j.tra.2010.08.003
Lyons, G., Jain, J., & Holley, D. (2007). 'The use of travel time by rail passengers in Great Britain.' Transportation Research Part A, 41(1), 107-120. doi:10.1016/j.tra.2006.05.012
Wardman, M., & Whelan, G. (2011). 'Twenty Years of Rail Crowding Valuation Studies: Evidence and Lessons from British Experience.' Transport Reviews, 31(3), 379-398
Mertens, S., Herberz, M., Hahnel, U. J. J., & Brosch, T. (2022). 'The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains.' PNAS, 119(1), e2107346118. doi:10.1073/pnas.2107346118
Maier, M., Bartos, F., Stanley, T. D., Shanks, D. R., Harris, A. J. L., & Wagenmakers, E.-J. (2022). 'No evidence for nudging after adjusting for publication bias.' PNAS, 119(31), e2200300119. doi:10.1073/pnas.2200300119
DellaVigna, S., & Linos, E. (2022). 'RCTs to Scale: Comprehensive Evidence From Two Nudge Units.' Econometrica, 90(1), 81-116. doi:10.3982/ECTA18709
Benartzi, S., Beshears, J., Milkman, K. L., Thaler, R. H., Shankar, M., Tucker-Ray, W., Congdon, W. J., & Galing, S. (2017). 'Should Governments Invest More in Nudging?' Psychological Science
Tor, A., & Klick, J. (2022). 'When Should Governments Invest More in Nudging? Revisiting Benartzi et al. (2017).' Review of Law and Economics, 18(3), 347-376. doi:10.1515/rle-2021-0048
Thaler, R. H., & Benartzi, S. (2004). 'Save More Tomorrow: Using Behavioral Economics to Increase Employee Saving.' Journal of Political Economy, 112(S1), S164-S187. doi:10.1086/380085
Chater, N., & Loewenstein, G. (2023). 'The i-frame and the s-frame: How focusing on individual-level solutions has led behavioral public policy astray.' Behavioral and Brain Sciences, 46, e147. doi:10.1017/S0140525X22002023