The Practitioner Experiment

What it is

The credit comes first, because it is real. No other book in this library has this column at all. Richard Shotton's The Choice Factory carries roughly twenty-two experiments and The Illusion of Choice roughly eleven, run by the author and named collaborators on questions the academic literature had not answered for his category. He names the collaborators. He reports sample sizes. He publishes at least one null result from his own work, which almost no trade author does, and he replicated an eighty-year-old finding twice rather than simply citing it. Doing the work and showing the number is more than the genre asks.

The problem is structural and has five parts, each survivable on its own. The experiments are unpublished and undeposited, so each one exists only as the sentence describing it. They are unrefereed, so no independent methodological check of any kind has been made. They are unregistered, so no stated hypothesis or analysis plan constrains how the result was arrived at. They run nearly uniformly positive, with about five nulls across the pair, which is the file-drawer signature from unregistered work, and no file-drawer disclosure appears in either book. And they are typographically identical to everything around them, so a reader meets an in-house test and a meta-analysis in the same prose, with the same confidence, and nothing marks the difference. Together those properties produce a class of evidence with all the surface features of a finding and none of its guarantees.

In effect

The class had to be tested against the vault's existing sourcing failures before it earned a name, and it is distinct from each. Source-class inflation levels sources of different standing inside one apparatus, so a weak entry and a strong one sit in adjacent numbered lines looking alike. Here the class never enters the apparatus at all, so there is no entry to inspect and no levelling to see. The unenumerated interview is the closest sibling and is also declared out of the apparatus, but an interview is a person's belief and does not claim to be data, and an unenumerated interview is usually covered by a blanket declaration at the head of the notes, so a diligent reader is warned once. No such declaration exists here. Blind notes are a navigation failure, where a mapping exists and can be rebuilt with effort, and here there is nothing to navigate to.

Both sides are owed. For the defence, trade books are allowed to report proprietary work, the alternative was not running the experiments at all, academic publication is not available to a consultancy on this timescale, and the existence of original data in a trade book is a net gain for the reader. For the prosecution, a sample size and a percentage set in the same typeface as a meta-analysis is a claim about evidentiary standing whether or not it is meant as one, and depositing a protocol and a dataset costs almost nothing and would convert the whole column from unusable to citable.

Where this publication lands: record it as a source class, credit it, and do not cite it. The practical rule is to treat a practitioner experiment as a well-informed hypothesis rather than as a result. Where the same claim is available from published work, cite that and drop the in-house number. Where it is not, say plainly in the copy that the source is the author's own unpublished test, and give the reader the chance the book did not.

What it does not say

It does not say the experiments are wrong, or that the results would fail if someone ran them again. Nobody knows, and that is the entire complaint.

It does not impute bad faith. An author has every incentive for a result to come out a particular way, which is not an accusation but the structural position of any consultancy demonstrating that its discipline works, and the remedy is registration rather than suspicion.

It does not extend to the rest of either book's apparatus, which is assessed separately, and it does not follow that a trade author should stop running experiments. The opposite follows.

---