Dismantling Design

What it is

Almost nothing that gets called a treatment is a single thing. An eight-week course is a technique, a teacher, a group of other people in the room, a fixed schedule, a set of expectations about what is supposed to happen, and the plain fact of turning up somewhere every week for two months. A therapy is a rationale, a relationship, a set of exercises and the experience of being taken seriously by someone for fifty minutes. A supplement is a compound, a routine and a decision to do something about a problem. Every one of those bundles can produce a real effect, and every one of them has several candidate reasons why.

An outcome trial against a waiting list tells you the bundle beat nothing. That is worth knowing and it is nearly silent on mechanism, because the waiting list differs from the treatment in all of the strands at once. If the course works because of the group, or the schedule, or the expectation, a waitlist comparison will show exactly the same result as if it works for the reason printed on the tin.

A dismantling design attacks this from the inside, by running arms that contain some components and not others. An active control attacks it from the outside, by building a comparison condition matched on everything except the one ingredient under test. Both are answering the same question, and both are far more expensive than an outcome trial, because they are looking for a difference between two things that work rather than between something and nothing, which needs a much larger sample to detect.

The reason this concept has a page on a consumer psychology site is that the distinction between "it works" and "it works for the stated reason" is where most popular science goes wrong, and it goes wrong in a way a non-specialist can catch. You do not need to read the statistics. You need to find the comparison group and ask what it was doing. If the answer is nothing, or a waiting list, or treatment as usual, then whatever the paper says about mechanism is a hypothesis with an outcome trial standing next to it, not evidence for the mechanism.

Applied to the case this vault came to it through: mindfulness-based programmes have decent outcome evidence and no mechanism evidence. That is not a criticism of the practice. It is a description of which trials have been run.

In effect

Research: taking cognitive therapy apart (Jacobson and colleagues, 1996)

Neil Jacobson and colleagues randomised 150 outpatients with major depression to three arms: behavioural activation alone, behavioural activation plus work on automatic thoughts, and the full cognitive therapy package including work on core schemas. The cognitive model says the schema work is where the change comes from, so the full package should have outperformed the stripped-down arms. It did not. Behavioural activation on its own did as well as the complete treatment, and the result held at six months. The therapy kept working. The explanation of why it worked did not survive contact with a design built to test it, and behavioural activation went on to become a stand-alone treatment on the strength of this study.

Everyday life: the question, and what a good answer looks like

The usable version is one sentence: what did the other group get? Applied to a supplement trial where the control got a placebo pill but not the accompanying advice, applied to a productivity method tested against doing nothing, applied to a school programme compared with an ordinary timetable, it separates claims about outcomes from claims about causes within about a minute. For an example of the effort a proper answer takes, MacCoon and colleagues published an active control for mindfulness-based stress reduction in Behaviour Research and Therapy in 2012: an eight-week Health Enhancement Program built from nutrition, functional movement, physical activity and music therapy, designed to be structurally equivalent to MBSR and credible to participants and providers while containing no mindfulness at all. In the validation, the two conditions matched on drop-out, classes attended, time in class and time spent on home practice. When MacCoon and colleagues later used it to test sustained attention, in PLoS ONE in 2014, the finding was in the title: no differences. Building the comparison was most of the work.

Commerce: eight weeks of mindfulness, sold on a mechanism nobody has isolated

Madhav Goyal and colleagues' 2014 review in JAMA Internal Medicine pooled 47 trials covering 3,515 participants and put mindfulness-based programmes at an effect size of 0.38 for anxiety and 0.30 for depression at eight weeks, with no evidence of any advantage over other active treatments. Something in the room is working, and that number does not say what. The commercial layer above it, apps, corporate programmes, teacher trainings and retreats, sells a mechanism: attention trained, craving reduced, the thinking mind quieted. Between the outcome evidence and the mechanism claim sits a dismantling trial nobody has run. Relaxation, structured attention, group contact, expectancy and the discipline of eight weeks of anything are all live candidates, and none has been ruled out.

What it does not say

A dismantling result does not say the treatment does not work. Jacobson and colleagues did not show cognitive therapy fails. They showed the component credited with the effect may not be the component producing it, which leaves a working treatment and a broken explanation.

A component that does nothing alone is not necessarily inert. Ingredients can interact, and a design that tests parts separately can miss something that only functions in combination. Dismantling is a strong test of a mechanism claim and a weaker test of a component's value.

Absence of a dismantling trial is not evidence against a mechanism. It is absence of evidence for one, and those are different. The mindfulness case is precisely this shape: the mechanism may well be what the tradition says it is, and nobody has shown it either way.

The design has its own failure modes. An active control can be built so well matched that it accidentally contains the ingredient under test, and a dismantling arm can be underpowered to the point where a real difference between two effective treatments is invisible. Both failures produce the same output, a null, and a null from a small dismantling trial is weak evidence in either direction. The same caution set out under preregistration applies here.

The rarity of these trials is mostly economics rather than motive. They cost more, take longer and are harder to publish than outcome trials, which is enough to explain their scarcity without anyone being accused of avoiding the answer.

---