Risk of Bias

What it is

The sources are mundane and they are everywhere. Assessors who knew which group they were rating. Participants who could tell which arm they were in. Outcomes selected after the data came back. People dropping out of one arm more than another.

None of that means a result is wrong. It means there is a second explanation available for it that the study did not close off.

In effect

A meta-analysis can report a positive average effect and, in the same paper, record that most of the studies feeding it were at moderate risk of bias. Both statements are the finding, and coverage almost always keeps the first.

Fincham and colleagues found a small-to-medium benefit for breathwork across twelve randomised trials, judged most of them at moderate risk of bias, and closed by urging caution to avoid what they call a miscalibration between hype and evidence. The authors' own result was positive and they wrote the warning anyway.

What it does not say

It does not say that a study at moderate risk of bias should be ignored. Most psychological and behavioural research sits there, for practical reasons, and a literature composed entirely of low risk-of-bias trials would be a very small literature.

It also does not substitute for publication bias, which is the separate problem of which studies get to exist at all.

---