Vote Counting

What it is

A tally feels like evidence because it has a number in it. The number is the wrong one.

Suppose five studies of the same question all report an association in the same direction. Reported as a vote count, that is five out of five, and it sounds close to conclusive. Reported properly, five studies have five effect sizes, five sample sizes and five sets of confidence intervals, and those quantities answer questions a tally cannot. How big is the thing? Were the five estimates compatible with each other, or does one sit a long way from the rest? Did the largest study find the smallest effect? Was any of the five designed to be judged on this outcome, or did they each measure a dozen things?

The failure runs in three separable directions, and each one is enough on its own.

Magnitude goes first. A count cannot distinguish five tiny effects from five large ones, because a vote is a vote. This is the error that matters most in disputes where the whole argument is about size rather than direction. Both camps in the argument over adolescent mental health and social media agree the association is negative. Nobody disputes the sign. The entire dispute is over the magnitude, which is exactly the quantity a tally throws away.

Precision goes second. A study of two hundred people and a study of two hundred thousand each cast one vote. Meta-analysis exists to weight an estimate by how precisely it was measured, so that a large well-powered study moves the pooled answer more than a small noisy one. A tally inverts that: because small studies are easier to run, and because small studies produce more extreme estimates in both directions, a count can end up driven by its least reliable members.

And the denominator goes third, which is the subtlest and the worst. A count of studies that found something tells you nothing about how many were conducted. If fifty studies were run and the five that found an effect were the five that got published, five out of five is exactly what the literature would look like. This is publication bias, and the whole point of it is that it is invisible from inside the published record. Formal synthesis has instruments for detecting it, imperfect ones, and a tally has none. "Five of five" and "five of fifty" produce the same sentence.

What makes vote counting worth a page rather than a footnote is that it does not require bad studies, bad faith or carelessness. It can be assembled from excellent papers by a careful person. The defect is in the arithmetic performed on them, and it survives the quality of the inputs. That is also what makes it hard to spot in prose: a sentence of the form "study after study has found" reads as a summary of a literature and is in fact a refusal to summarise one.

The question that converts a tally into a synthesis is always the same, and it is short. How big, and how precisely measured?

In effect

Research: five studies, none of them named

The clearest instance this publication has met is in Jonathan Haidt's The Anxious Generation (2024), at page 150: "I have found five studies that looked at the rollout of high-speed internet around the world, and all five found evidence of damage to mental health."

That sentence is doing a great deal of work in the book's argument, and it is a vote count. None of the five studies is named in the main text. No effect size is given for any of them. No assessment is offered of whether five is the complete set of studies of that kind, which is the denominator question. A reader cannot check it, cannot weight it, and cannot tell whether the five agree about magnitude or merely about sign.

Set beside it one of the studies in that class, read at source. Marta Golin's 2022 paper in Health Economics instrumented German broadband access with technological features of the national telephone network. Her own words: "The results are suggestive that broadband Internet leads to worse mental health for women (primarily those aged 17-30) but not for men, thus widening the gender gap in mental disorders." Three things in that sentence vanish inside a tally. Suggestive is the author's own hedge. The population is adults and young adults, not children. The exposure is a telecommunications infrastructure rather than a phone or a social media account. A vote records the direction and loses the hedge, the population and the exposure, which between them are most of what a reader needs in order to know what the study is about.

Everyday life: the shop-shelf review score

Almost everyone has stood in front of two products, one with four five-star reviews and one with forty reviews averaging four stars, and felt the first look better. It is the same arithmetic. The count of favourable verdicts is doing the persuading and the sample size is not visible in the number that got printed on the shelf edge.

The fix is also the same, and it is a single question: how many people bought it, not how many liked it. Review platforms that display the count alongside the average are making the denominator visible, which is precisely the information a tally hides. The reason the mistake is so natural is that counting is easy and weighting is not, and that asymmetry is what drives the habit in academic writing too.

Commerce and product design: the logo wall

A business-software vendor's home page carries nine customer logos and nine favourable quotations. The implied claim is that the product works. What is absent is the denominator, how many customers were asked and how many declined, and the magnitude, how much better anything actually got for any of them.

The page is a vote count with the ballots displayed and the electorate hidden, and it is persuasive for exactly the reason it is uninformative. Compare what the honest version looks like: a case study with a measured outcome, a time period, and a named baseline. "Support tickets fell 18 per cent over two quarters across a team of 40" is a magnitude with a sample attached. "Nine happy customers" is a tally. Marketing departments know the difference, which is why the logo wall survives: it borrows the authority of evidence at none of the cost.

The same logic runs through press coverage of research. A headline that reports how many studies agree, rather than what they found, has performed a vote count on the reader's behalf and usually without telling them.

What it does not say

Calling an argument a vote count is not a claim that its conclusion is wrong. Five studies pointing the same way is genuinely some evidence, and a tally that happens to be drawn from a complete, well-powered and consistent literature can land on the right answer. The label describes what the argument has established, which is a direction, and declines to let it travel as if it had established a size.

Nor is it a criticism of the studies being counted. Golin's paper is peer-reviewed and careful, states its own hedge, and is not responsible for how a tally uses it.

And it is not a judgement about anyone's care or motives. The book in which this publication's clearest instance of a vote count appears is the same book that volunteers, in its own endnotes, that the decisive experiment on its own thesis has never been run, and then designs that experiment. Those two facts sit a few pages apart. Both belong in the record.

One real limit on the label itself: formal synthesis is not a cure for everything a tally gets wrong. A meta-analysis of five studies on the wrong population is a precise answer to the wrong question, and the tests for publication bias are weak at small numbers of studies. Vote counting is the worse of the two available options, not the only flawed one.

Who developed it

No single originator. Vote counting is a pre-meta-analytic habit of research synthesis, named and criticised in the methods literature rather than proposed by anyone, and this publication verified no attribution for the term. The vault's note records the originator as unknown rather than guessing, and the honest statement is that the practice is what formal synthesis was built to replace.

---