Sample Composition

What it is

Three distinct questions hide behind the word sample, and the reform movement of the last fifteen years has answered two of them.

Size is a power question. A study with forty-one participants may be too small to detect the effect it reports. The remedy is more people, and the field now has power analysis, larger default samples and a working vocabulary for the problem.

Replicability is a stability question. A finding may not survive an independent attempt. The remedy is preregistration, registered reports and another run, and the field now has all three.

Composition is a reference-class question. It asks what population an estimate is an estimate of. The remedy is not more participants, because more participants from the same pool make the estimate of a local effect more precise without widening the population the estimate is about. That asymmetry is the whole content of this page.

The reform machinery does not ask where the participants were. A preregistration specifies hypotheses, measures and analysis plans. A power calculation specifies a number. Neither asks for a country, and the consequence is measurable: Leah Petrutiu and colleagues found 13.1 per cent of the 1,293 samples they coded across five British Psychological Society journals for 2021 to 2023 named no country at all. Their reading of that silence is the sharpest thing in the paper, that groups considered normative tend to go unexplained, so many of those were probably recruited in the West too.

And the incentive runs the other way, which explains the persistence better than an oversight would. Joseph K. Goodman and Gabriele Paolacci record the practice in their own tutorial's words: the majority of crowdsourcing workers reside in the United States, and most researchers restrict participation to United States residents in order to increase homogeneity. Homogeneity lowers variance. Lower variance makes an effect easier to detect. A detectable effect is what gets published. So a narrow sample is a design choice that improves the statistics of a study while narrowing what the study is about. That is a trade-off, stated in the literature by the people making it, and it is an account of incentives rather than an imputation of motive to any researcher.

Composition is also the variable nobody was optimising. Over the decade in which power and preregistration practices changed out of recognition, the American share of authors and samples in six leading psychology journals fell by roughly ten percentage points, and the change came almost entirely from other English-speaking and Western European countries.

In effect

In research: a multi-site project that varied one thing and held another constant

Many Labs 2 ran 28 findings across 125 samples, 15,305 participants and 36 countries, and found little heterogeneity between WEIRD and less WEIRD cultures. The 2024 reply from the authors of the original critique points out that 39 per cent of that pool was United States-based and that participants came mostly from university subject pools and crowdsourcing platforms. Both statements are about composition. The project varied the country and held the kind of participant close to constant, so on the composition question it tested less than the headline numbers suggest, which is why both sides call the design the issue.

In everyday life: the survey that asks the wrong thing twice

A news story reporting that people do something is reporting an estimate from a population. Ask which population and the answer is usually either absent or a panel. The useful habit is not scepticism about the number but a single question about the frame: among whom? A result about British undergraduates and a result about British adults are different results, and the headline will not distinguish them.

In commerce: more respondents, same frame

A brand testing packaging doubles its sample from 500 to 1,000 and the confidence interval narrows. Nothing about the population has changed. If all 1,000 came from one country's panel, the study is now a more precise measurement of the same narrow thing, and the money bought precision rather than reach. The buyer's question is which cell of the frame is empty, not how large the sample is.

What it does not say

It does not say a narrow sample invalidates a finding. It says the finding is about the population it was measured on, which is a statement about scope rather than about truth.

It does not say size and replicability are unimportant. They are different problems with different remedies, and conflating all three is what this distinction exists to prevent.

It is not a research claim in its own right. It is a methodological distinction, stated here in this publication's own words, and it has no single originator. The audits it rests on are counts, not experiments, and the specific figures on this page belong to the papers named in the sources below rather than to the distinction itself.

It also does not supply a threshold. Nothing here says how wide a frame must be before a claim about people is licensed, because nobody has established one.

---