Non-Naivety

What it is

The standard assumption of an experiment is that the participant has not seen the stimulus. It is rarely stated, because for most of the twentieth century it did not need to be: an undergraduate subject pool was large, local, and turned over every year, and a given student met a given task once.

A shared online labour pool breaks the assumption quietly. The participants are not students fulfilling a course requirement but workers choosing tasks for money, from a population that several hundred laboratories draw on simultaneously, and the same person can meet a trolley problem, an anchoring item or an ultimatum game many times across many studies, for many different researchers.

Neil Stewart and colleagues sized the pool by capture-recapture in 2015, and the figure is the one that turns a worry into arithmetic. From 114,460 completed sessions and 33,408 unique participants across 689 sessions and seven laboratories in the United States, Europe and Australia between January 2012 and March 2015, they estimate the effective size of the active population a typical laboratory can access at about 7,300 workers, with the time for half the workers to leave the pool and be replaced at about seven months, and each laboratory's own pool overlapping, often extensively, with the hundreds of others using the platform.

Put beside a large undergraduate subject pool, the comparison is not flattering. A few thousand people, mostly in one country, shared with hundreds of laboratories, and half gone within the year.

The occupational half of the problem has also been measured. Eyal Peer and colleagues, comparing platforms in 2022, report that 42 and 41 per cent of participants on their two crowdsourcing conditions said they used the site as their main source of income, against 8 per cent on one alternative panel and 7 per cent on another, and report significant effects on data quality for the site, for hours spent on it per week, and for whether it was the participant's main income, with significant interactions among all three. Two things travel with that paper and are stated here rather than buried: an erratum exists which this publication has not read, and a competing platform published a reply disputing the conclusions, also not read. The existence of both is recorded and nothing is attributed to either.

This is a composition problem rather than a size problem, which is why it is invisible where people look for it. A study with eight hundred responses has eight hundred responses. It does not have eight hundred people who had never seen the task.

In effect

In research: the famous paradigm, run for a decade on the same few thousand people

An effect first demonstrated on undergraduates in the 1970s, then run repeatedly on a shared online pool across the 2010s, is being measured partly on respondents who may have answered a version of it before, for another laboratory, for payment. Nothing in a preregistration asks, no journal requires the figure, and the capture-recapture estimate above is the only quantity available for reasoning about how often it happens.

In everyday life: the fair half, which is substantial

The platform's virtues are real and documented. Joseph K. Goodman and Gabriele Paolacci record that crowdsourced workers match or beat undergraduate samples on attention and conscientiousness, produce the same biases at similar effect sizes, and let researchers without a laboratory run adequately powered work. Mechanical Turk carried 27 per cent of the 1,350 surveys and experiments they counted across four volumes of one journal, rising from 9 to 43 per cent. Their own warning is the one to keep: more representative than students does not mean representative.

In commerce: the panel you have already used

A brand running the same concept test through the same panel every quarter is buying responses from a population that is smaller than the invoice implies and partly professional. The practical check is not the sample size on the first slide. It is whether the supplier can say how many of those respondents it had used in the previous twelve months, and how many of them answer surveys for a living.

What it does not say

It does not say crowdsourced data is bad data. The same literature that sizes the pool records that it produces comparable effect sizes and better attention than the student samples it replaced.

It does not quantify how much prior exposure changes a result. The capture-recapture figure sizes the pool; it does not measure the bias from repeat exposure, and this publication knows of no study that puts a general number on that.

The platform-comparison figures are about one set of platforms at one time, and the dispute around that paper is recorded above rather than resolved. Nothing here is a claim about any particular platform's current data quality.

And the 7,300 figure is an estimate of what a typical laboratory can reach, not a census of the platform. It was read at abstract level, which is where the authors state it.

---