Regressive research: The pitfalls of post hoc data selection in the study of unconscious mental processes
Shanks identifies a statistical fault running through a large body of research on unconscious processing. The standard design measures two things: performance on some task, and awareness of the thing supposedly influencing it. The conclusion that processing was unconscious requires performance above chance combined with awareness at chance. In practice, average awareness usually exceeds chance, so researchers select after the fact the participants, trials or items where awareness happened to be lowest, and analyse those. His argument is that this selection guarantees the result through regression to the mean: people who scored lowest on the awareness measure did so partly by chance and will score higher if measured again, so their apparent unawareness is partly an artefact of having been chosen for it. He works the argument through recent research on implicit learning and subliminal perception. This is a methodological paper that undercuts a great many findings at once, and this vault cites it wherever unconscious influence is claimed.
Shanks identifies a statistical fault running through a large body of research on unconscious processing. The standard design measures two things: performance on some task, and awareness of the thing supposedly influencing it. The conclusion that processing was unconscious requires performance above chance combined with awareness at chance. In practice, average awareness usually exceeds chance, so researchers select after the fact the participants, trials or items where awareness happened to be lowest, and analyse those. His argument is that this selection guarantees the result through regression to the mean: people who scored lowest on the awareness measure did so partly by chance and will score higher if measured again, so their apparent unawareness is partly an artefact of having been chosen for it. He works the argument through recent research on implicit learning and subliminal perception. This is a methodological paper that undercuts a great many findings at once, and this vault cites it wherever unconscious influence is claimed.
Written from the abstract, OpenAlex.
What our sources record
Reproduced as it was written when this source was checked, figures and all.
A model with no automatic/controlled distinction reproduces the Payne pattern.
Where we use it
- What Forty-One Participants Buys, Why We Look
Reliability of this reference
The link below is the right work, with one detail that does not line up. Title, author and year match, but the venue in the supplied citation is wrong: the paper is Psychonomic Bulletin & Review 24(3), 752-775 (online 2016, issue 2017), not JEP: General 146(8), 1216-1222. Worth opening before quoting a figure from it.
The source
https://doi.org/10.3758/s13423-016-1170-y
DOI: 10.3758/s13423-016-1170-y