The Unquantified Claim

The Unquantified Claim

Reviewed by Faisal Zaidi

An unquantified claim is a comparative claim made without any of the magnitudes that would let a reader check it. It is this site's own name for a pattern rather than a term from the literature. The cleanest specimen is Grit, which compares its construct to conscientiousness across its length and states no effect size, no correlation, no odds ratio and no regression coefficient anywhere in 335 pages, while the entire published dispute about it is quantitative.

What it is

The pattern is not an absence of sources. It is an absence of the specific numbers on which a comparison could be assessed. A book can cite papers, name researchers and describe studies, and still leave a reader with no way to judge whether the thing being claimed is large or small.

The grit case is unusually clean because the missing quantity is exactly the quantity the external argument turns on. The published case against the construct consists of three numbers: a true-score correlation of .84 between grit and conscientiousness; a benchmark of .63 between two different conscientiousness inventories; and incremental-validity regressions in which overall grit adds nothing to the prediction of academic performance once conscientiousness is controlled. A reader of the trade book cannot check any of that, because the book supplies none of the numbers that would let them.

The contrast with the papers is the demonstration. The founding 2007 paper gives variance explained study by study, at 4.8, 4.8, 6.3, 3.9, 1.4 and 3.8 per cent, and its own abstract says grit accounted for an average of four per cent of the variance in success outcomes. It also prints four limitations: exclusive reliance on self-report with social desirability named, the retrospective character of the scale with the concession that a case could be made that the research shows past behaviour predicting future behaviour, severe range restriction on intelligence in four of the six studies, and the possibility that grit is a mere epiphenomenon. None of the six percentages, and none of the four limitations, appears in the trade book.

The same absence runs in the other direction, which is what makes this a pattern about documents rather than a complaint about one author. Lauren Eskreis-Winkler's 2014 paper supplies three of the four headline claims in the book's first chapter, and in the same tables reports that physical fitness, general intelligence, standardised achievement scores, being female and weeks already employed each out-predicted grit in the studies concerned, several of them by large margins. The book names none of them. In the special-forces study, physical fitness carried about four times grit's share of the variance. In the sales study, weeks already employed carried seven times it.

What makes the pattern worth a name is that it is detectable without access to anything the reader does not have. A comparative claim with no magnitude attached is visible on the page, and the question of what the number would have been is answerable by going to the paper.

In effect

Four per cent, in the abstract, absent from the book

The research case is the founding paper's own headline. Grit accounted for an average of four per cent of the variance in success outcomes across six studies. That figure is in the abstract of the paper the whole literature rests on and is the single fairest number anybody could give the construct. Its absence from the popular account is what allows a comparative claim about conscientiousness to be made without the comparison ever appearing.

Reading a trade book with one question in hand

In practice the pattern gives a reader a cheap test. When a book says one thing matters more than another, ask how much more, and look for the answer in the same paragraph. If the magnitude is absent across an entire argument, the honest conclusion is not that the claim is false but that the book has not supplied the means to assess it, and the paper should be consulted before the claim is repeated.

Why a procurement decision needs the number

For an institution deciding whether to buy an assessment or a programme, the magnitude is the decision. The published record here contains a construct correlating at .84 with an established trait, adding no incremental variance for academic performance, with a perseverance subscale that does add at increments of .023 to .085 in explained variance. Those are the figures a buyer needs and they exist. A recommendation built on the trade account alone would be made without any of them, which is the practical cost of the pattern.

What it does not say

It does not say the claims are false. It says they cannot be checked from the document that makes them.

It does not impute intent. The absence is a property of a text, and this site draws no inference about why it is there.

It is not a claim about sourcing in general. The books this pattern has been found in cite real papers; what is missing is the magnitudes, not the references.

It is not a term from the research literature. It is this site's own name for a recurring editorial failure, and it is labelled as such.

Who developed it

The name is this site's own. The quantities it points at are Marcus Credé's, from the 2017 meta-analysis read in full at source, and Angela Duckworth's own, from the 2007 founding paper and the 2019 West Point megaanalysis, both also read in full at source.


Sources

  1. Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Journal of Personality and Social Psychology, 92(6), 1087-1101. doi:10.1037/0022-3514.92.6.1087. Full text read at source, including Table 3 and the limitations section.
  2. Credé, M., Tynan, M. C., & Harms, P. D. (2017). Journal of Personality and Social Psychology, 113(3), 492-511. doi:10.1037/pspp0000102. Full text read at source.
  3. Eskreis-Winkler, L., Shulman, E. P., Beal, S. A., & Duckworth, A. L. (2014). "The grit effect: predicting retention in the military, the workplace, school and marriage." Frontiers in Psychology, 5, article 36. doi:10.3389/fpsyg.2014.00036. Full text read at source with every table.
  4. Duckworth, A. L. (2017). Grit: Why Passion and Resilience Are the Secrets to Success. Vermilion. Read in full by the vault.