Simultaneous Controls

What it is

Controlling for one variable at a time leaves every other variable free to carry the association, because age, education, income and the rest are correlated with each other and with how much television people report watching, so a relationship can survive each single-control test and have nothing left once they are entered together. The single-control procedure is not wrong in itself; it just does not answer the question people take it to answer.

Michael Hughes's nine controls were age, sex, race, income, education, hours worked per week outside the home, church attendance, membership of voluntary associations and population size, and that list is the useful part because it shows what a serious simultaneous control set looks like, meaning every plausible correlate of both the exposure and the outcome rather than two demographics. Paul M. Hirsch's paper also pointed out discrepancies in the items, samples and coding categories employed, so the controls charge arrived alongside a broader complaint, and his conclusion is stated at full strength in his own words, that the cultivation hypothesis lacks empirical support and that the very data presented in its support argue strongly for rejecting the assertion that it has any scientific basis in fact.

In effect

The question to ask of any survey result is how the controls were entered, because a paper reporting a relationship that survives adjustment for age, then for education, then for income, each separately, has not shown the relationship survives adjustment for all three. Hughes's second move is the one worth copying, since he took items from the same survey that the originators had used but had not analysed, ran the same procedure, and found results that fail to support the contention, which is a point about which outcomes got reported rather than about how they were modelled and is not answered by better statistics. The charge did not end the matter, because the Gerbner team replied to Hirsch in 1981 and to Hughes twice, Hirsch published a second part, and a further reply followed; all five reply items are confirmed to exist and none of their contents was read for this page, so what either side argued in the exchange is not reported here.

What it does not say

It does not establish that there is no effect, because two reanalyses of particular items in particular survey years are evidence about those items and those years, while the pooled estimate across five decades of the literature is still positive at about one tenth, so a relationship that disappears under simultaneous controls in 1975 and 1977 General Social Survey data has not been shown to be absent everywhere. It does not settle the dispute either, because the dispute was answered at length and this record cannot characterise the replies, which means the exchange is reported here as unresolved rather than as won. It is also not a claim that controls are always the right move, since adjusting for a variable that sits downstream of the exposure removes real effect rather than confounding, and nothing in Hirsch's or Hughes's papers as read here establishes which of the nine covariates are confounders and which are not, so the honest version of the charge is narrower and still damaging: the original analyses did not test what they were presented as testing. And no coefficient from either reanalysis appears on this site, because neither abstract supplies one and neither full text was read.

---