ICBS Everywhere

Posted October 28, 2009

I am breaking my nearly month-long silence with something my 6th grader brought home today.
First I must say that the teacher responsible for this is one of the best I have seen, and I have seen some great teachers. Smart people who care take action and that is exactly what this teacher did. However, bad science followed by bad reporting of it leads to wasted effort on the parts of teachers, parents, activists, and anyone else who acts on the findings.

Today my son analyzed the composition of an article in class, but he was also told to take it home and ask a parent to sign it. His teacher’s intent was immediately apparent; the article was an Associated Press piece from 2006 which appeared on dozens of news sites and suggested that watching television or playing video games on school nights is bad for grades. A search will turn up many scary headlines for the article, including Study: TV At Night Hurts Grades, TV Hurts School Performance, and Watching TV on school night bad for grades. The study reported in the article was published in the October 2006 issue of the Journal Pediatrics.

Middle school students who watch TV or play video games during the week do worse in school, a new study finds, but weekend viewing and gaming doesn’t affect school performance much.

Even without looking at the Journal article, it is clear that the study and the media’s reporting of it are seriously flawed. The biggest problem is also the most common — a form of overgeneralization in which we assume that correlation means cause. This fallacious thinking leads to conclusions such as:

• Appliances make us smarter. Children raised in homes with more appliances get better grades than those with fewer appliances.
• Pirates can save us from global warming. As global temperatures rise, the number of pirates decreases.
• Strong religious beliefs cause poverty. Religiousity is higher in countries with high poverty rates than in countries is less poverty.
• The internet causes autism. As the rate of internet use rises, so does the rate of autism diagnosis.
• Buying ice cream makes one violent and unable to swim. As ice cream sales rise, so do violent crime and drowning deaths.

Of course these are mostly silly conclusions, but they clearly illustrate the problem with such claims.

To draw conclusions about causal relationships (e.g., that A causes B), three requirements must be met:

1. The variables must covary. In other words, they must be correlated. Note that correlation is a necessary condition for causal conclusions, but it is not sufficient.
2. The cause must precede the effect. A must occur before B.
3. There is no other variable which also covaries with A.

To meet the last requirement, we must conduct what is called a “true” experiment; we must randomly assign subjects to levels/values of the input variable (the cause). If we do not, any criteria used to assign subjects to groups can explain the outcome. Plausible rival hypotheses must be eliminated before we can trust a causal conclusion. Using some of the examples above, some plausible rival hypotheses are:

• Children of higher Socio-Economic Status (S.E.S.) have greater access to resources, are healthier, and have more appliances in their homes than children of lower S.E.S. Children raised in homes with more appliances get better grades than those with fewer appliances.
• Education reduces religiousity and people in countries with higher rates of poverty are less educated than those in countries with lower poverty. Religiousity is higher in countries with high poverty rates than in countries is less poverty.
• Summer heat and longer days cause ice cream sales to rise, violent crime to increase, and more people to engage in swimming and other water recreation more often and for longer time periods than fall, winter, and spring. As ice cream sales rise, so do violent crime and drowning deaths.

Sometimes, as in this case, experimental research is not possible. It is both difficult and somewhat unethical to randomly assign children to conditions which control their television and video game playing habits for long periods of time. So, correlational research is valuable. However, its limitations should be shouted from the rooftops, especially when findings have as great an effect on the choices people make as these potentially do.

There are certainly cases in which enough evidence exists from correlational research to suspect a causal connection among variables. In these cases, converging evidence from other methodological approaches helps us to determine if the requirements for causal inference are met. For example, very few people would deny that smoking causes cancer. Despite our inability to randomly assign subjects to smoke, the evidence from many different approaches converges to an undeniable conclusion. That is far from true regarding television watching, video game playing, and grades.

The overgeneralization to cause is only part of the bigger picture. It is the other little things which leave me with a sense that science fails at informing consumers of truths.

The study involved a very large sample, however, the measures were almost entirely self-report. Given that the participants were middle-school students, I find this design very disturbing and deem the findings unreliable. Grade performance was measured using a single, highly subjective/relative question:

“How would you describe your grades last year?”
(excellent, good, average, or below average)

Not only do children of this age not know what average grades are, but their answers will be based much more on how they feel about their grades than on the grades themselves.

The questions of how much time is spent watching television and playing video games would probably be somewhat accurate measures if answered by parents. However, children are rarely cognisant of time passage and do not estimate such things well. The authors attempted to control for confounding variables (which could explain the relationship) by measuring parenting style using, once again, the children’s answers to questions about their mothers. Finally, their control for S.E.S. included two factors: the children’s reports of whether or not their parents had graduated from high school and a weight based on the school’s demographic make-up. These are highly unlikely to be accurate enough measures to control for anything.

The results section of the journal article was unclear and suspicious. Their big finding was that weeknight television viewing and video game playing was correlated with lower grades. However, they did not report such a correlation. Instead they compared the proportion of students who said their grades were excellent across viewing times to the proportion of students who said their grades were below average. The figures they present illustrate these comparisons, but no statistics are given, despite complete tables of correlations among covariates (which are not of interest). What’s more is that the few p-values they do report are above what most would consider significant given the number of statistical tests they conducted (the more tests you run, the greater the probability that one will be significant by chance) and the effect sizes for most of the correlations and “risk factors” are extremely small, especially once they adjusted for those weak variables such as S.E.S. Given a large enough sample, many practically insignificant things may be found and that could be part of what is happening here, but I see absolutely nothing in this study which would lead me to the conclusion that cutting my children’s television time on school nights would improve their grades.

Finally, they excluded a very large portion of the participants (16.4%) due to missing data. There were significant differences in the key measures (time watching television, for example) between the participants who were excluded and those included as well as in the covariates of parenting style and S.E.S. A significant difference in the dependent variable (grade) was also found. The authors report these differences, but exclude the participants completely anyway and make no mention of it in their discussion. This is a very serious threat to the validity of all of their conclusions.

This study was big news. Google it and see for yourself. But what did it really tell us? My conclusion: it told us nothing. My decision: nothing in my household will change as a result of reading this study.

Sharif, I., & Sargent, J. (2006). Association Between Television, Movie, and Video Game Exposure and School Performance PEDIATRICS, 118 (4) DOI: 10.1542/peds.2005-2854

Comments for this page are closed.