Science and Spin Are Very Bad Bedfellows

It seems that the best motivation for me (to write) is frustration.

A press release by UC Berkeley about a study that was recently published on the relationships among religiosity, compassion, and prosocial behavior has been making the rounds over the last couple of days, waved by proud atheists as evidence of superiority and bashed by the more skeptical as bad science (even though most haven’t appeared to have read the study). The latter has been exacerbated by the fact that the majority of reports include very big mistakes.

I’m going to analyze the study, but I will post that separately since it is likely to be long and I also want to discuss the incredible mess of BS this has become. In that post I will also discuss the reasons some of the errors that seem minor here are actually quite serious.

I blame the press release. Unfortunately, as the culture of higher education becomes more consumer-oriented, strapped-for-cash universities must market themselves strategically, and scientists are forced to compete for funding, the integrity of science is diminished. I don’t know if the study’s authors were involved, but my guess is that the press release’s author is mostly responsible for spinning the findings, omitting important parts of the findings, and cherry-picking statements by the scientists to make it appear that their speculative explanations for those findings are solid conclusions. I think that spin, along with the fact that the findings involve an interaction (a notoriously difficult concept to grasp due to its non-linear nature), confused science writers, many of whom are not in the practice of reading studies and many of whom added their own spin to the mix.

The study, published in the Journal of Social Psychological and Personality Science, is titled “My Brother’s Keeper? Compassion Predicts Generosity More Among Less Religious Individuals”.  The first error of the press release was made in its title: “Highly religious people are less motivated by compassion than are non-believers” by incorrectly describing the subjects of the study. It also errs in a few minor ways (e.g., describing the study as “three experiments” when one only one was experimental, one was analysis of existing data, and one was quasi experimental), but the biggest problems are practically criminal in the science world. The author correctly (with the exception of the description of the subjects) reported that the relationship between compassion and prosocial behavior was stronger among less religious participants than more religious participants, but omitted the findings which clearly showed that the more religious participants were, in general, more compassionate and generous overall.

When sloppy reporting took over, the result was an utter mess.

On the one hand, the authors claim in their introduction to be interested in what motivates less religious people to act prosocially, so perhaps the spin was the plan all along. However, if they were not interested in religiosity as a variable – if they were not interested in comparing the more religious to the less religious, then they should not have limited their study to the population of interest.

Something I found interesting is that the third paragraph of the press release makes a statement which should have sent red flags up because there was no follow-up that made sense. It started with:

The results challenge a widespread assumption that acts of generosity and charity are largely driven by feelings of empathy and compassion, researchers said.

And yet the rest of the piece focused on the fact that empathy and compassion DID drive generosity and charity in less religious participants, even if it repeatedly incorrectly referred to those participants as “non-believers”. If the author had included the findings that more religious participants were more generous, this would have made sense.

The author errs again with this circular definition of “compassion”, conflating the dictionary definitions of “compassion” and “prosocial behavior” with an effect:

Compassion is defined in the study as an emotion felt when people see the suffering of others which then motivates them to help, often at a personal risk or cost.

Compassion cannot be defined in a study as an emotion felt by someone. That’s a variable we can’t measure directly. In a study, we use operational definitions. In this case, compassion was defined differently in each of the three studies within the article.

The author continues the spin with statements such as (bold mine):

When they looked into how much compassion motivated participants to be charitable in such ways as giving money or food to a homeless person, non-believers and those who rated low in religiosity came out ahead: “These findings indicate that although compassion is associated with pro-sociality among both less religious and more religious individuals, this relationship is particularly robust for less religious individuals,” the study found.

What the bold suggests is that participants who fell into the “low religiosity” category were more prosocial and/or more compassionate. That’s not what the finding means. What was greater was simply the relationship between compassion and prosocial behavior. Because the “high religiosity” participants were more compassionate, they were also more prosocial overall (marginal significance – see discussion below).

Finally, this statement is grossly misleading:

Those who scored low on the religiosity scale, and high on momentary compassion, were more inclined to share their winnings with strangers than other participants in the study.

What participants do in a study is really not interesting. What their behavior tells us about how people behave in the world is. Because there was no statistical analysis comparing the “low-religiosity/high-momentary compassion” group to the other groups, this finding does not allow us to infer anything about the population of interest. You and I could complete a game of Scrabble with a final score of 102 to 103 (respectively), but I would not brag to my friends that I am the better Scrabble player.

Most reports of the study either posted the press release as-is or quoted large chunks of it. Headlines ranged from the simple and correct, if misleading, “Compassion may motivate faithful less” to the still incorrect, but closer “Confirmed: Atheists more motivated by compassion in charitable giving than believers are” to the blatantly incorrect statement suggested by, “Are Religious People Less Compassionate?”

Almost none of the reports describe the samples correctly. The samples were broken into two groups based on measured religiosity, a common practice in social psychology. Thus, there was a range of values and those who scored in the top half were considered “more religious” and the bottom half “less religious”. The “less religious” group cannot be described as “athiests”, “agnostics”, “non-believers”, or even “people low in religiosity”. The “more religious” group cannot be described as “highly religious”, either.

One site that I usually find more accurate contained a much more serious error than an incorrect sample description. It reported a finding not found in the press release that is actually the opposite of what was reported in the study:

Two other experiments also confirmed that more religious participants seemed to be less generous.

They should have stuck to quotes of the press release.

The worst by far, though was on a site called “PsychCentral“. The first paragraph reads:

A provocative new study from the University of California, Berkeley suggests highly religious individuals are less likely to help a stranger than less religious people.

Um. No. That’s not what the study suggests. At all.

and this a few paragraphs down:

Experts say the results challenge a widespread assumption that acts of generosity and charity are largely driven by feelings of empathy and compassion.

Why they added “experts say” to this is a mystery because it’s basically a lie, but this statement is particularly baffling when you consider that the progression of statements: 1) nonreligious are more generous 2) nonreligious are more motivated by compassion to be generous 3) results challenge the assumption that generosity is driven by compassion. On what planet does that make sense?

I’d lament that people don’t actually read, but the worst part is that someone actually wrote this. Sure, it’s mostly quotes and they made stuff up, but they changed some words and moved things around, so they had to pay attention to something.

A friend shared one decent report; it did not seem that the author read the journal article, but at least he read one report (on that is almost a duplication of the press release and put some thought into it. He noted that the findings reported an interaction – that generosity was more related to compassion among the less religious than it was among the more religious – and questioned who was more generous overall. If the press released had not omitted those findings, he would have had an answer, or at least a theoretically-likely hypothesis. Still, he did a little bit of research on his own and noted some well-known findings that the religious tend to give more to secular charities than atheists.

Finally, the press release included a few statements by the authors which were highly speculative. It is standard for authors to discuss possible explanations for their findings, but they are often presented to the public as conclusions the authors reached. In this case as in many others, reports of the study often imply that these things are findings when the study did not examine them at all. For example, the last paragraph of the press release is:

“Overall, this research suggests that although less religious people tend to be less trusted in the U.S., when feeling compassionate, they may actually be more inclined to help their fellow citizens than more religious people,” Willer said.

The report ended the same way, except they changed the last word to “concluded”. This statement serves as a hypothesis for another study, but it is not a finding of the study and cannot provide a conclusion.

I have heard a lot of explanations of what motivates more religious people, but most have been based on personal beliefs or experiences – duty to God, moral obligation, sense of community, etc.  This question may have been answered and looking to other studies might yield something, but this study does not address it – it did not set out to address that question.

A full analysis of the journal article will follow in a separate post later today.

UPDATE: Analysis can be found here.
ANOTHER UPDATE: The award for the most botched report now goes to MSN for this doosey.

Print Friendly, PDF & Email

4 comments to Science and Spin Are Very Bad Bedfellows