I hope I grabbed your attention with that title, but do not expect to find the answer to that question here. What I am going to discuss today is a study that many people seem to think answers that question, but it doesn’t.

As I noted in my last post, the study I’ll be discussing was grossly misreported, starting with its press release. Since the study itself appears to be behind a pay wall for most people, I’ll describe as much detail as I can in a blog post as I discuss the study’s validity and findings of the study, published in the Journal of Social Psychological and Personality Science and titled “My Brother’s Keeper? Compassion Predicts Generosity More Among Less Religious Individuals”.

But for those who are not at all interested in the research methods or a breakdown of why I rate the quality of the study the way I do, I will give you the the bottom line so you can skip the rest or only read the sections that interest you (I’ve used headings to make it easier).

Summary

I think that the findings will hold up to replication, despite some issues I have with the way they did a few things. Overall, the research quality is quite high.

The groups they compared did not include atheists, agnostics, believers, non-believers, highly religious, or any other label that you can throw at it. In the studies they used raw religiosity scores and made some comparisons of “higher” and “lower” using values from the distribution. In a sense, the compared those who scored in the lower half of the sample to those who scored in the upper half.

They found:

• Differences in prosocial behavior cannot be dismissed as due to political affiliation, socio-economic status, or other factors often held up as responsible.
• Religiosity is correlated with trait compassion; the more religious, the more compassionate.
• Trait compassion is related to prosocial behavior in general. This relationship is stronger in the less religious than in the more religious. This does not mean that the less religious are more compassionate (see number 1) or that the less religious are more prosocial. It just means that compassion is a bigger factor in prosocial behavior in the less religious.
• The findings of the first study can be interpreted one way that isn’t discussed in the paper: when the relationship between compassion and religiosity is accounted for, the more religious are not more prosocial than the less religious.
• The findings in the second study, which involved inducing feelings of compassion, were similar for generosity, except that the more religious were more prosocial even after accounting for compassion.
• The findings of the second study also included a different pattern when the prosocial behavior was giving to charity. Compassion induced more giving, but the effect was weak and did not differ much across religiosity. Religiosity had a significant affect on charity. This can be explained by the guidelines provided by many churches for how much of one’s salary one should give.
• In the third study, in which state compassion (how compassionate the individual felt at that time) was measured and the prosocial behavior measure involved real-world cash, religiosity was not related to either compassion or prosocial behavior.
• In the third study, state compassion was positively correlated with prosocial behavior, but the effect was greater in the less religious than in the more religious.

What the findings as a whole say to me, and what I believe the press report tried, but failed, to express, at least with convincing support: We do not need religion to be prosocial. We need compassion.

This is great news for secularists.

However, it doesn’t say anything negative about religion or the religious, nor does it provide anything that should make atheists feel superior. It just shows that one can be good without God; that motivations can come from other sources.

Now on to the details…

NOTE: to keep this as short as possible, I’ve included a lot of links to terms and demonstrations. Where I describe problems in more detail I still water-down quite a bit. I will do my best to make it understandable without rambling on and on, but keep in mind that it takes many years to learn enough about research design and statistics to understand why some of these are problematic. Furthermore, not all researchers will agree on the consequences of some of these problems. I am still learning this stuff myself (probably always will be learning).

The Study (description)

The article reports three studies, each related to the relationship between compassion and prosocial behavior in less-religious individuals. I have created graphs using the information in the paper, but in some cases I did not have exact numbers, so while the relationships are visually accurate, there are only values where I could use exact numbers.

Theoretical Foundation

The introduction discusses research which documents that religiosity is associated with prosocial behavior. Specifically, religious people give more and volunteer more than nonreligious people, over and above what they give to and do for religious organizations. The researchers note that the nonreligious do give; when we compare groups, we do so using averages. However, it may be that the motivations for prosocial behavior vary in a way that interacts with religiosity. In other words, the more religious among us may be motivated to prosocial behavior by one set of factors and the less motivated by another.

The researchers hypothesized that compassion is a more influential factor in prosocial behavior for the less religious than for the more religious among us.

Study 1

The first study examined the relationships among religiosity and traits of compassion and prosocial tendencies. What this basically means is that situational factors were not involved; traits are a matter of personality or attitude. For example, “trait anxiety” refers to how anxious a person is in general, while “state anxiety” refers to how anxious that same individual feels in a given situation.

This study involved analyzing data from a 2004 “survey”. I put that term in quotes because it usually refers to a set of questions that do not measure more than what is apparent at face value. Established measures of latent variables (variables which cannot be measured directly such as feelings and attitudes) are usually called an “inventory” or “scale” and we refer to them loosely as “measures”. In this case, the survey involved such measures and I want to make that clear.

The sample was comprised of 1337 participants and covariates (variables other than those of interest which could explain differences among the groups) of gender, political orientation, and education were included in the analysis. The variables of interest were religiosity, compassion, and prosocial behavior. Religious identity (identification with a specific religion or no religion) was also considered.

Results

Correlations

• Covariates had little impact on the results.
• Trait compassion was positively correlated with religiosity* and prosocial behavior. On average, the more compassionate the individual, the more religious they were and the more the more prosocial they were.
• The relationship between religiosity and prosocial behavior was marginally significant (statistically).

Hypothesis Test (See Figure 1)

Figure 1: Study 1 results. "Higher" and "lower" are defined here as 1 SD from the mean.

• A regression analysis revealed an interaction of religiosity and compassion on prosocial behavior. What this means: The effect of compassion on prosocial behavior differed among levels of religiosity.
• More specifically, the level of trait compassion affected prosocial behavior less as religiosity increased.
• There was also a main effect of compassion, but that was apparent in the correlational analysis.
• There was no main effect of religiosity on prosocial behavior. This is interesting, because they found a marginally significant correlation, but it does not mean the there are no difference in prosocial behavior. I would interpret these findings, when put together, as suggestive of little or no difference between the more religious and the less religious in prosocial behavior over and above the differences accounted for by compassion.

The authors discuss the findings a little differently, though, focusing on the differences in the way that compassion affected prosocial behavior (the interaction in the first hypothesis test result) and ignoring the way that the effect of religiosity disappeared when compassion was entered into the equation. It seems more interesting to me to treat compassion as the moderator. It also makes more sense in the end.

Study 2

This study was experimental in that the researchers manipulated state compassion. In other words, they induced feelings of compassion in half of the participants and compared the amount of prosocial behavior those participants engaged in to the amount of such behavior in a control condition.

The sample included 101 participants and the study was conducted online, so the age range was exceptional (from 18 to 68 years). Participants were randomly assigned to one of the two conditions and each watched a short video under the guise that there would be a test of memory afterward.  The videos were established manipulations of feelings of compassion and neutral emotion (i.e., other researchers tested their effectiveness). Following the video, participants completed two tasks which are well-established measures of prosocial behavior commonly used in such research.

Results

• Again, covariates had little impact on the results.

Hypothesis Tests (See Figure 2)

Figure 3: Results of Study 3. Values are relative.

There were two tests since the participants completed to different prosocial tasks, one involving generosity and the other involving charity.

For the generosity task:

• This time there were a main effects of both religiosity and compassion on prosocial behavior. The more religious, the more prosocial. Those who watched the compassion-inducing video were the more prosocial on average than those who watched the neutral video.
• The interaction appeared again in the manner as in Study 1.

For the charity task:

• There were main effects of both religiosity and compassion on prosocial behavior.
• There was no interaction.

This is where they screw up, in my opinion.

The pattern of the moderation was in the predicted direction but failed to reach statistical significance.

This is not an acceptable statement unless the findings are marginal. This was not. The p-value was .408. This is not even close to meaningful. Still, they went ahead with the analysis of the interaction and reported an effect of compassion on charity for the less religious participants and no effect for the more religious.  The problem is that post-hoc analysis like this assumes that a significant interaction was observed. Their tests inflated alpha (the probability of a Type I Error) and can only mislead. They stated that they had found “partial support” for their hypothesis, but they did not in this case.

The relationships in the generosity task are very clear when we look at a Figure 2. The interaction is the interesting finding. Compassion had little effect on the more religious, but a very large effect on the less religious, who gave practically nothing when compassion was not induced. There is no analysis to tell us if the less religious surpassed the religious by a statistically significant amount when compassion was induced, but they were clearly out done by the more religious when not made to feel compassion.

The charity task showed no such interaction and the authors did not include a graph of this effect that I could recreate, nor did they provide the information to make one.

Study 3

For this study, the sample of 120 completed a state compassion inventory (a measure of their feelings of general compassion at the moment) and a series of “economic tasks designed to measure their generosity, trust, trustworthiness, and motivation to reward others’ generosity.” What differed in this study, however, was that the ‘points’ they earned in these tasks could be exchanged for cash at the end of the study. Participants did not know how much cash, but they knew that the more points they earned, the more cash they would receive.

Results

Hypothesis Tests (See Figure 3)

The findings of this study were very different from the other two.

• State compassion was not related to religiosity.
• Religiosity was not related to prosocial behavior.
• There was an interaction of religiosity and compassion on prosocial behavior. The amount of compassion felt had more of an effect on the behavior the less religious than it did on the more religious.

The graph of these findings, a reproduction of their graph since they did not provide information to create one that would make more sense (to me anyway), is a bit misleading. The values are z-scores, so they are relative to one another and not actual values. What is interesting, though is how little the prosocial score varied in the more religious group and how that line barely dips below the mean value (represented by 0).

There is also a problem with the press release in that it makes the claim that the high state compassion/less religious group out-performed the others. There is no statistical analysis comparing the groups in that way, so this is a misstatement. We do not know if less religious individuals are more generous than more religious when motivated to act prosocially. We just know that they are more generous when motivated by compassion than when compassion is low.

The Study Overall

As I noted, my opinion of the studies as a whole is relatively high, but I do have some major criticisms. Some of the language makes me cringe (e.g., results are the product of statistical tests, so “We tested our results”…), but I have seen more and more of this as scientific reports in general have grown sloppier.  Study design and method is much more important, as is the quality of the reporting beyond language.

The authors also throw around the term “robust”, claiming in the first study that the relationship between compassion and prosocial behavior is “particularly robust” for less religious individuals. That term refers to findings which are “sturdy” and will stand up when some supports are removed – effects which appear to hold up in different situations. Since this was one analysis of one data set, that term just doesn’t work. It does not fit in any of their uses of it.

In fact, they err in Study 2 by saying that the effect was “attenuated” for the more religious. That term is relative; attenuated compared to what? The effect was not “robust” in one condition and “attenuated” in another; they can only be compared to each other. The effect was greater in the less religious than the more religious.

Missing Information

There are a number of bits of information which are considered to be, at minimum, required for a good research report. A general rule of thumb for methods and results sections is to include enough (without being redundant) information to allow other researchers to replicate (in a strict sense) the study and to confirm that the statistical findings are properly interpreted.

I am not sure that this article meets that criterion. The methods are pretty well fleshed out and the paper is full of statistics, but some descriptive statistics are missing that I would have liked to have seen (e.g., means reported overall for measures, but not by group) and there was not enough of the right information to recreate them.

Grouping the Data and Errors of Generalization

One overall criticism which warrants discussion is in the grouping of data. There are some problems with this and they are related. The sensitivity of the religiosity measure is one problem that, by itself, is not a big target for criticism. Combined with the second problem of grouping participants, though, it becomes more serious.

The practice of comparing groups of people based on a variable which is distributed on a spectrum is a common one. The question the researcher wants to answer is important in deciding whether to group and, in this case, I do not disagree with that choice, but I question how they grouped and how it was communicated. If the data are clustered (the distribution is multi-modal), grouping is simplified, but if the data are distributed more loosely, it can be tricky and dangerous.

First, the researcher loses information, therefore they lose sensitivity and usually lose power. The sensitivity problem is relevant in the first study, but mostly because it makes the findings difficult to interpret.

Second, if the way that the grouping is communicated is not consistent and clear, it is likely to be misinterpreted, compounding any existing problems with the method. I discussed this problem in my last post. Most of the reports referred to the groups compared as “highly religious” verses “atheists and agnostics” or something like that. However, where are all of the people in the middle (i.e., most likely the bulk of the sample)?  Within each group there was variation in religiosity and comparisons are made using averages. Generalizing only works when the samples are representative of the population of interest and this applies in either direction of the generalization (i.e., specific to mixed or mixed to specific).

Third, researchers must decide where to draw the lines between high and low (and anything in between). Since the majority of variables in psychology are normal distributed (therefore symmetrical), the lines are usually drawn using rankings of sample values and the most common way to split a sample in half is to put all values above the median into “higher” and those below into “lower” (called a “median split”). However, ease is not a good reason to use this technique.   Here is an interesting demonstration of the dangers of dichotomizing normally-distributed variables.

But… religiosity is not usually distributed normally; it’s usually skewed. Skew means that it’s not symmetrical, so a median-split would make even less sense.

In this case, it seems that the authors tried to have the best of both worlds by treating religiosity as a continuous variable, but doing post-hoc analysis on it, discussing it, and graphing it as if it were dichotomous, choosing values which were one standard deviation from the mean in both directions as the central tendencies of each group. The biggest problem with this is the assumption of normality. If the variable is not normally-distributed (and I suspect that it is not), this grouping is a bit tough to swallow.

When this problem is mixed with a limited range as it is in the first study (the religiosity scale only had four points), it’s a problem. The four possible values were 1 = no religion, 2 = not very strong (religious identity), 3 = somewhat strong, and 4 = strong. Since the mean was 2.99, the bulk of the sample were fairly religious. one standard deviation (1.03) below the mean is not exactly in non-believerland and one above is off the scale (literally). It is just very difficult to see where “higher” leaves off and “lower” takes over.

Although the range is adequate in the other two studies, the problem of discussing groups which do not actually exist and have fuzzy definitions remains. In my opinion that is one of the reasons it was so misreported.

But, overall, the research is of a relatively high quality and interesting. I would like to see more variation in the prosocial tasks, given that the outcome of the charity task was so different from the tasks of generosity.

It seems that the less religious are at least as generous as the more religious, but their reasons for acting prosocially differ. I would like to see the day when, as a group, we are generous and prosocial consistently, without the need to be provoked and without needing to feel an emotional connection to the receiver.

Saslow, L., Willer, R., Feinberg, M., Piff, P., Clark, K., Keltner, D., & Saturn, S. (2012). My Brother’s Keeper? Compassion Predicts Generosity More Among Less Religious Individuals Social Psychological and Personality Science DOI: 10.1177/1948550612444137

It seems that the best motivation for me (to write) is frustration.

A press release by UC Berkeley about a study that was recently published on the relationships among religiosity, compassion, and prosocial behavior has been making the rounds over the last couple of days, waved by proud atheists as evidence of superiority and bashed by the more skeptical as bad science (even though most haven’t appeared to have read the study). The latter has been exacerbated by the fact that the majority of reports include very big mistakes.

I’m going to analyze the study, but I will post that separately since it is likely to be long and I also want to discuss the incredible mess of BS this has become. In that post I will also discuss the reasons some of the errors that seem minor here are actually quite serious.

I blame the press release. Unfortunately, as the culture of higher education becomes more consumer-oriented, strapped-for-cash universities must market themselves strategically, and scientists are forced to compete for funding, the integrity of science is diminished. I don’t know if the study’s authors were involved, but my guess is that the press release’s author is mostly responsible for spinning the findings, omitting important parts of the findings, and cherry-picking statements by the scientists to make it appear that their speculative explanations for those findings are solid conclusions. I think that spin, along with the fact that the findings involve an interaction (a notoriously difficult concept to grasp due to its non-linear nature), confused science writers, many of whom are not in the practice of reading studies and many of whom added their own spin to the mix.

The study, published in the Journal of Social Psychological and Personality Science, is titled “My Brother’s Keeper? Compassion Predicts Generosity More Among Less Religious Individuals”.  The first error of the press release was made in its title: “Highly religious people are less motivated by compassion than are non-believers” by incorrectly describing the subjects of the study. It also errs in a few minor ways (e.g., describing the study as “three experiments” when one only one was experimental, one was analysis of existing data, and one was quasi experimental), but the biggest problems are practically criminal in the science world. The author correctly (with the exception of the description of the subjects) reported that the relationship between compassion and prosocial behavior was stronger among less religious participants than more religious participants, but omitted the findings which clearly showed that the more religious participants were, in general, more compassionate and generous overall.

When sloppy reporting took over, the result was an utter mess.

On the one hand, the authors claim in their introduction to be interested in what motivates less religious people to act prosocially, so perhaps the spin was the plan all along. However, if they were not interested in religiosity as a variable – if they were not interested in comparing the more religious to the less religious, then they should not have limited their study to the population of interest.

Something I found interesting is that the third paragraph of the press release makes a statement which should have sent red flags up because there was no follow-up that made sense. It started with:

The results challenge a widespread assumption that acts of generosity and charity are largely driven by feelings of empathy and compassion, researchers said.

And yet the rest of the piece focused on the fact that empathy and compassion DID drive generosity and charity in less religious participants, even if it repeatedly incorrectly referred to those participants as “non-believers”. If the author had included the findings that more religious participants were more generous, this would have made sense.

The author errs again with this circular definition of “compassion”, conflating the dictionary definitions of “compassion” and “prosocial behavior” with an effect:

Compassion is defined in the study as an emotion felt when people see the suffering of others which then motivates them to help, often at a personal risk or cost.

Compassion cannot be defined in a study as an emotion felt by someone. That’s a variable we can’t measure directly. In a study, we use operational definitions. In this case, compassion was defined differently in each of the three studies within the article.

The author continues the spin with statements such as (bold mine):

When they looked into how much compassion motivated participants to be charitable in such ways as giving money or food to a homeless person, non-believers and those who rated low in religiosity came out ahead: “These findings indicate that although compassion is associated with pro-sociality among both less religious and more religious individuals, this relationship is particularly robust for less religious individuals,” the study found.

What the bold suggests is that participants who fell into the “low religiosity” category were more prosocial and/or more compassionate. That’s not what the finding means. What was greater was simply the relationship between compassion and prosocial behavior. Because the “high religiosity” participants were more compassionate, they were also more prosocial overall (marginal significance – see discussion below).

Finally, this statement is grossly misleading:

Those who scored low on the religiosity scale, and high on momentary compassion, were more inclined to share their winnings with strangers than other participants in the study.

What participants do in a study is really not interesting. What their behavior tells us about how people behave in the world is. Because there was no statistical analysis comparing the “low-religiosity/high-momentary compassion” group to the other groups, this finding does not allow us to infer anything about the population of interest. You and I could complete a game of Scrabble with a final score of 102 to 103 (respectively), but I would not brag to my friends that I am the better Scrabble player.

Most reports of the study either posted the press release as-is or quoted large chunks of it. Headlines ranged from the simple and correct, if misleading, “Compassion may motivate faithful less” to the still incorrect, but closer “Confirmed: Atheists more motivated by compassion in charitable giving than believers are” to the blatantly incorrect statement suggested by, “Are Religious People Less Compassionate?”

Almost none of the reports describe the samples correctly. The samples were broken into two groups based on measured religiosity, a common practice in social psychology. Thus, there was a range of values and those who scored in the top half were considered “more religious” and the bottom half “less religious”. The “less religious” group cannot be described as “athiests”, “agnostics”, “non-believers”, or even “people low in religiosity”. The “more religious” group cannot be described as “highly religious”, either.

One site that I usually find more accurate contained a much more serious error than an incorrect sample description. It reported a finding not found in the press release that is actually the opposite of what was reported in the study:

Two other experiments also confirmed that more religious participants seemed to be less generous.

They should have stuck to quotes of the press release.

The worst by far, though was on a site called “PsychCentral“. The first paragraph reads:

A provocative new study from the University of California, Berkeley suggests highly religious individuals are less likely to help a stranger than less religious people.

Um. No. That’s not what the study suggests. At all.

and this a few paragraphs down:

Experts say the results challenge a widespread assumption that acts of generosity and charity are largely driven by feelings of empathy and compassion.

Why they added “experts say” to this is a mystery because it’s basically a lie, but this statement is particularly baffling when you consider that the progression of statements: 1) nonreligious are more generous 2) nonreligious are more motivated by compassion to be generous 3) results challenge the assumption that generosity is driven by compassion. On what planet does that make sense?

I’d lament that people don’t actually read, but the worst part is that someone actually wrote this. Sure, it’s mostly quotes and they made stuff up, but they changed some words and moved things around, so they had to pay attention to something.

A friend shared one decent report; it did not seem that the author read the journal article, but at least he read one report (on livescience.com) that is almost a duplication of the press release and put some thought into it. He noted that the findings reported an interaction – that generosity was more related to compassion among the less religious than it was among the more religious – and questioned who was more generous overall. If the press released had not omitted those findings, he would have had an answer, or at least a theoretically-likely hypothesis. Still, he did a little bit of research on his own and noted some well-known findings that the religious tend to give more to secular charities than atheists.

Finally, the press release included a few statements by the authors which were highly speculative. It is standard for authors to discuss possible explanations for their findings, but they are often presented to the public as conclusions the authors reached. In this case as in many others, reports of the study often imply that these things are findings when the study did not examine them at all. For example, the last paragraph of the press release is:

“Overall, this research suggests that although less religious people tend to be less trusted in the U.S., when feeling compassionate, they may actually be more inclined to help their fellow citizens than more religious people,” Willer said.

The rawstory.com report ended the same way, except they changed the last word to “concluded”. This statement serves as a hypothesis for another study, but it is not a finding of the study and cannot provide a conclusion.

I have heard a lot of explanations of what motivates more religious people, but most have been based on personal beliefs or experiences – duty to God, moral obligation, sense of community, etc.  This question may have been answered and looking to other studies might yield something, but this study does not address it – it did not set out to address that question.

A full analysis of the journal article will follow in a separate post later today.

UPDATE: Analysis can be found here.
ANOTHER UPDATE: The award for the most botched report now goes to MSN for this doosey.