I hope I grabbed your attention with that title, but do not expect to find the answer to that question here. What I am going to discuss today is a study that many people seem to think answers that question, but it doesn’t.

As I noted in my last post, the study I’ll be discussing was grossly misreported, starting with its press release. Since the study itself appears to be behind a pay wall for most people, I’ll describe as much detail as I can in a blog post as I discuss the study’s validity and findings of the study, published in the Journal of Social Psychological and Personality Science and titled “My Brother’s Keeper? Compassion Predicts Generosity More Among Less Religious Individuals”.

But for those who are not at all interested in the research methods or a breakdown of why I rate the quality of the study the way I do, I will give you the the bottom line so you can skip the rest or only read the sections that interest you (I’ve used headings to make it easier).

Summary

I think that the findings will hold up to replication, despite some issues I have with the way they did a few things. Overall, the research quality is quite high.

The groups they compared did not include atheists, agnostics, believers, non-believers, highly religious, or any other label that you can throw at it. In the studies they used raw religiosity scores and made some comparisons of “higher” and “lower” using values from the distribution. In a sense, the compared those who scored in the lower half of the sample to those who scored in the upper half.

They found:

• Differences in prosocial behavior cannot be dismissed as due to political affiliation, socio-economic status, or other factors often held up as responsible.
• Religiosity is correlated with trait compassion; the more religious, the more compassionate.
• Trait compassion is related to prosocial behavior in general. This relationship is stronger in the less religious than in the more religious. This does not mean that the less religious are more compassionate (see number 1) or that the less religious are more prosocial. It just means that compassion is a bigger factor in prosocial behavior in the less religious.
• The findings of the first study can be interpreted one way that isn’t discussed in the paper: when the relationship between compassion and religiosity is accounted for, the more religious are not more prosocial than the less religious.
• The findings in the second study, which involved inducing feelings of compassion, were similar for generosity, except that the more religious were more prosocial even after accounting for compassion.
• The findings of the second study also included a different pattern when the prosocial behavior was giving to charity. Compassion induced more giving, but the effect was weak and did not differ much across religiosity. Religiosity had a significant affect on charity. This can be explained by the guidelines provided by many churches for how much of one’s salary one should give.
• In the third study, in which state compassion (how compassionate the individual felt at that time) was measured and the prosocial behavior measure involved real-world cash, religiosity was not related to either compassion or prosocial behavior.
• In the third study, state compassion was positively correlated with prosocial behavior, but the effect was greater in the less religious than in the more religious.

What the findings as a whole say to me, and what I believe the press report tried, but failed, to express, at least with convincing support: We do not need religion to be prosocial. We need compassion.

This is great news for secularists.

However, it doesn’t say anything negative about religion or the religious, nor does it provide anything that should make atheists feel superior. It just shows that one can be good without God; that motivations can come from other sources.

Now on to the details…

NOTE: to keep this as short as possible, I’ve included a lot of links to terms and demonstrations. Where I describe problems in more detail I still water-down quite a bit. I will do my best to make it understandable without rambling on and on, but keep in mind that it takes many years to learn enough about research design and statistics to understand why some of these are problematic. Furthermore, not all researchers will agree on the consequences of some of these problems. I am still learning this stuff myself (probably always will be learning).

The Study (description)

The article reports three studies, each related to the relationship between compassion and prosocial behavior in less-religious individuals. I have created graphs using the information in the paper, but in some cases I did not have exact numbers, so while the relationships are visually accurate, there are only values where I could use exact numbers.

Theoretical Foundation

The introduction discusses research which documents that religiosity is associated with prosocial behavior. Specifically, religious people give more and volunteer more than nonreligious people, over and above what they give to and do for religious organizations. The researchers note that the nonreligious do give; when we compare groups, we do so using averages. However, it may be that the motivations for prosocial behavior vary in a way that interacts with religiosity. In other words, the more religious among us may be motivated to prosocial behavior by one set of factors and the less motivated by another.

The researchers hypothesized that compassion is a more influential factor in prosocial behavior for the less religious than for the more religious among us.

Study 1

The first study examined the relationships among religiosity and traits of compassion and prosocial tendencies. What this basically means is that situational factors were not involved; traits are a matter of personality or attitude. For example, “trait anxiety” refers to how anxious a person is in general, while “state anxiety” refers to how anxious that same individual feels in a given situation.

This study involved analyzing data from a 2004 “survey”. I put that term in quotes because it usually refers to a set of questions that do not measure more than what is apparent at face value. Established measures of latent variables (variables which cannot be measured directly such as feelings and attitudes) are usually called an “inventory” or “scale” and we refer to them loosely as “measures”. In this case, the survey involved such measures and I want to make that clear.

The sample was comprised of 1337 participants and covariates (variables other than those of interest which could explain differences among the groups) of gender, political orientation, and education were included in the analysis. The variables of interest were religiosity, compassion, and prosocial behavior. Religious identity (identification with a specific religion or no religion) was also considered.

Results

Correlations

• Covariates had little impact on the results.
• Trait compassion was positively correlated with religiosity* and prosocial behavior. On average, the more compassionate the individual, the more religious they were and the more the more prosocial they were.
• The relationship between religiosity and prosocial behavior was marginally significant (statistically).

Hypothesis Test (See Figure 1)

Figure 1: Study 1 results. "Higher" and "lower" are defined here as 1 SD from the mean.

• A regression analysis revealed an interaction of religiosity and compassion on prosocial behavior. What this means: The effect of compassion on prosocial behavior differed among levels of religiosity.
• More specifically, the level of trait compassion affected prosocial behavior less as religiosity increased.
• There was also a main effect of compassion, but that was apparent in the correlational analysis.
• There was no main effect of religiosity on prosocial behavior. This is interesting, because they found a marginally significant correlation, but it does not mean the there are no difference in prosocial behavior. I would interpret these findings, when put together, as suggestive of little or no difference between the more religious and the less religious in prosocial behavior over and above the differences accounted for by compassion.

The authors discuss the findings a little differently, though, focusing on the differences in the way that compassion affected prosocial behavior (the interaction in the first hypothesis test result) and ignoring the way that the effect of religiosity disappeared when compassion was entered into the equation. It seems more interesting to me to treat compassion as the moderator. It also makes more sense in the end.

Study 2

This study was experimental in that the researchers manipulated state compassion. In other words, they induced feelings of compassion in half of the participants and compared the amount of prosocial behavior those participants engaged in to the amount of such behavior in a control condition.

The sample included 101 participants and the study was conducted online, so the age range was exceptional (from 18 to 68 years). Participants were randomly assigned to one of the two conditions and each watched a short video under the guise that there would be a test of memory afterward.  The videos were established manipulations of feelings of compassion and neutral emotion (i.e., other researchers tested their effectiveness). Following the video, participants completed two tasks which are well-established measures of prosocial behavior commonly used in such research.

Results

• Again, covariates had little impact on the results.

Hypothesis Tests (See Figure 2)

Figure 3: Results of Study 3. Values are relative.

There were two tests since the participants completed to different prosocial tasks, one involving generosity and the other involving charity.

For the generosity task:

• This time there were a main effects of both religiosity and compassion on prosocial behavior. The more religious, the more prosocial. Those who watched the compassion-inducing video were the more prosocial on average than those who watched the neutral video.
• The interaction appeared again in the manner as in Study 1.

For the charity task:

• There were main effects of both religiosity and compassion on prosocial behavior.
• There was no interaction.

This is where they screw up, in my opinion.

The pattern of the moderation was in the predicted direction but failed to reach statistical significance.

This is not an acceptable statement unless the findings are marginal. This was not. The p-value was .408. This is not even close to meaningful. Still, they went ahead with the analysis of the interaction and reported an effect of compassion on charity for the less religious participants and no effect for the more religious.  The problem is that post-hoc analysis like this assumes that a significant interaction was observed. Their tests inflated alpha (the probability of a Type I Error) and can only mislead. They stated that they had found “partial support” for their hypothesis, but they did not in this case.

The relationships in the generosity task are very clear when we look at a Figure 2. The interaction is the interesting finding. Compassion had little effect on the more religious, but a very large effect on the less religious, who gave practically nothing when compassion was not induced. There is no analysis to tell us if the less religious surpassed the religious by a statistically significant amount when compassion was induced, but they were clearly out done by the more religious when not made to feel compassion.

The charity task showed no such interaction and the authors did not include a graph of this effect that I could recreate, nor did they provide the information to make one.

Study 3

For this study, the sample of 120 completed a state compassion inventory (a measure of their feelings of general compassion at the moment) and a series of “economic tasks designed to measure their generosity, trust, trustworthiness, and motivation to reward others’ generosity.” What differed in this study, however, was that the ‘points’ they earned in these tasks could be exchanged for cash at the end of the study. Participants did not know how much cash, but they knew that the more points they earned, the more cash they would receive.

Results

Hypothesis Tests (See Figure 3)

The findings of this study were very different from the other two.

• State compassion was not related to religiosity.
• Religiosity was not related to prosocial behavior.
• There was an interaction of religiosity and compassion on prosocial behavior. The amount of compassion felt had more of an effect on the behavior the less religious than it did on the more religious.

The graph of these findings, a reproduction of their graph since they did not provide information to create one that would make more sense (to me anyway), is a bit misleading. The values are z-scores, so they are relative to one another and not actual values. What is interesting, though is how little the prosocial score varied in the more religious group and how that line barely dips below the mean value (represented by 0).

There is also a problem with the press release in that it makes the claim that the high state compassion/less religious group out-performed the others. There is no statistical analysis comparing the groups in that way, so this is a misstatement. We do not know if less religious individuals are more generous than more religious when motivated to act prosocially. We just know that they are more generous when motivated by compassion than when compassion is low.

The Study Overall

As I noted, my opinion of the studies as a whole is relatively high, but I do have some major criticisms. Some of the language makes me cringe (e.g., results are the product of statistical tests, so “We tested our results”…), but I have seen more and more of this as scientific reports in general have grown sloppier.  Study design and method is much more important, as is the quality of the reporting beyond language.

The authors also throw around the term “robust”, claiming in the first study that the relationship between compassion and prosocial behavior is “particularly robust” for less religious individuals. That term refers to findings which are “sturdy” and will stand up when some supports are removed – effects which appear to hold up in different situations. Since this was one analysis of one data set, that term just doesn’t work. It does not fit in any of their uses of it.

In fact, they err in Study 2 by saying that the effect was “attenuated” for the more religious. That term is relative; attenuated compared to what? The effect was not “robust” in one condition and “attenuated” in another; they can only be compared to each other. The effect was greater in the less religious than the more religious.

Missing Information

There are a number of bits of information which are considered to be, at minimum, required for a good research report. A general rule of thumb for methods and results sections is to include enough (without being redundant) information to allow other researchers to replicate (in a strict sense) the study and to confirm that the statistical findings are properly interpreted.

I am not sure that this article meets that criterion. The methods are pretty well fleshed out and the paper is full of statistics, but some descriptive statistics are missing that I would have liked to have seen (e.g., means reported overall for measures, but not by group) and there was not enough of the right information to recreate them.

Grouping the Data and Errors of Generalization

One overall criticism which warrants discussion is in the grouping of data. There are some problems with this and they are related. The sensitivity of the religiosity measure is one problem that, by itself, is not a big target for criticism. Combined with the second problem of grouping participants, though, it becomes more serious.

The practice of comparing groups of people based on a variable which is distributed on a spectrum is a common one. The question the researcher wants to answer is important in deciding whether to group and, in this case, I do not disagree with that choice, but I question how they grouped and how it was communicated. If the data are clustered (the distribution is multi-modal), grouping is simplified, but if the data are distributed more loosely, it can be tricky and dangerous.

First, the researcher loses information, therefore they lose sensitivity and usually lose power. The sensitivity problem is relevant in the first study, but mostly because it makes the findings difficult to interpret.

Second, if the way that the grouping is communicated is not consistent and clear, it is likely to be misinterpreted, compounding any existing problems with the method. I discussed this problem in my last post. Most of the reports referred to the groups compared as “highly religious” verses “atheists and agnostics” or something like that. However, where are all of the people in the middle (i.e., most likely the bulk of the sample)?  Within each group there was variation in religiosity and comparisons are made using averages. Generalizing only works when the samples are representative of the population of interest and this applies in either direction of the generalization (i.e., specific to mixed or mixed to specific).

Third, researchers must decide where to draw the lines between high and low (and anything in between). Since the majority of variables in psychology are normal distributed (therefore symmetrical), the lines are usually drawn using rankings of sample values and the most common way to split a sample in half is to put all values above the median into “higher” and those below into “lower” (called a “median split”). However, ease is not a good reason to use this technique.   Here is an interesting demonstration of the dangers of dichotomizing normally-distributed variables.

But… religiosity is not usually distributed normally; it’s usually skewed. Skew means that it’s not symmetrical, so a median-split would make even less sense.

In this case, it seems that the authors tried to have the best of both worlds by treating religiosity as a continuous variable, but doing post-hoc analysis on it, discussing it, and graphing it as if it were dichotomous, choosing values which were one standard deviation from the mean in both directions as the central tendencies of each group. The biggest problem with this is the assumption of normality. If the variable is not normally-distributed (and I suspect that it is not), this grouping is a bit tough to swallow.

When this problem is mixed with a limited range as it is in the first study (the religiosity scale only had four points), it’s a problem. The four possible values were 1 = no religion, 2 = not very strong (religious identity), 3 = somewhat strong, and 4 = strong. Since the mean was 2.99, the bulk of the sample were fairly religious. one standard deviation (1.03) below the mean is not exactly in non-believerland and one above is off the scale (literally). It is just very difficult to see where “higher” leaves off and “lower” takes over.

Although the range is adequate in the other two studies, the problem of discussing groups which do not actually exist and have fuzzy definitions remains. In my opinion that is one of the reasons it was so misreported.

But, overall, the research is of a relatively high quality and interesting. I would like to see more variation in the prosocial tasks, given that the outcome of the charity task was so different from the tasks of generosity.

It seems that the less religious are at least as generous as the more religious, but their reasons for acting prosocially differ. I would like to see the day when, as a group, we are generous and prosocial consistently, without the need to be provoked and without needing to feel an emotional connection to the receiver.

Saslow, L., Willer, R., Feinberg, M., Piff, P., Clark, K., Keltner, D., & Saturn, S. (2012). My Brother’s Keeper? Compassion Predicts Generosity More Among Less Religious Individuals Social Psychological and Personality Science DOI: 10.1177/1948550612444137

Each semester I teach at least one course with a co-requisite laboratory course in which students conduct psychological research in small groups. Due to certain requirements of the American Psychological Association, these studies are not eligible for publication in an approved journal. Although students sometimes meet the requirements and re-run these studies in order to present or publish the findings, the original studies are considered pilot studies.

Some of these studies are very well-designed and executed and the findings are often interesting new discoveries. So, I have decided that it is about time I shared some of them and I will begin today with a timely finding about impression formation.

This semester, Brittany Reid, Lisa Aguilar, and Nare Setyan were interested in factors involved in judgments of intelligence and credibility as applied to marketing and image management. Specifically, they wondered if a hedonistic culture (party attitude) promoted by a group resulted in lower judgments of intelligence and credibility than traditional cultures. In other words, if you advertise that you like to party, will people think that you are less intelligent?
There is very little in the scientific literature regarding the assumptions people make about the relationship between hedonic behavior and intelligence. In fact, we could find none. There are mixed findings regarding the factors involved in judgments of intelligence. Most find that men are judged more intelligent than women, although no practical sex differences exist in general measures. Some studies have found that unattractive men are judged as more intelligent than attractive men and that the reverse is true for women. Many studies have found the opposite. Some have even suggested that attractive people are judged as more intelligent than unattractive people because they are more intelligent.

One club, two websites: a traditional academic site (left) and one which emphasizes a 'party' attitude (right)

We know that the way people dress, the number of piercings, the number of tattoos and all sorts of other things affect our judgments of people’s intelligence, competence, and a host of other attributes. The truth is, in the absence of other information and in some cases even when explicit information (e.g., about intelligence) is provided, appearances matter. So what about behavior? How does that affect the impressions people form of others?

The researchers hypothesized that the creators of a hedonistic (party attitude) group are judged as less intelligent and less credible than those of a traditional group. To test this hypothesis, they created two websites for a university psychology club.

Appeared on the home page of both versions.

• The main photo on the front page was clearly taken at a party whereas the main photo on the traditional front page was a group of students.
• The “spring break” photo gallery contained photos of parties and women in bikinis and people drinking at parties. The same gallery on the traditional site contained photos of students building houses for a charity. The events galleries included similar, but more subtle differences in the activities depicted.
• The executive board’s biographies focused on casual attributes whereas the traditional board’s bios discussed science.

Participants (28 of them; the responses of men and women did not differ in any of the measures) were randomly assigned to one of the two conditions. They were instructed to study the website for 5 minutes and that they would be asked to remember it later. Afterward, they navigated away from the website and completed a survey about the site. They were asked to indicate the degree to which they agreed with each of 10 statements using a 5-point Likert scale. The statements were:

• There should be more sites of this nature.
• This website promotes education.
• I would visit this website again.
• This website promotes “a good time”.
• I would recommend this site to others.
• I trust this site for credible information.
• The creator of this website is intelligent.
• This site is boring.
• I would view this site in my own time.
• I learned something new from viewing this website.

I will not bore you with the statistical output, but for those interested, I will note the following: The hypotheses were tested though independent samples t-tests comparing the judgments for numbers 6 and 7, then responses to all questions were also compared. The additional analysis is exploratory, but adjustments to alpha would only change the outcome of #10, which resulted in a p-value of .01. All other p-values are less than or equal to .001.

The findings are best illustrated by listing the agreements which did and did not differ among the groups. Tests of the research hypotheses appear in bold. The largest difference was in responses to question number 1:

So, the presence of a ‘party attitude’ did not affect evaluations of the site itself. The traditional site was not considered more boring and participants were no less likely to visit it than the ‘party’ site. Surprisingly, agreement on whether the sites promote ‘a good time’ did not differ, either.

However, the traditional site produced higher ratings of education promotion and participants were more likely to say they learned something from it. Participants were more likely to recommend the site to others and feel that more sites of its nature should exist.

Most importantly, however, are the research hypotheses themselves, which address the views that participants held about the website’s creators and the credibility of the information on the site. Clearly, the creators of the ‘party attitude’ site were judged as less intelligent than the creators of the traditional site. Of course the reasons for this are not clear in this study. Participants may not have felt that people who party are less intelligent, but rather that people who chose to emphasize this behavior on a site promoting an academic (and science-related) club were less intelligent. This is a testable hypothesis, just not tested here.

What is most concerning for this context, however, is the difference in credibility. The purpose of the club as described on the websites is, “To encourage professional activity and involvement.” The goal is not to form a social club. Of course the images of partying are not contradictory (even on the traditional site, the images involve groups having fun), but they do not promote the cause. These seemingly unrelated endeavors (academic and hedonic) appear to mix like oil and water and for a university club wishing to promote a scientific field, credibility is vital.

There are some things to keep in mind when drawing conclusions from these findings. Some of these strengthen the argument, some are limiting:

• Participants only viewed one website. They did not compare the websites side-by-side. This is a strength as participants were given no clues to the purpose of the experiment.
• This was a true experiment and, therefore, causal conclusions are reasonable.
• Everything is relative and when we discuss scientific findings, we are always talking about comparisons. A medication works relative to no medication. A teaching method is better than a different method. In this case, the traditional academic website is compared to one in which relatively more images and verbiage referred to leisure social interactions (parties).
• The participants were college students in a science field. The proportion planning to work in the field is probably a minority (it changes) and the proportion planning a career in research is very small. Many students major in psychology without awareness of the scientific rigor required for a degree. Still, the participants of this study have been trained in research methods and there is some reason to think that they may care more about science and academics than the general public. I am not sure this fact is important when considering the implications, but it may be.
• There were some mistakes which resulted in unintended differences between the websites. For example, on one, the biographies are aligned vertically and on the other, one is offset to the right. There is a link to the calendar at the top of one and not the other. The gallery links are in a different order. I agree with the researchers that there is no reason to think that any of these minor differences affected the results.

That just about sums up the experiment. I leave it to you to decide what these findings suggests about impression formation and management in other pursuits and fields. Certainly they are not generalizable to every situation, but they do provide food for thought.