It seems that the best motivation for me (to write) is frustration.

A press release by UC Berkeley about a study that was recently published on the relationships among religiosity, compassion, and prosocial behavior has been making the rounds over the last couple of days, waved by proud atheists as evidence of superiority and bashed by the more skeptical as bad science (even though most haven’t appeared to have read the study). The latter has been exacerbated by the fact that the majority of reports include very big mistakes.

I’m going to analyze the study, but I will post that separately since it is likely to be long and I also want to discuss the incredible mess of BS this has become. In that post I will also discuss the reasons some of the errors that seem minor here are actually quite serious.

I blame the press release. Unfortunately, as the culture of higher education becomes more consumer-oriented, strapped-for-cash universities must market themselves strategically, and scientists are forced to compete for funding, the integrity of science is diminished. I don’t know if the study’s authors were involved, but my guess is that the press release’s author is mostly responsible for spinning the findings, omitting important parts of the findings, and cherry-picking statements by the scientists to make it appear that their speculative explanations for those findings are solid conclusions. I think that spin, along with the fact that the findings involve an interaction (a notoriously difficult concept to grasp due to its non-linear nature), confused science writers, many of whom are not in the practice of reading studies and many of whom added their own spin to the mix.

The study, published in the Journal of Social Psychological and Personality Science, is titled “My Brother’s Keeper? Compassion Predicts Generosity More Among Less Religious Individuals”.  The first error of the press release was made in its title: “Highly religious people are less motivated by compassion than are non-believers” by incorrectly describing the subjects of the study. It also errs in a few minor ways (e.g., describing the study as “three experiments” when one only one was experimental, one was analysis of existing data, and one was quasi experimental), but the biggest problems are practically criminal in the science world. The author correctly (with the exception of the description of the subjects) reported that the relationship between compassion and prosocial behavior was stronger among less religious participants than more religious participants, but omitted the findings which clearly showed that the more religious participants were, in general, more compassionate and generous overall.

When sloppy reporting took over, the result was an utter mess.

On the one hand, the authors claim in their introduction to be interested in what motivates less religious people to act prosocially, so perhaps the spin was the plan all along. However, if they were not interested in religiosity as a variable – if they were not interested in comparing the more religious to the less religious, then they should not have limited their study to the population of interest.

Something I found interesting is that the third paragraph of the press release makes a statement which should have sent red flags up because there was no follow-up that made sense. It started with:

The results challenge a widespread assumption that acts of generosity and charity are largely driven by feelings of empathy and compassion, researchers said.

And yet the rest of the piece focused on the fact that empathy and compassion DID drive generosity and charity in less religious participants, even if it repeatedly incorrectly referred to those participants as “non-believers”. If the author had included the findings that more religious participants were more generous, this would have made sense.

The author errs again with this circular definition of “compassion”, conflating the dictionary definitions of “compassion” and “prosocial behavior” with an effect:

Compassion is defined in the study as an emotion felt when people see the suffering of others which then motivates them to help, often at a personal risk or cost.

Compassion cannot be defined in a study as an emotion felt by someone. That’s a variable we can’t measure directly. In a study, we use operational definitions. In this case, compassion was defined differently in each of the three studies within the article.

The author continues the spin with statements such as (bold mine):

When they looked into how much compassion motivated participants to be charitable in such ways as giving money or food to a homeless person, non-believers and those who rated low in religiosity came out ahead: “These findings indicate that although compassion is associated with pro-sociality among both less religious and more religious individuals, this relationship is particularly robust for less religious individuals,” the study found.

What the bold suggests is that participants who fell into the “low religiosity” category were more prosocial and/or more compassionate. That’s not what the finding means. What was greater was simply the relationship between compassion and prosocial behavior. Because the “high religiosity” participants were more compassionate, they were also more prosocial overall (marginal significance – see discussion below).

Finally, this statement is grossly misleading:

Those who scored low on the religiosity scale, and high on momentary compassion, were more inclined to share their winnings with strangers than other participants in the study.

What participants do in a study is really not interesting. What their behavior tells us about how people behave in the world is. Because there was no statistical analysis comparing the “low-religiosity/high-momentary compassion” group to the other groups, this finding does not allow us to infer anything about the population of interest. You and I could complete a game of Scrabble with a final score of 102 to 103 (respectively), but I would not brag to my friends that I am the better Scrabble player.

Most reports of the study either posted the press release as-is or quoted large chunks of it. Headlines ranged from the simple and correct, if misleading, “Compassion may motivate faithful less” to the still incorrect, but closer “Confirmed: Atheists more motivated by compassion in charitable giving than believers are” to the blatantly incorrect statement suggested by, “Are Religious People Less Compassionate?”

Almost none of the reports describe the samples correctly. The samples were broken into two groups based on measured religiosity, a common practice in social psychology. Thus, there was a range of values and those who scored in the top half were considered “more religious” and the bottom half “less religious”. The “less religious” group cannot be described as “athiests”, “agnostics”, “non-believers”, or even “people low in religiosity”. The “more religious” group cannot be described as “highly religious”, either.

One site that I usually find more accurate contained a much more serious error than an incorrect sample description. It reported a finding not found in the press release that is actually the opposite of what was reported in the study:

Two other experiments also confirmed that more religious participants seemed to be less generous.

They should have stuck to quotes of the press release.

The worst by far, though was on a site called “PsychCentral“. The first paragraph reads:

A provocative new study from the University of California, Berkeley suggests highly religious individuals are less likely to help a stranger than less religious people.

Um. No. That’s not what the study suggests. At all.

and this a few paragraphs down:

Experts say the results challenge a widespread assumption that acts of generosity and charity are largely driven by feelings of empathy and compassion.

Why they added “experts say” to this is a mystery because it’s basically a lie, but this statement is particularly baffling when you consider that the progression of statements: 1) nonreligious are more generous 2) nonreligious are more motivated by compassion to be generous 3) results challenge the assumption that generosity is driven by compassion. On what planet does that make sense?

I’d lament that people don’t actually read, but the worst part is that someone actually wrote this. Sure, it’s mostly quotes and they made stuff up, but they changed some words and moved things around, so they had to pay attention to something.

A friend shared one decent report; it did not seem that the author read the journal article, but at least he read one report (on livescience.com) that is almost a duplication of the press release and put some thought into it. He noted that the findings reported an interaction – that generosity was more related to compassion among the less religious than it was among the more religious – and questioned who was more generous overall. If the press released had not omitted those findings, he would have had an answer, or at least a theoretically-likely hypothesis. Still, he did a little bit of research on his own and noted some well-known findings that the religious tend to give more to secular charities than atheists.

Finally, the press release included a few statements by the authors which were highly speculative. It is standard for authors to discuss possible explanations for their findings, but they are often presented to the public as conclusions the authors reached. In this case as in many others, reports of the study often imply that these things are findings when the study did not examine them at all. For example, the last paragraph of the press release is:

“Overall, this research suggests that although less religious people tend to be less trusted in the U.S., when feeling compassionate, they may actually be more inclined to help their fellow citizens than more religious people,” Willer said.

The rawstory.com report ended the same way, except they changed the last word to “concluded”. This statement serves as a hypothesis for another study, but it is not a finding of the study and cannot provide a conclusion.

I have heard a lot of explanations of what motivates more religious people, but most have been based on personal beliefs or experiences – duty to God, moral obligation, sense of community, etc.  This question may have been answered and looking to other studies might yield something, but this study does not address it – it did not set out to address that question.

A full analysis of the journal article will follow in a separate post later today.

UPDATE: Analysis can be found here.
ANOTHER UPDATE: The award for the most botched report now goes to MSN for this doosey.

NOTE: this post also appears on the wonderful site about crazy coincidence, theoddsmustbecrazy.com.

Monty Hall

I have always been amused and intrigued by responses to “The Monty Hall Problem”, especially when I talk about it to audiences with a high concentration of engineers and mathematicians. If you are familiar with it, but you’ve always struggled with an unsettled feeling of “this can’t be right”, read further and let me know if my explanation of the solution helps to alleviate the discomfort. If you are not familiar, I guarantee you will give your brain a workout by reading on.

First posed to statisticians in 1975, “The Monty Hall Problem” is well-known among academics because it still sparks debate. Many seem to think that disagreements about its solution stem from issues in the clarity of the problem, but I contend that it really stems from human flaws in the way that we process information.

I often discuss this problem in statistics and cognitive psychology courses for several reasons. It is a great exercise in probability calculation and it can be used to teach basic mathematical modeling (and its purpose). An added benefit, since almost all of my students were psychology majors, is that it also illustrates a flaw in human cognition as well as a pattern of problem solving. Even a knowledgeable statistician feels the need to run simulations to see the solution in action. Even then, fully grasping the mechanisms behind the answer often requires brute force cognition.

In general, human beings have a very difficult time wrapping their brains around concepts of probability. It is much like a visual illusion; we know that the lines are parallel/the circles are the same size/there is no motion, but we can’t make our brains process it in a way that represents that reality. It’s just not how our visual system works. I hypothesize that one of the reasons that probability is such a difficult field for most people is that it involves theory and models, which are distinct from observations and we must represent them differently in our minds to properly deal with them. Applications of probability often involve switching gears from the realm of models to data or vice versa and this is where I think most mathematicians get side-swiped in The Monty Hall Problem.

The Poser

In essence, here’s the problem:

You are a contestant on Let’s Make a Deal!and Monty loves your creative costume (a teddy bear carrying a human doll), so he calls on you to make a deal. Monty says, “There are three doors – Door #1, Door #2, and Door #3. Pick one and you get to keep whatever is behind it.”You’ve seen the show (you weren’t just walking down Ventura Boulevard in a teddy bear costume for fun), so you know that it is highly likely that there is a coveted BRAND NEW CAR! behind one of those three doors. If you choose wrong, however, you might end up with an ostrich…

Everyone hopes for a car. Some get donkeys or other animals.

You choose Door #3.

Monty then says, “Let’s see what’s behind Door #1!” and the door opens to reveal one of the many consolation prizes (and product placements), a lifetime supply of Rice a la Roly.

Cool! You might get that car after all!

Well, the show was successful because the shell-game-huckster-style of Monty Hall rarely stopped there. In this case, he does what he often does, offers to let you switch from your first choice (Door #3) to the only remaining option, Door #2.

Should this woman switch?

Should you? Does it matter?

Not the Problem

Before I get into the solution, let me first deflect a common complaint from mathematicians. The most well-known version of the problem, from its Wikipedia entry:

Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1 [but the door is not opened], and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?

This version does not specifically state the name of the show or indicate the way that game shows of its era worked. If you have never seen the television show (i.e., you are younger than 35), or any game show of its kind, let me explain. Monty is in control of almost everything that happens. The only thing “contestants” can do is make choices when Monty offers them. As you will see, they had more control over their odds of winning than once thought, but Monty manipulates some of the build-up by choosing which items to reveal at different steps in the game.

Unfortunately, many probability theorists and mathematicians took issue with the lack of clarity in the problem (context is important sometimes). This provides a face-saving ‘other version’ for the geeks who get it wrong the first time. But whenever I hear comments like, “Okay, given this version, that Monty knows where the car is.” I usually think, “Of COURSE he knows where the car is! There is no other way to play the game!” and wish that people were more able to accept that they are just as human as everyone else.

The problem itself is written clearly, though: it specifically states that a door without a car behind it is revealed before you are given the option to switch. If the situation was a fully-randomized, double-blind game (like “Deal or No Deal”), then the option to switch would not even be on the table if the car is behind the revealed door. There would be no problem in that case. Therefore, the problem is a question of whether you should switch in a controlled setting – one in which the only participant who doesn’t know the location of the big prize is you.

The issue of knowledge is a factor in our processing of the problem, but it’s not what Monty knows that matters. It’s what you (the subject of the problem) know.

So, let’s put that complaint behind us and get back to the problem.

The Answer, and How to See it for Yourself

Hopefully, if the problem is new to you, you’ve spent some time trying to solve it instead of going with your first gut feeling, which was probably, “It doesn’t matter.”

It does. You should switch.

If you don’t believe me, try running some simulations. You’ll have to run a lot in order to get a large enough sample to be certain to see the trend, but here are a few ways to do it:

• Use your favorite program (MATLAB, R, etc.). There is a good database of pre-written simulators for this here. I am partial to Excel myself, even though it’s a bit more cumbersome. I just don’t remember enough code to use another program.

• Use a web-based simulator. Do it at least a hundred times, choosing to switch for half of the trials, and keep a tally your results.

• Use a die to simulate the outcome, assigning 1-3 to “Door #2″ and 4-6 to “Door #3″ (e.g., if you roll a 5, Door #3 is the one with the car). Roll at least a hundred times, choosing to switch for half of the trials (before rolling!). Keep a tally of the results.

What you will see is that switching will result in winning a car in approximately 2/3rd of the trials while staying will only provide a win in 1/3rd of them.

I know what you’re thinking. “But, there are only two doors left, so it should be 50/50!”

Why it is so Difficult to Accept

Human cognitive development is an interesting process. We learn to interpret information from the environment very quickly so that we can respond to that environment, but learning to reason hypothetically takes more time. Even adults with scientific training have a difficult time separating the concept of variables (each has a set of possible values) and data (values which are known).

In practice, hypothetical situations are often conditional (e.g., “If A, then B”). We tend to use information about what is to reason about what could be. We do this because it often works, but it is one of the ways in which our brains can lead us astray. For example, given the premise, “If I study, I will get a good grade on the exam”, what is the most sound conclusion when presented with a good grade? The most common response is, “I must have studied”, but that is not sound. In this premise, studying provides a guarantee for a good grade, but there is no statement that studying is the only way to get a good grade. It does not, for example, read, “If and only if I study…”

In the case of the Monty Hall Problem, the probability of winning is set before you pick a door. No matter which door you choose, the probability is 1/3rd. This is because there is a 1/3rd probability that the car is behind the door you chose given the information you had when you chose it. In reality, the car is behind one of the doors, so the probability it is behind Door #2 is 100% if it is there and 0% if it is not there. Probability is not a useful way to discuss what is or what happened; it is a tool for predicting what is likely to be true/happen.

The new information provided by revealing a loser changes the circumstances and this where we get trapped in our representations of models and data, possibilities and facts.

You had a 1/3rd chance of winning because there were three, equally-likely locations to choose from. It seems as if cutting the choice down to two should change the odds of winning to 1/2. It seems that way because we are focused on the probability that a given piece of information is true (e.g., that the car is behind Door #1) and not the probability that an event will occur (e.g., that we will win the car). The probability that we will win the car relies on the number of possible states of reality. This, in turn, initially relies on the number of locations for the car. When the situation changes, we try to adjust probabilities based on possible locations (which has changed) rather than on the number of possible states of reality (which has not).

Basically, when Monty makes the second offer, the offer is to switch from the door we have (#3) to the door we don’t have (#1 or #2). It does not matter that only one of those doors is left; there is still only a 1/3rd chance that our door has the car and a 2/3rd chance that the set of the other two contains the car.

Getting Un-Stuck

If you change the way you represent the problem from the beginning, the solution might seem more reasonable. Specifically, instead of thinking in terms of assigning probabilities to doors, think in terms of assigning probabilities to outcomes: winning verses losing.

Three possible states of reality, each with one winner and two losers

Let’s go step by step…

Monty asks you to pick a door from three choices. Behind one of those doors is a car. There are three possible locations and it must be in one of them, so there are three possible states of reality.

You choose to bet on Door #3; there is a 1/3rd chance that you will win the car.

There is a 2/3rd chance that you will not win the car.

This would be true no matter which door you chose.

Monty reveals that one of the remaining doors is a loser. At least one will be a loser since there is only one winner and you can choose only one. The car, however, does not move. Even though there are only two locations left, so there are still three possible states of reality. What’s changed is that we now know more about each of those possible states (there are fewer locations for the car to be):

So, if we model the problem in terms of the probability of winning with Door #3, the model itself does not change after the losing door is revealed. What changes is that we would no longer want to choose that door, so it is no longer among our options. This leaves us with only two options: keep the door we have or switch to the remaining door. The odds of winning/losing with Door #3 have not changed, but eliminating an option allows us to make a better choice – switch.