As I noted in my last post, the study I’ll be discussing was grossly misreported, starting with its press release. Since the study itself appears to be behind a pay wall for most people, I’ll describe as much detail as I can in a blog post as I discuss the study’s validity and findings of the study, published in the Journal of Social Psychological and Personality Science and titled “My Brother’s Keeper? Compassion Predicts Generosity More Among Less Religious Individuals”.

But for those who are not at all interested in the research methods or a breakdown of why I rate the quality of the study the way I do, I will give you the the bottom line so you can skip the rest or only read the sections that interest you (I’ve used headings to make it easier).

Summary

I think that the findings will hold up to replication, despite some issues I have with the way they did a few things. Overall, the research quality is quite high.

The groups they compared did not include atheists, agnostics, believers, non-believers, highly religious, or any other label that you can throw at it. In the studies they used raw religiosity scores and made some comparisons of “higher” and “lower” using values from the distribution. In a sense, the compared those who scored in the lower half of the sample to those who scored in the upper half.

They found:

• Differences in prosocial behavior cannot be dismissed as due to political affiliation, socio-economic status, or other factors often held up as responsible.
• Religiosity is correlated with trait compassion; the more religious, the more compassionate.
• Trait compassion is related to prosocial behavior in general. This relationship is stronger in the less religious than in the more religious. This does not mean that the less religious are more compassionate (see number 1) or that the less religious are more prosocial. It just means that compassion is a bigger factor in prosocial behavior in the less religious.
• The findings of the first study can be interpreted one way that isn’t discussed in the paper: when the relationship between compassion and religiosity is accounted for, the more religious are not more prosocial than the less religious.
• The findings in the second study, which involved inducing feelings of compassion, were similar for generosity, except that the more religious were more prosocial even after accounting for compassion.
• The findings of the second study also included a different pattern when the prosocial behavior was giving to charity. Compassion induced more giving, but the effect was weak and did not differ much across religiosity. Religiosity had a significant affect on charity. This can be explained by the guidelines provided by many churches for how much of one’s salary one should give.
• In the third study, in which state compassion (how compassionate the individual felt at that time) was measured and the prosocial behavior measure involved real-world cash, religiosity was not related to either compassion or prosocial behavior.
• In the third study, state compassion was positively correlated with prosocial behavior, but the effect was greater in the less religious than in the more religious.

What the findings as a whole say to me, and what I believe the press report tried, but failed, to express, at least with convincing support: We do not need religion to be prosocial. We need compassion.

This is great news for secularists.

However, it doesn’t say anything negative about religion or the religious, nor does it provide anything that should make atheists feel superior. It just shows that one can be good without God; that motivations can come from other sources.

Now on to the details…

NOTE: to keep this as short as possible, I’ve included a lot of links to terms and demonstrations. Where I describe problems in more detail I still water-down quite a bit. I will do my best to make it understandable without rambling on and on, but keep in mind that it takes many years to learn enough about research design and statistics to understand why some of these are problematic. Furthermore, not all researchers will agree on the consequences of some of these problems. I am still learning this stuff myself (probably always will be learning).

The Study (description)

The article reports three studies, each related to the relationship between compassion and prosocial behavior in less-religious individuals. I have created graphs using the information in the paper, but in some cases I did not have exact numbers, so while the relationships are visually accurate, there are only values where I could use exact numbers.

Theoretical Foundation

The introduction discusses research which documents that religiosity is associated with prosocial behavior. Specifically, religious people give more and volunteer more than nonreligious people, over and above what they give to and do for religious organizations. The researchers note that the nonreligious do give; when we compare groups, we do so using averages. However, it may be that the motivations for prosocial behavior vary in a way that interacts with religiosity. In other words, the more religious among us may be motivated to prosocial behavior by one set of factors and the less motivated by another.

The researchers hypothesized that compassion is a more influential factor in prosocial behavior for the less religious than for the more religious among us.

Study 1

The first study examined the relationships among religiosity and traits of compassion and prosocial tendencies. What this basically means is that situational factors were not involved; traits are a matter of personality or attitude. For example, “trait anxiety” refers to how anxious a person is in general, while “state anxiety” refers to how anxious that same individual feels in a given situation.

This study involved analyzing data from a 2004 “survey”. I put that term in quotes because it usually refers to a set of questions that do not measure more than what is apparent at face value. Established measures of latent variables (variables which cannot be measured directly such as feelings and attitudes) are usually called an “inventory” or “scale” and we refer to them loosely as “measures”. In this case, the survey involved such measures and I want to make that clear.

The sample was comprised of 1337 participants and covariates (variables other than those of interest which could explain differences among the groups) of gender, political orientation, and education were included in the analysis. The variables of interest were religiosity, compassion, and prosocial behavior. Religious identity (identification with a specific religion or no religion) was also considered.

Results

Correlations

• Covariates had little impact on the results.
• Trait compassion was positively correlated with religiosity* and prosocial behavior. On average, the more compassionate the individual, the more religious they were and the more the more prosocial they were.
• The relationship between religiosity and prosocial behavior was marginally significant (statistically).

Hypothesis Test (See Figure 1)

Figure 1: Study 1 results. "Higher" and "lower" are defined here as 1 SD from the mean.

• A regression analysis revealed an interaction of religiosity and compassion on prosocial behavior. What this means: The effect of compassion on prosocial behavior differed among levels of religiosity.
• More specifically, the level of trait compassion affected prosocial behavior less as religiosity increased.
• There was also a main effect of compassion, but that was apparent in the correlational analysis.
• There was no main effect of religiosity on prosocial behavior. This is interesting, because they found a marginally significant correlation, but it does not mean the there are no difference in prosocial behavior. I would interpret these findings, when put together, as suggestive of little or no difference between the more religious and the less religious in prosocial behavior over and above the differences accounted for by compassion.

The authors discuss the findings a little differently, though, focusing on the differences in the way that compassion affected prosocial behavior (the interaction in the first hypothesis test result) and ignoring the way that the effect of religiosity disappeared when compassion was entered into the equation. It seems more interesting to me to treat compassion as the moderator. It also makes more sense in the end.

Study 2

This study was experimental in that the researchers manipulated state compassion. In other words, they induced feelings of compassion in half of the participants and compared the amount of prosocial behavior those participants engaged in to the amount of such behavior in a control condition.

The sample included 101 participants and the study was conducted online, so the age range was exceptional (from 18 to 68 years). Participants were randomly assigned to one of the two conditions and each watched a short video under the guise that there would be a test of memory afterward.  The videos were established manipulations of feelings of compassion and neutral emotion (i.e., other researchers tested their effectiveness). Following the video, participants completed two tasks which are well-established measures of prosocial behavior commonly used in such research.

Results

• Again, covariates had little impact on the results.

Hypothesis Tests (See Figure 2)

Figure 3: Results of Study 3. Values are relative.

There were two tests since the participants completed to different prosocial tasks, one involving generosity and the other involving charity.

For the generosity task:

• This time there were a main effects of both religiosity and compassion on prosocial behavior. The more religious, the more prosocial. Those who watched the compassion-inducing video were the more prosocial on average than those who watched the neutral video.
• The interaction appeared again in the manner as in Study 1.

For the charity task:

• There were main effects of both religiosity and compassion on prosocial behavior.
• There was no interaction.

This is where they screw up, in my opinion.

The pattern of the moderation was in the predicted direction but failed to reach statistical significance.

This is not an acceptable statement unless the findings are marginal. This was not. The p-value was .408. This is not even close to meaningful. Still, they went ahead with the analysis of the interaction and reported an effect of compassion on charity for the less religious participants and no effect for the more religious.  The problem is that post-hoc analysis like this assumes that a significant interaction was observed. Their tests inflated alpha (the probability of a Type I Error) and can only mislead. They stated that they had found “partial support” for their hypothesis, but they did not in this case.

The relationships in the generosity task are very clear when we look at a Figure 2. The interaction is the interesting finding. Compassion had little effect on the more religious, but a very large effect on the less religious, who gave practically nothing when compassion was not induced. There is no analysis to tell us if the less religious surpassed the religious by a statistically significant amount when compassion was induced, but they were clearly out done by the more religious when not made to feel compassion.

The charity task showed no such interaction and the authors did not include a graph of this effect that I could recreate, nor did they provide the information to make one.

Study 3

For this study, the sample of 120 completed a state compassion inventory (a measure of their feelings of general compassion at the moment) and a series of “economic tasks designed to measure their generosity, trust, trustworthiness, and motivation to reward others’ generosity.” What differed in this study, however, was that the ‘points’ they earned in these tasks could be exchanged for cash at the end of the study. Participants did not know how much cash, but they knew that the more points they earned, the more cash they would receive.

Results

Hypothesis Tests (See Figure 3)

The findings of this study were very different from the other two.

• State compassion was not related to religiosity.
• Religiosity was not related to prosocial behavior.
• There was an interaction of religiosity and compassion on prosocial behavior. The amount of compassion felt had more of an effect on the behavior the less religious than it did on the more religious.

The graph of these findings, a reproduction of their graph since they did not provide information to create one that would make more sense (to me anyway), is a bit misleading. The values are z-scores, so they are relative to one another and not actual values. What is interesting, though is how little the prosocial score varied in the more religious group and how that line barely dips below the mean value (represented by 0).

There is also a problem with the press release in that it makes the claim that the high state compassion/less religious group out-performed the others. There is no statistical analysis comparing the groups in that way, so this is a misstatement. We do not know if less religious individuals are more generous than more religious when motivated to act prosocially. We just know that they are more generous when motivated by compassion than when compassion is low.

The Study Overall

As I noted, my opinion of the studies as a whole is relatively high, but I do have some major criticisms. Some of the language makes me cringe (e.g., results are the product of statistical tests, so “We tested our results”…), but I have seen more and more of this as scientific reports in general have grown sloppier.  Study design and method is much more important, as is the quality of the reporting beyond language.

The authors also throw around the term “robust”, claiming in the first study that the relationship between compassion and prosocial behavior is “particularly robust” for less religious individuals. That term refers to findings which are “sturdy” and will stand up when some supports are removed – effects which appear to hold up in different situations. Since this was one analysis of one data set, that term just doesn’t work. It does not fit in any of their uses of it.

In fact, they err in Study 2 by saying that the effect was “attenuated” for the more religious. That term is relative; attenuated compared to what? The effect was not “robust” in one condition and “attenuated” in another; they can only be compared to each other. The effect was greater in the less religious than the more religious.

Missing Information

There are a number of bits of information which are considered to be, at minimum, required for a good research report. A general rule of thumb for methods and results sections is to include enough (without being redundant) information to allow other researchers to replicate (in a strict sense) the study and to confirm that the statistical findings are properly interpreted.

I am not sure that this article meets that criterion. The methods are pretty well fleshed out and the paper is full of statistics, but some descriptive statistics are missing that I would have liked to have seen (e.g., means reported overall for measures, but not by group) and there was not enough of the right information to recreate them.

Grouping the Data and Errors of Generalization

One overall criticism which warrants discussion is in the grouping of data. There are some problems with this and they are related. The sensitivity of the religiosity measure is one problem that, by itself, is not a big target for criticism. Combined with the second problem of grouping participants, though, it becomes more serious.

The practice of comparing groups of people based on a variable which is distributed on a spectrum is a common one. The question the researcher wants to answer is important in deciding whether to group and, in this case, I do not disagree with that choice, but I question how they grouped and how it was communicated. If the data are clustered (the distribution is multi-modal), grouping is simplified, but if the data are distributed more loosely, it can be tricky and dangerous.

First, the researcher loses information, therefore they lose sensitivity and usually lose power. The sensitivity problem is relevant in the first study, but mostly because it makes the findings difficult to interpret.

Second, if the way that the grouping is communicated is not consistent and clear, it is likely to be misinterpreted, compounding any existing problems with the method. I discussed this problem in my last post. Most of the reports referred to the groups compared as “highly religious” verses “atheists and agnostics” or something like that. However, where are all of the people in the middle (i.e., most likely the bulk of the sample)?  Within each group there was variation in religiosity and comparisons are made using averages. Generalizing only works when the samples are representative of the population of interest and this applies in either direction of the generalization (i.e., specific to mixed or mixed to specific).

Third, researchers must decide where to draw the lines between high and low (and anything in between). Since the majority of variables in psychology are normal distributed (therefore symmetrical), the lines are usually drawn using rankings of sample values and the most common way to split a sample in half is to put all values above the median into “higher” and those below into “lower” (called a “median split”). However, ease is not a good reason to use this technique.   Here is an interesting demonstration of the dangers of dichotomizing normally-distributed variables.

But… religiosity is not usually distributed normally; it’s usually skewed. Skew means that it’s not symmetrical, so a median-split would make even less sense.

In this case, it seems that the authors tried to have the best of both worlds by treating religiosity as a continuous variable, but doing post-hoc analysis on it, discussing it, and graphing it as if it were dichotomous, choosing values which were one standard deviation from the mean in both directions as the central tendencies of each group. The biggest problem with this is the assumption of normality. If the variable is not normally-distributed (and I suspect that it is not), this grouping is a bit tough to swallow.

When this problem is mixed with a limited range as it is in the first study (the religiosity scale only had four points), it’s a problem. The four possible values were 1 = no religion, 2 = not very strong (religious identity), 3 = somewhat strong, and 4 = strong. Since the mean was 2.99, the bulk of the sample were fairly religious. one standard deviation (1.03) below the mean is not exactly in non-believerland and one above is off the scale (literally). It is just very difficult to see where “higher” leaves off and “lower” takes over.

Although the range is adequate in the other two studies, the problem of discussing groups which do not actually exist and have fuzzy definitions remains. In my opinion that is one of the reasons it was so misreported.

But, overall, the research is of a relatively high quality and interesting. I would like to see more variation in the prosocial tasks, given that the outcome of the charity task was so different from the tasks of generosity.

It seems that the less religious are at least as generous as the more religious, but their reasons for acting prosocially differ. I would like to see the day when, as a group, we are generous and prosocial consistently, without the need to be provoked and without needing to feel an emotional connection to the receiver.

Saslow, L., Willer, R., Feinberg, M., Piff, P., Clark, K., Keltner, D., & Saturn, S. (2012). My Brother’s Keeper? Compassion Predicts Generosity More Among Less Religious Individuals Social Psychological and Personality Science DOI: 10.1177/1948550612444137

Related posts:

1. Science and Spin Are Very Bad Bedfellows
2. There is No Debate Over Evolution
3. Is that a fallacy in your pocket or can you cite some sources? A response to Women and Feminism at TAM8

Monty Hall

I have always been amused and intrigued by responses to “The Monty Hall Problem”, especially when I talk about it to audiences with a high concentration of engineers and mathematicians. If you are familiar with it, but you’ve always struggled with an unsettled feeling of “this can’t be right”, read further and let me know if my explanation of the solution helps to alleviate the discomfort. If you are not familiar, I guarantee you will give your brain a workout by reading on.

First posed to statisticians in 1975, “The Monty Hall Problem” is well-known among academics because it still sparks debate. Many seem to think that disagreements about its solution stem from issues in the clarity of the problem, but I contend that it really stems from human flaws in the way that we process information.

I often discuss this problem in statistics and cognitive psychology courses for several reasons. It is a great exercise in probability calculation and it can be used to teach basic mathematical modeling (and its purpose). An added benefit, since almost all of my students were psychology majors, is that it also illustrates a flaw in human cognition as well as a pattern of problem solving. Even a knowledgeable statistician feels the need to run simulations to see the solution in action. Even then, fully grasping the mechanisms behind the answer often requires brute force cognition.

In general, human beings have a very difficult time wrapping their brains around concepts of probability. It is much like a visual illusion; we know that the lines are parallel/the circles are the same size/there is no motion, but we can’t make our brains process it in a way that represents that reality. It’s just not how our visual system works. I hypothesize that one of the reasons that probability is such a difficult field for most people is that it involves theory and models, which are distinct from observations and we must represent them differently in our minds to properly deal with them. Applications of probability often involve switching gears from the realm of models to data or vice versa and this is where I think most mathematicians get side-swiped in The Monty Hall Problem.

The Poser

In essence, here’s the problem:

You are a contestant on Let’s Make a Deal!and Monty loves your creative costume (a teddy bear carrying a human doll), so he calls on you to make a deal. Monty says, “There are three doors – Door #1, Door #2, and Door #3. Pick one and you get to keep whatever is behind it.”You’ve seen the show (you weren’t just walking down Ventura Boulevard in a teddy bear costume for fun), so you know that it is highly likely that there is a coveted BRAND NEW CAR! behind one of those three doors. If you choose wrong, however, you might end up with an ostrich…

Everyone hopes for a car. Some get donkeys or other animals.

You choose Door #3.

Monty then says, “Let’s see what’s behind Door #1!” and the door opens to reveal one of the many consolation prizes (and product placements), a lifetime supply of Rice a la Roly.

Cool! You might get that car after all!

Well, the show was successful because the shell-game-huckster-style of Monty Hall rarely stopped there. In this case, he does what he often does, offers to let you switch from your first choice (Door #3) to the only remaining option, Door #2.

Should this woman switch?

Should you? Does it matter?

Not the Problem

Before I get into the solution, let me first deflect a common complaint from mathematicians. The most well-known version of the problem, from its Wikipedia entry:

Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1 [but the door is not opened], and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?

This version does not specifically state the name of the show or indicate the way that game shows of its era worked. If you have never seen the television show (i.e., you are younger than 35), or any game show of its kind, let me explain. Monty is in control of almost everything that happens. The only thing “contestants” can do is make choices when Monty offers them. As you will see, they had more control over their odds of winning than once thought, but Monty manipulates some of the build-up by choosing which items to reveal at different steps in the game.

Unfortunately, many probability theorists and mathematicians took issue with the lack of clarity in the problem (context is important sometimes). This provides a face-saving ‘other version’ for the geeks who get it wrong the first time. But whenever I hear comments like, “Okay, given this version, that Monty knows where the car is.” I usually think, “Of COURSE he knows where the car is! There is no other way to play the game!” and wish that people were more able to accept that they are just as human as everyone else.

The problem itself is written clearly, though: it specifically states that a door without a car behind it is revealed before you are given the option to switch. If the situation was a fully-randomized, double-blind game (like “Deal or No Deal”), then the option to switch would not even be on the table if the car is behind the revealed door. There would be no problem in that case. Therefore, the problem is a question of whether you should switch in a controlled setting – one in which the only participant who doesn’t know the location of the big prize is you.

The issue of knowledge is a factor in our processing of the problem, but it’s not what Monty knows that matters. It’s what you (the subject of the problem) know.

So, let’s put that complaint behind us and get back to the problem.

The Answer, and How to See it for Yourself

Hopefully, if the problem is new to you, you’ve spent some time trying to solve it instead of going with your first gut feeling, which was probably, “It doesn’t matter.”

It does. You should switch.

If you don’t believe me, try running some simulations. You’ll have to run a lot in order to get a large enough sample to be certain to see the trend, but here are a few ways to do it:

• Use your favorite program (MATLAB, R, etc.). There is a good database of pre-written simulators for this here. I am partial to Excel myself, even though it’s a bit more cumbersome. I just don’t remember enough code to use another program.

• Use a web-based simulator. Do it at least a hundred times, choosing to switch for half of the trials, and keep a tally your results.

• Use a die to simulate the outcome, assigning 1-3 to “Door #2″ and 4-6 to “Door #3″ (e.g., if you roll a 5, Door #3 is the one with the car). Roll at least a hundred times, choosing to switch for half of the trials (before rolling!). Keep a tally of the results.

What you will see is that switching will result in winning a car in approximately 2/3rd of the trials while staying will only provide a win in 1/3rd of them.

I know what you’re thinking. “But, there are only two doors left, so it should be 50/50!”

Why it is so Difficult to Accept

Human cognitive development is an interesting process. We learn to interpret information from the environment very quickly so that we can respond to that environment, but learning to reason hypothetically takes more time. Even adults with scientific training have a difficult time separating the concept of variables (each has a set of possible values) and data (values which are known).

In practice, hypothetical situations are often conditional (e.g., “If A, then B”). We tend to use information about what is to reason about what could be. We do this because it often works, but it is one of the ways in which our brains can lead us astray. For example, given the premise, “If I study, I will get a good grade on the exam”, what is the most sound conclusion when presented with a good grade? The most common response is, “I must have studied”, but that is not sound. In this premise, studying provides a guarantee for a good grade, but there is no statement that studying is the only way to get a good grade. It does not, for example, read, “If and only if I study…”

In the case of the Monty Hall Problem, the probability of winning is set before you pick a door. No matter which door you choose, the probability is 1/3rd. This is because there is a 1/3rd probability that the car is behind the door you chose given the information you had when you chose it. In reality, the car is behind one of the doors, so the probability it is behind Door #2 is 100% if it is there and 0% if it is not there. Probability is not a useful way to discuss what is or what happened; it is a tool for predicting what is likely to be true/happen.

The new information provided by revealing a loser changes the circumstances and this where we get trapped in our representations of models and data, possibilities and facts.

You had a 1/3rd chance of winning because there were three, equally-likely locations to choose from. It seems as if cutting the choice down to two should change the odds of winning to 1/2. It seems that way because we are focused on the probability that a given piece of information is true (e.g., that the car is behind Door #1) and not the probability that an event will occur (e.g., that we will win the car). The probability that we will win the car relies on the number of possible states of reality. This, in turn, initially relies on the number of locations for the car. When the situation changes, we try to adjust probabilities based on possible locations (which has changed) rather than on the number of possible states of reality (which has not).

Basically, when Monty makes the second offer, the offer is to switch from the door we have (#3) to the door we don’t have (#1 or #2). It does not matter that only one of those doors is left; there is still only a 1/3rd chance that our door has the car and a 2/3rd chance that the set of the other two contains the car.

Getting Un-Stuck

If you change the way you represent the problem from the beginning, the solution might seem more reasonable. Specifically, instead of thinking in terms of assigning probabilities to doors, think in terms of assigning probabilities to outcomes: winning verses losing.

Three possible states of reality, each with one winner and two losers

Let’s go step by step…

Monty asks you to pick a door from three choices. Behind one of those doors is a car. There are three possible locations and it must be in one of them, so there are three possible states of reality.

You choose to bet on Door #3; there is a 1/3rd chance that you will win the car.

There is a 2/3rd chance that you will not win the car.

This would be true no matter which door you chose.

Monty reveals that one of the remaining doors is a loser. At least one will be a loser since there is only one winner and you can choose only one. The car, however, does not move. Even though there are only two locations left, so there are still three possible states of reality. What’s changed is that we now know more about each of those possible states (there are fewer locations for the car to be):

So, if we model the problem in terms of the probability of winning with Door #3, the model itself does not change after the losing door is revealed. What changes is that we would no longer want to choose that door, so it is no longer among our options. This leaves us with only two options: keep the door we have or switch to the remaining door. The odds of winning/losing with Door #3 have not changed, but eliminating an option allows us to make a better choice – switch.

No related posts.

I am sure that most people have not thought much about why it’s called “critical thinking” and, when asked, would probably assume that “critical” refers to its importance (especially skeptics). However, they would only be partially correct. “Critical” means “crucial”, but it also refers to discriminating judgments. Critical thinking is an process in which we distinguish accurate information from inaccurate information and/or determine the best course of action given a set of possibilities.

The reason criticism is so important is a bit abstract, but it can be at least partially illustrated through the principle of falsification. Essentially, attempting to falsify a hypothesis is the best way to determine if it is likely to be true.

Most skeptics are familiar with the confirmation bias, which is the very strong human tendency to favor confirming information. We notice, remember, believe, and assign more weight to information that confirms what we already believe to be true than other information. What is discussed less often is how this bias extends to how we seek knowledge and test hypotheses. Humans tend to experiment to determine relationships in the world, but we do so in a confirmatory manner when a better approach is to attempt to falsify. That is, we tend to try this or try that to see what happens. However, the best test of a hypothesis is one in which the outcome tells us what is likely to be true by eliminating other possibilities. This is how science works most of the time and I think that we can agree that science works better than any other method.

A good illustration of this human tendency is the Wason 4-Card Task. Fortunately, it is also a good way to explain why falsification is ideal.

In the Wason 4-Card Task (A.K.A. the Wason Selection Task), participants are given the following problem:

There are four cards in front of you, each with a letter on one side and a number on the other side. The rule is: If there is a vowel on one side of a card, then there must be an even number on the other side.

Which cards must be turned over to determine if the rule holds true? (How many cards, minimum, and which ones?) 

The most popular answers in this task are the E and 4 cards or just the E card. This demonstrates an approach to hypothesis testing which is confirmatory in nature. That is, we search for examples of the first condition, then confirm that the second condition is met. This is similar to how most people deal with other types of questions, such as whether or not a pill reduces headache pain. How would you find out? Would you should take the pill when you have headache pain, then judge the pill effective if the pain is gone an hour later?

Unfortunately, this approach is not a very effective means of gaining knowledge. To understand why, let’s look at what we can learn from turning over each card.

Only the bold information tells us anything about whether the rule holds true or not and only in the cases in which it is violated do we learn something meaningful enough to draw a conclusion. The correct answer to the basic Wason Task is that both the E and 7 cards are required. The E card can confirm the rule, but what is important is that it, and the 7 card, have the ability to falsify it. Turning over the 4 card is unnecessary because the rule does not even apply unless it has been followed, therefore, we learn nothing interesting.

If my statement about is confusing, you are not alone. This task is one of the most well-studied in the field of cognitive psychology, yet why humans are so bad at it is still a bit of a mystery. One of the hypotheses is that people have a difficult time recognizing that the rule is not bi-directional. In other words, the rule is:

If there is a vowel on one side, there is an even number on the other side.

it is NOT:

If there is a vowel on one side, there is an even number on the other side and vice-versa.

However, when conditional statements like this are made, we often assume that they are bi-directional unless we have context which tells us otherwise. For example, rules like these provide context which suggests more than what is stated:

If it is Monday, then the banks are open.

If I buy a ticket, I will win something in the raffle.

Because we have schemas that tell us that banks are open on weekdays (at least), we understand that the reverse of the first conditional (If the banks are open, then it is Monday) is not necessarily true. In the second conditional, our knowledge of how raffles work tells us that a raffle ticket must have been purchased if the author won something in it. In each case, however, we are not evaluating the statements on their own merit. We introduce information from our knowledge base. The problem with this kind of reasoning is that we assume that our knowledge is accurate.

When these statements are included in arguments, we draw conclusions based, at least in part, on our current understanding of the world, which could be wrong.

One example that I often used in class illustrates how strong this bias can be and how it affects our ability to determine cause and effect relationships. Given the following argument, students are asked to judge whether the argument is valid or invalid. They are reminded that “valid” refers only to whether the argument is logically sound and not whether the statements within it are true (truth is a separate property):

Major Premise: If I learn a lot, I will get a good grade on the test. 

Minor Premise: I got a good grade on the test.

Conclusion: Therefore, I learned a lot. 

On its surface, the argument appears valid. However, it is not. There is nothing in the major premise to indicate that learning is the only way to get a good grade on the test – that learning is required for a good grade. It merely states that learning will guarantee one. Unfortunately, some dedicated students who have successfully used flash cards and other rehearsal techniques in courses in which memorization is helpful – introductory courses with multiple-choice exams and so forth – will assume that this method of studying equates to learning. When they fail in a more rigorous upper-division course in which conceptual understanding is necessary, they will be unlikely to attribute that failure to their study methods.

Going back to the example of how to determine if a pill takes away your headache pain, the method most people follow is to turn over the E card and nothing else. If your headache pain does not go away (outcome: an odd number), then you will know that the pill doesn’t work. However, what if the headache does go away? Is it possible that headaches go away on their own? Could taking the pill induce a placebo effect? What you need is to design an experiment in which you can eliminate all other possibilities (headaches go away by themselves, placebo effects, etc.). What you need to do is set the pill up to fail. If the headache goes away when it should (turning over the E card and finding an even number) AND it does not go away when it should not (turning over the 7 card and finding a consonant), then we can conclude that the pill worked.

The confirmational approach is like asking a psychic to predict the flip of a coin, but only counting the times in which the psychic was correct. This is an excellent way to find out if they have supernatural powers, but only if you also know the rate at which they are incorrect. When skeptics test psychics, they compare the outcome to what would be expected by chance. If they can consistently outperform blind guessing, the only logical conclusion is that they must not be guessing. To eliminate alternative explanations, skeptics ensure that the psychic is not receiving information from other sources such as body language cues.

Now, what does all of this have to do with criticism?

What science does, both in single studies and as a general process, is eliminate incorrect hypotheses. In general, scientists come up with likely hypotheses and theories based on what we are reasonably certain is true (as many as humanly possible), then eliminate them until only one remains. We eliminate them by finding faults with them, by demonstrating that they can’t be true (or are highly unlikely to be true).

Critical thinking works this way, too. The most effective process of evaluating information is one which looks for faults, inconsistencies, and weaknesses. Applying critical thinking to plans and ideas, otherwise we waste resources and time as well as risk harm. After we have completed a project, even a successful one, analysis should not focus on “what we did right”. We can’t know why we succeeded without comparisons to similar situations in which we failed. Focusing on what we think is good is like counting the “hits” and ignoring the “misses”.

Related posts:

1. Is that a fallacy in your pocket or can you cite some sources? A response to Women and Feminism at TAM8
2. Reaching Out and Geeking Out
3. Naughty Elmo Revisited

Many Skeptics, Daniel Loxton and Massimo Pigliucci (So you think you’re a skeptic, don’t you?) come to mind, have discussed the need to restrict one’s public discussions in the name of Skepticism to topics within one’s area of expertise. In the absence of such expertise, we should only convey to the public a scientific consensus, if one exists. So how is a non-scientist or someone working in a different field supposed to know whether a scientific consensus exists and/or what that consensus is?

Well, that is what I had initially intended to write about today. I am afraid this post goes a little off-track, but it still covers important ground.

This post began as a set of corrections to some of the misleading statements in a recent post by Rebecca Watson in which she points out what she believes is wrong with Ben Radford’s use of an article< .a> to support his argument that images of thin women in media do not “encourage” eating disorders such as anorexia nervosa. A friend alerted me to this post, knowing that I participated in a discussion on Facebook on the matter.

In her post, Rebecca accuses Ben of cherry-picking and quoting out of context. In a response to Rebecca , Ben defends himself quite well regarding the accusations that he misrepresented the authors (there were two), but I think that there is a lot missing from the discussion that is important and, in fact, this is shaping up to be a great example of why organized skepticism is needed and why more working scientists should get involved.

First, I share Rebecca’s concern that readers will assume that Botta’s findings support Ben’s claim. Ben writes,

Rebecca is assuming that the quotes were selected as representing the conclusions of those particular studies from which they were cited. I made no such claim.

This is true, but Rebecca assumed what most laypersons are likely to assume. It is far too easy to mislead when discussing complex topics such as this one. Evidence to support one’s assertions is difficult, too, when the assertion is that a claim is untrue. I think that Ben’s argument would have been better served by a more detailed discussion of the complexity of the issue and, perhaps, a quote from a recent review of the literature. Those are difficult to come by, but they provide the “big picture” view to which he referred.

Quoting from the introductions of research reports as Ben did is never a good idea, in my opinion, because it is hearsay. However, it is also problematic when people assume that a researcher’s conclusion can support an argument; it can’t. What counts as evidence is a finding.

For example, what prompted me to participate in the Facebook conversation was Rebecca’s comment:

Study after study shows that straight women and gay men develop eating disorders because our society tells them that they must be thin and pretty in order to attract a man (eg: [link to study by Siever]), and that attracting a man should be their #1 goal in life.

The study to which she linked found correlations among disordered eating, body dissatisfaction, the importance placed on physical attractiveness, and sexual orientation by gender. How that was translated into a causal chain involving societal influences and personal goals is quite alarming, but the paper itself provides some clues. The discussion is longer than any other section of the paper, less than parsimonious, and somewhat speculative. This is one of the reasons that researchers cite and discuss findings, not the conclusions other scientists draw.

As I explained to Rebecca,

Correlation does not equal cause.

Causal conclusions are extremely difficult to draw when you cannot randomly assign subjects to conditions and you cannot randomly assign people to be a straight women or gay men. It takes massive amounts of converging evidence from a variety of studies which eliminate rival hypotheses.

So to say that “straight women and gay men develop eating disorders because our society tells them [anything]” is a very bold statement that is not supported by the evidence. It is certainly not supported by the study to which you linked.

But the confirmation bias leads people to follow their current beliefs and demand to be proven wrong. For example, Rebecca ended her post with,

In the Facebook thread, I genuinely wanted to see what evidence actually supported the idea that the link between media and body image is a myth, but I got nothing from Radford or anyone observing the thread. So, I’m forced to continue to side with what appears to be the consensus opinion: the media’s portrayal of the thin ideal most likely negatively impacts the body image of those who process those images poorly.

First, the “who process those images poorly” part is a new assertion and one that reveals her understanding (or lack of understanding) of the findings. What does it mean to “process images poorly” anyway? She initially wrote, “processing them the wrong way”. Is there are “right way” and a “wrong way”? Botta’s variable of “image processing” involves whether the individual questions the weight of characters as realistic, whether they believe they are realistic, and whether they compare themselves to the characters. There’s no right or wrong here.

What’s more important is her comment that she wanted evidence that the claim is untrue. As I explained to Rebecca on Facebook, that’s not how skepticism works. The burden of evidence is on those making the claim, not those refuting it.

This post is becoming much longer than I’d intended, so I will skip criticism of Posavac’s article and Rebecca’s misrepresentation of the findings. Instead I’d like to discuss the Botta article as it seems to be at the center of this battle. My interpretation of it is quite different from Rebecca’s.

Some background that I found interesting: Botta is not a psychologist, but a professor of communications. Now, expertise can come from many places. Formal education in an area is not a guarantee that one is an expert (although, given no other information, it is much more likely that someone with a PhD in a given area knows that area well than someone without a PhD in that area). Likewise, one can gain expertise through years informal study and practice. My point, though, is that we should not assume that Botta is an expert on eating disorders.

What is more telling about this particular article is the journal in which it was published: the Journal of Communication. It is not published in the psychological literature where psychologists are likely to see and criticize it. A communications journal is not an inappropriate venue for this article, however, because it is not about eating disorders.

This 11 year old study is about body image.

In fact, Botta used the EDI (Eating Disorder Inventory), a well-established, valid, reliable inventory of disordered eating. However, she chose to use only those sub-scales related to body image disturbance, thus making generalization to disordered eating impossible. She did include a measure of “bulimic behaviors”. It’s really very puzzling. I cannot think of an explanation for this, but one would not be relevant. What is relevant: she did not measure disordered eating with the exception of “bulimic behaviors” and references to other eating disorders as a conclusion to this study would be a GROSS overgeneralization.

There are many, many studies regarding both immediate and long-term effects of media exposure on body image and the findings are not consistent, particularly those examining television viewing. Of course, the quality, methodology, and generalizability of those studies are not consistent, either. As always, null findings are likely to be underreported.

There is no scientific consensus that “ideal thin” media is a direct causal factor in body dissatisfaction. Skepticism of any claim to this effect is certainly warranted.

This area of research is fraught with methodological problems similar to the study of acupuncture and psychotherapy. Body image measures rely on self-report and studies involving media imagery make hypotheses obvious to the participants. What’s more, most studies involve short experiments from which generalizations about real-world effects are extremely difficult.

What the massive body of literature on body image suggests: person variables (characteristics of individuals) such as self-esteem (Jarry & Kossert, 2007), attachment orientation (Greenwood & Pietromonaco, 2004), neuroticism (Daily & Buunk, 2009), and even other components of personality (Roberts & Good, 2010) have direct effects on body image and/or mediate/moderate the relationship between media and body image. In other words, any relationship which exists between media exposure and body image is complex and, at the very least, indirect.

But that is not even what Ben was talking about in his original piece, nor is it what Rebecca started to discuss (given the title of her post). The topic was the question of whether the “popular view” that exposure to images of thin women promotes eating disorders is myth.

Even a surface treatment of that question has me scratching my head a bit given the nearly equally accepted “popular view” that there is an epidemic of obesity in this country.

While body image is highly correlated with eating disorders – it is even one of the diagnostic criteria – there is no evidence that poor body image, or even the desire to be thin, causes eating disorders. Stating that eating disorders are a direct result of wanting to be thin is a lot like saying that obsessive-compulsive disorder is the result of wanting to be organized or clean. Assuming a causal link exists, the direction of cause could easily be the opposite of what people think.

But let’s assume that what we’re really talking about is body image and not eating disorders. Do Botta’s findings really refute Ben’s assertions? Given that he did not discuss the study in detail, I am not sure if Ben knows, but let’s take a look.

Botta’s study is a complex analysis of a number of self-report measures which involves a large number of statistical tests. As such, some significant relationships are likely to occur by chance and it is important to consider the specific hypotheses when interpreting them. I’ve noted where Botta reported significant relationships that I believe should be considered with caution. That said, here is a list of what she found:

Factors in whether participants endorsed a thin ideal

What did NOT predict endorsement of a thin ideal:

• total television exposure
• exposure to “thin dramas” [shows like Melrose Place and Beverly Hills, 90210 - did I mention this study was more than decade old?]
• the extent to which participants reported questioning characters’ bodies when watching television

What DID predict endorsement of a thin ideal:

• the extent to which participants compared themselves to the characters
• the extent to which participants viewed the characters as realistic

Factors in body dissatisfaction

What did NOT predict body dissatisfaction:

• total television exposure
• exposure to “thin dramas”
• the extent to which participants reported questioning characters’ bodies when watching television
• the extent to which participants viewed the characters as realistic

What DID predict body dissatisfaction:

• Body Mass Index
• endorsement of the thin ideal
• the extent to which participants compared themselves to the characters

Botta also reported a significant interaction of endorsement of a thin ideal and total television exposure, but with a p-value of .03. Given the large number of tests produced in this kind of analysis, it pays to be a little more conservative and consider only those less than .01 as significant. Likewise, she reported that ethnicity predicted body dissatisfaction, however, she only reports the p-value as less than .05.

Factors in drive for thinness

What did NOT predict drive for thinness:

• total television exposure
• exposure to “thin dramas”
• the extent to which participants reported questioning characters’ bodies when watching television

What DID predict drive for thinness:

• Body Mass Index
• endorsement of the thin ideal
• the extent to which participants compared themselves to the characters

Again, Botta also reported that viewing the characters as realistic was related to drive for thinness, but with a p-value of less than .05.

Factors in bulimic “action tendencies”

What did NOT predict bulimic tendencies:

• total television exposure

What DID predict bulimic tendencies:

• Body Mass Index
• endorsement of the thin ideal
• the extent to which participants compared themselves to the characters

Again, Botta also reported that exposure to “thin dramas” predicted bulimic tendencies, but not in the direction you might think. The more they reported being exposed to “thin dramas”, the LESS they reported engaging in bulimic behaviors. Regardless, with a p-value of less than .05, I don’t believe it needs to be explained as anything more than an odd finding. In addition, there was an interaction of thin ideal endorsement with the questioning of characters that was difficult to fully interpret or explain.

So, do Botta’s findings support the claim that media images cause eating disorders?

Not. At. All.

Do they suggest that media images cause eating disorders?

Nope.

Do they suggest that media images cause body dissatisfaction?

Again, nope.

Do they suggest that media images teach women that thin is best?

No.

What do her findings suggest?

The amount of media exposure, even specifically to thin characters, is not directly related to whether women think that thin is best, body dissatisfaction, drive for thinness, or bulimic tendencies.

If her findings are similar to those of other researchers (and they are), they actually kinda support Ben’s assertion [we're still in the "what if we were talking about body image" state].

What Botta’s findings also suggest: Women who compare themselves to characters on TV are more likely to think that “thin is best” than women who do not compare themselves to those characters. Also, thin ideal endorsement and current BMI are related to body dissatisfaction (not a surprise), drive for thinness, and bulimic behaviors.

So, it seems to me that what these findings tell us, beyond “media is not the problem”, is that women who are unhappy with their bodies engage in behaviors which are likely to make them even more unhappy. They have warped views of what is ideal and compare themselves to people who, in their judgment, fit that view. How this equates to “every word of it disagrees with Radford’s assertion that media images have no relationship to body image” I don’t know.

I think it is plain, though, that the issue is complex and so is the literature about it. It is fairly easy for the average human to view this kind of literature as supporting their current view of the world. It is also human to defend that view, even when it is not supported, and to ignore explanations of why they should be skeptical. That’s one of the reasons we need Skeptics (like Ben Radford).

Some References

Botta, R. (1999). Television images and adolescent girls’ body image disturbance Journal of Communication, 49 (2), 22-41 DOI: 10.1111/j.1460-2466.1999.tb02791.x

Dailey, S.E., & Buunk, A.P. (2009). Female body dissatisfaction after exposure to overweight and thin media images: The role of body mass index and neuroticism Personality and Individual Differences, 47 (1), 47-51 : 10.1016/j.paid.2009.01.044

Greenwood, D.N., & Pietromonaco, P.R. (2004). The interplay among attachment orientation, idealized media images of women, and body dissatisfaction: A social psychological analysis in The psychology of entertainment media: Blurring the lines between entertainment and persuasion. Shrum, L. J. (Ed.), 291-308 Other: 2003-88226-016

Jarry JL, & Kossert AL (2007). Self-esteem threat combined with exposure to thin media images leads to body image compensatory self-enhancement. Body image, 4 (1), 39-50 PMID: 18089250

Roberts, A., & Good, E. (2010). Media images and female body dissatisfaction: The moderating effects of the Five-Factor traits Eating Behaviors, 11 (4), 211-216 DOI: 10.1016/j.eatbeh.2010.04.002

Siever, M. (1994). Sexual orientation and gender as factors in socioculturally acquired vulnerability to body dissatisfaction and eating disorders. Journal of Consulting and Clinical Psychology, 62 (2), 252-260 DOI: 10.1037//0022-006X.62.2.252

Related posts:

1. Reaching Out and Geeking Out
2. Narcissism + Incompetence = Ignorance and More Incompetence
3. Is that a fallacy in your pocket or can you cite some sources? A response to Women and Feminism at TAM8

Dragon*Con is now just over three weeks away and the schedule is as solid as these things get, so I’m giving you the highlights as promised. I will be one BUSY Skeptic! [Edit: By the way, Dragon*Con will take place over Labor Day weekend, September 3rd through 6th.]

For those who have not heard of it, Dragon*Con an enormous SciFi/Fantasy convention held in Atlanta over Labor Day weekend. Skeptrack, the skeptic fan track, is the brainchild of Derek Colonduno and Robynn McCarthy (A.K.A., “Swoopy”), hosts of Skepticality. As usual, they work their butts off and many skepticism activists do their best to make that work count by participating in talks and panels designed to spread the critical thinking bug to geeks everywhere.

Dragon*Con is very different from conventions like The Amaz!ng Meeting, and not just the costumes. TAM is, at heart, a Dog-and-Pony-Show for skepticism. At Dragon*Con activists do the work of activism and education. There are a number of fan tracks, including a Podcasting Track directed by Swoopy, and Space & Science Tracks, who work closely with Derek and Swoopy. In addition to the obvious Star Wars Track, Trek Track, and Whedonverse Track, there are literature, costuming, and writing tracks. There are also two tracks related to Skeptrack: Paranormal Track and X Track.

The former is self-explanatory, I hope, and this year I will be attending a workshop on that track by Ben Radford on investigating the paranormal (details below). I am very interested in the methods used for this kind of thing as they differ a great deal from the kind of work that I do.

Last year most of Skeptrack was streamed live, but there were some problems. I believe they will try this again this year with, hopefully, fewer headaches. If so, I will provide a link on Facebook and Twitter as soon as it is available, which might be as late as the first day.

If you are attending or planning to stream it live, here is where and when you can find me:

Skepticism, Scams, & Consumerism: Ranging from psychology behind sales to consumer rights, we discuss how as skeptics we identify and challenge dodgy products and pseudosciences.
Time: Friday 2:30pm – 3:30pm
Location: Hilton 205/206/207
I will be moderating this panel and talking about the psychology of purchasing behavior. Panelists include Matt Lowry, Rachael Dunlop, Richard Saunders, & Tom Merritt.

The Calculus Diaries – Lose Weight, Win in Vegas, Survive a Zombie Apocalypse: Fun examples of math applications in the real world and why it’s important to understand even just the basic concepts.
Time: Friday 5:30pm – 6:30pm
Location: Hilton 202
This panel is on the Science Track and will mostly be the work of Jennifer Ouellette, whose book with that title will be released August 1st, just in time for me to read it on the plane. As a statistician, I hope I can contribute something worthwhile to the discussion.

How Your Brain Works, and How to Fool It: Our perception of reality is driven more by expectation, belief, and desire than by sensory input. An examination of how we fool ourselves.
Time: Friday 8:30pm – 9:30pm
Location: Hilton 202
NOTE: This is not the original title and I think it is a little misleading. The original title was “What You See Is [not always] What You Get (or WYSInaWYG)”. The topic is not about how to fool your brain, but how your brain fools you. I believe Jason Schneiderman plans to join me and would be a welcome addition.

Skepticism and Sexuality: When do we get skeptical about sex, the media? When alien cults want to save African clitorises, this panel is here to discuss the facts.
Time: Friday 10:00pm – 11:00pm
Location: Hilton 205/206/207
I will bring with me some of the most interesting myths about the psychology of sex, courtesy of my friend a colleague who teaches the best course on the topic evah. My fellow panelists include Heidi Anderson, Ben Radford, Kylie Sturgess, & Ginger Campbell. Desiree Schell will moderate.

Naturally Skeptical? The Psychology Behind Being a Skeptic: A round table discussion on the factors that do (and don’t!) contribute to becoming a questioner of the paranormal and pseudoscientific.
Time: Saturday 10:00am – 11:00am
Location: Hilton 205/206/207
I will once again take the reigns as moderator and lead a discussion of the psychology of critical thinking and open-mindedness. I am very pleased that Scott Lilienfeld has decided to join us. He has written quite a bit about pseudoscience in our field, including his latest collaboration, 50 Great Myths of Popular Psychology: Shattering Widespread Misconceptions about Human Behavior. Other panelists, who are not exactly chopped liver themselves, include D.J. Grothe, Matt Lowry, Kylie Sturgess, & Pamela Gay.

Mathematical Modeling Pitfalls: Model don’t always accurately reflect messy reality, particularly where human behavior is concerned. Algorithms can only mimic human behavior, and there is a lot of room for bias and error as a result.
Time: Sunday 11:30am – 12:30pm
Location: Hilton 205/206/207
I am not actually on the schedule for this, but Jennifer Ouellete invited me to contribute when I mentioned my love-hate relationship with modeling and my frustrations in teaching that models are not what they model. If I get it together in time, I will present what I think is a cool visual of what you can do with a simple model that shows how they can be effective science. Well, I think it’s cool, anyway!

Raising Skeptical Geeks: A few known skeptical parents talk about issues and how they a raising their kids to be better rational thinkers.
Time: Sunday 2:30pm – 3:30pm
Location: Hilton Crystal Ballroom
This is sure to be the highlight of my Dragon*Con experiences as I love to talk about my kids. I am collecting stories and funny quotes to share. I am also thrilled that fellow panelists will be three good friends and one of the geekiest dads (by all appearances) around: Daniel Loxton, Heidi Anderson, Desiree Schell, & Adam Savage.

Skepticism and Education: JREF now has a Director of Educational Programs – what else is being done out there and how can skeptics help educate the next generation?
Time: Sunday 4:00pm – 5:00pm
Location: Hilton 205/206/207
Of course this is on my list of favorites as well as one of the most important of the panels on which I will serve. Other panelists include D.J. Grothe, Michael Blanford, Daniel Loxton, Pamela Gay, & Matt Lowry. Kylie Sturgess will moderate.

Women: Myths, Feminism And Skepticism: Puzzled by feminine mystique? Searching for your ‘Inner Velma’? Join investigators on gender, pop-culture and what science REALLY tells us!
Time: Monday 2:30pm – 3:30pm
Location: Hilton 205/206/207
Oh, so many myths, so little time! This topic is rich and Heidi Anderson, Desiree Schell, Pamela Gay, Kylie Sturgess, and myself are looking forward to a fact-packed dicussion.

Skeptrack Sign-Off Wrap Up and Feedback: Join the skeptrack guests and speakers for a discussion about how things went, last minute news, and how we can make things better next year!
Time: Monday 4:00pm – 5:00pm
Location: Hilton 205/206/207
Unfortunately, I will not make this discussion and also make my flight home. I am sure, though, that there will be much greatness in the room.

That wraps up my obligations. Here are just a few the events I am looking forward to attending as an audience member (if possible – the * indicates an event I cannot attend due to a scheduling conflict, but recommend):

CSI: Paranormal
Time: Friday 1:00pm – 2:00pm
Location: Hilton 205/206/207
Presenters/Panel Members: Joe Nickell

*Skeptically Speaking Live!
Time: Friday 8:30pm – 9:30pm
Location: Hilton 205/206/207
Presenters/Panel Members: Desiree Schell

Skeptical Coffee Talk
Time: Saturday 8:30am – 9:30am
Location: Hilton 205/206/207
Presenters/Panel Members: James Randi, D.J. Grothe, & Joe Nickell

Paranormal Investigation Workshop
Time: Saturday 1:00pm – 3:30pm
Location: Sheraton
Presenters/Panel Members: Ben Radford

*I Very Much Doubt That!
Time: Saturday 1:00pm – 2:00pm
Location: Hilton Crystal Ballroom
Presenters/Panel Members: James Randi

*Mystery Investigators Children’s Show
Time: Saturday 2:30pm – 3:30pm
Location: Hilton 205/206/207
Presenters/Panel Members: Richard Saunders & Rachael Dunlop

Monster Talk Podcast Live
Time: Saturday 4:00pm – 5:00pm
Location: Hilton 205/206/207
Presenters/Panel Members: Blake Smith & Ben Radford

Skeptic Zone Live!
Time: Sunday 7:00pm – 8:00pm
Location: Hilton 205/206/207
Presenters/Panel Members: Richard Saunders, Rachael Dunlop, Kylie Sturgess, & Brian Brushwood

Skepticism 2.0: Blogging
Time: Monday 11:30am – 12:30pm
Location: Hilton 205/206/207
Presenters/Panel Members: Daniel Loxton, Rebecca Watson, Brian Dunning, Rachael Dunlop, & Kylie Sturgess

Martial Arts Mysticism
Time: Monday 1:00pm – 2:00pm
Location: Hilton 205/206/207
Presenters/Panel Members: John Clements

You can find the full schedules as they become available on the track websites. I hope to see you there!

Related posts:

1. While I Was Sleeping
2. The Amaz!ng Meeting [TAM9]: Some Notes
3. Is that a fallacy in your pocket or can you cite some sources? A response to Women and Feminism at TAM8

In other words, does this look like anything other than what it clearly is? If so, what?

By the way, if you have seen this image before (it was tweeted this morning), please say so.

Then, move on to the previous post to find out what I “saw” and what I hope to learn.

Scroll down for comments!

Related posts:

1. Naughty Elmo Revisited

SPOILER!!

Read this post first, please.

SPOILER STARTS HERE

Here’s the story…

I caught this tweet this morning:
@BadAstronomer: Beam me up some breakfast. Warp factor three egg omelette. http://twitpic.com/27mfxu

I will pretty much click on anything that starts with “beam me up”, so I did and, as you know, I saw this:

Well, I really saw this:

I even heard theme music in my head. I imagined “NCC-1701″ written on the hull in salsa. Because the original series is the cornerstone and James T. Kirk is the greatest captain of all time, space, film, and literature.

But… would I have perceived it as such if:

• the tweet did not include references to Star Trek?
• the tweet was not from Phil Plait, whom I know to be an avid fellow scifi fanatic?
• the tweet was not from Phil Plait, whom I know to be at Comic-con right now for several reasons?
• I was not lamenting not being at Comic-con myself?
• had not just rearranged a shelf on which our Original Series collection sat, taunting me?

Would Phil Plait had perceived the icon of the god that is Captain James T. Kirk if he were not at Comic-con? If he were not a sci-fi fanatic? You get the picture.

The experience of seeing the DVDs or being at (or thinking about) Comic-con is called a “prime” in the psychological literature, because it activates related information, lowering your perceptual threshold for it. The prime prepares you, in a way, to receive related input.

The tweet itself is a suggestion; it is information about to experience. You then use, in part, top-down processes to interpret the image given that information.

Some good examples of how this works using another sense (hearing) can be found in one of my first blog entries about a very naughty Elmo doll.

Priming effects sometimes confound pareidolia (the tendency to perceive familiar patterns in otherwise meaningless input). However, in laboratory tests, such as that conducted by Vokey & Read and some work a student of mine presented at the Western Psychological Association convention this year, little is perceived from very ambiguous sensory information without priming or suggestion. The “Elmo” post includes several illustrations of this.

So, the question remains: Did we see the Enterprise because it really does look like the Enterprise, or we just amazingly geeky?

Either way, I’m going to add it to my collection. If my readers are even HALF as geeky as me…

Vokey JR, & Read JD (1985). Subliminal messages. Between the devil and the media. The American psychologist, 40 (11), 1231-9 PMID: 4083611

Related posts:

1. Naughty Elmo Revisited
2. Reaching Out and Geeking Out

In my mini-review of The Amazing Meeting 8 I mentioned that there were two very dark spots in an otherwise amazing (sometimes the word just fits) weekend.

I was not ready to discuss these in detail, but when I stumbled over this blog post by Blag Hag Jen McCreight, I felt that at least one should be discussed and I would like to do so through the filter of one of the best talks of the weekend, given by Massimo Pigliucci.

McCreight addresses the question of sexism, saying:

The one annoying thing I saw was the perpetuation of the Sexy vs. Smart binary in talks.

I saw none of this in talks. She gives two examples: Michael Shermer’s talk included a Los Angeles County Fair commercial from a series which has been shown for several years now.

This series is meant to portray a stereotype of geography, not the attractiveness (or the gender; they could have easily used the dumb surfer boy image) of the actors. I can understand this getting past much of the audience. Those of us who live in southern California and have seen the entire series likely take it for granted.

That said, the video seemed to have little to do with the rest of his talk and seemed a bit too “look at these dumb people”; I cringed myself when I saw it. So this is probably worthy of discussion, but I do not think it is a strong example of associating appearance with intelligence.

McCreight also accuses SkepDoc Harriet Hall of sexism:

Whenever she mentioned Jenny McCarthy in her talk as an example of someone saying something stupid (which Jenny McCarthy certainly does often), she would include a picture of her bending over in a bikini or some other scantily clad outfit. Why was this effective? Why not use a photo of Jenny McCarthy in a suit?

Why is it sexist for Harriet Hall to show Jenny McCarthy, a former model and Playboy bunny, in a swimsuit rather than something more modest? If McCarthy were, say, a cashier by trade, the image of her in a cashier’s smock would have been just as appropriate, no?

The purpose of the images was to show that frightened parents will favor the message of someone entirely unqualified to give medical advice over their MD. McCarthy is qualified to have her picture taken and did so “scantily clad” for years.

McCreight also repeated something central to her own talk (which I am not ready to review in its entirety):

The stereotype goes that women can sexy/attractive/beautiful and stupid/ditsy/unscientific, or they can be smart/witty/scientific and frumpy/plain/ugly. This myth annoys the hell out of me, especially because it’s so common.

This is where I put on my “Massimo” glasses and discuss expertise.

Media stereotypes are not “myths”. In fact, they do not necessarily reflect what individuals in society actually believe. These definitions are important, especially when one’s argument relies on them. When you make statements about one thing (media portrayals), but you are really talking about something else (behaviors and attitudes), you need to prepared to cite sources which clearly show that these are interchangeable; the distinction matters.

The truth is that attractive persons are more likely to be associated with an occupation that is held in high regard, including scientist, than less attractive persons. That’s the halo effect. It is very well-established in the psychological literature and not limited to men or even human beings.

Some of the points Jen made are valid criticisms, but the valid criticisms are overshadowed by vague, uninformed statements. Many of the points rely on whether her general claims of “this is what people think” are accurate. She does not cite sources which show that she knows “what people think”, nor is her background in psychology or a related field, which might provide some evidence of expertise in this area.

“I’ve seen it” is not evidence, something a young scientist in training (and many older, experienced ones) must constantly remind themselves in order to overcome our brain’s desire to think that it is.

McCreight defended TAM organizers by repeating a statement made many times by Jeff Wagg about speakers at TAM7:

Last year, 8 women were invited to speak at TAM. 2 said yes. 1 of those women had to cancel.

I have never heard Jeff compare this with the number of men who were invited and how many of those accepted or canceled. Without that comparison, this information tells us nothing.

Frankly, however, I care much more about the quality of the speakers than their gender, but given the number of high-quality speakers available who are women and the ratio seen at other events, the lopsidedness at TAM in past years was a bit disturbing. I thought they did a great job all around this year and didn’t need to be defended.

I thought the sex workshop was on Sunday…

Regarding the “Feminism & Skepticism Workshop”, although I am not the person she quoted, I was sitting directly behind McCreight and walked out when “Angry Vagina Craft Time” was announced.

My take? There are three criteria which should have been met for a topic or activity to be included in this workshop:

1. It is a feminism issue.
2. It is a skepticism issue.
3. The discussion is well-researched and well reasoned.

Although there were definitely some good points, much of what was discussed prior to “Angry Vagina Craft Time” failed to meet one or more of these criteria, especially #3.

Asking people to make vaginas (term used loosely) out of felt and googlie eyes did not make me uncomfortable, but infantilizing women’s genitalia and calling it “light humor” made me a bit angry – yes, I had an angry vagina. And an angry jaw. It could have made many women very uncomfortable, yet it served no purpose that I could see short of a “fuck you” to those who have criticized the workshop’s organizers in the past for such things.

I left because I had seen enough.

Overall, in regard to sexism at TAM8, I thought this year was a huge improvement over last. I attribute this largely to a different mix of attendees. I really wish that friends who were turned off by the culture last year could have experienced it. Perhaps they would see the community differently.

To sum up my experiences and in answer to McCreight’s questions: There were exactly two times during the weekend when I was offended. That workshop was one of them. Ironic, isn’t it?

Related posts:

1. Reaching Out and Geeking Out
2. This Stuff is Not Easy!
3. There is No Debate Over Evolution

After a few seconds I realized what she was dealing with and asked, “The Aporkalypse?”

Of course we were talking about the H1N1 scare which, at the time, was still called “swine flu”.

When I explained that, no, I couldn’t give her anything because we simply didn’t know enough, she said, “But these people are buying boxes full of hand sanitizer!”

Now, that I could help her with. There is literature addressing the ability of hand sanitizer to prevent illness, but we simply did not know enough about this new strain of virus to predict what would happen.

Nevertheless, most people jumped to conclusions. Those who did not panic summarily dismissed the issue. This kind of thing recyles itself often (climate change, economics, etc.) and other issues, such as scientific illiteracy, mask the more basic phenomenon: We don’t like uncertainty, so we tend to make decisions about what is true with the information we have, even if that information is insufficient.

This is particularly problematic for skeptics who, once they become fired up about a topic, are often blinded by their passions. Of course, this is true for all of us, but since I spend a lot of my time with skeptics, my anecdotes are about them.

For example, in May of last year, I was a little surprised about the reaction, especially of PZ Myers to this court ruling reported by The Orange County Register:

James Corbett, a 20-year teacher at Capistrano Valley High School, was found guilty of referring to Creationism as “religious, superstitious nonsense” during a 2007 classroom lecture, denigrating his former Advanced Placement European history student, Chad Farnan.

The reader comments are more of the same polarized complaints of pursecution that I have come to expect from Pharyngula, but there are a few voices of reason.

If this is not the first time you have read this blog, you might wonder why I am not upset by this ruling myself. I am not upset because I actually read the ruling and it seems obvious to me that the ruling itself is a win for science education.

If you read the original complaint, you will discover that this was a public high school history teacher whose mocking of religion during lectures would put PZ to shame, yet comments such as “When you put on your Jesus glasses, you can’t see the truth” were ruled as having a pedagogical, secular purpose. In fact, a very long list of clearly derogatory comments (some of which are quoted here) were fine, but this single comment was ruled as having no secular purpose because it was made outside the context of the classroom in his role as an advisor for the school newspaper.

Ed Brayton covers this pretty well, I think, in two posts on the matter.

I bring up this example because the reactions on Facebook and other blogs were much like PZ’s knee-jerk, “How can this happen!” and I think that many people would have a completely different perspective if they took a few minutes to get some facts.

I now realize that it is highly likely that PZ’s reaction would be the same, however, and should have know that at the time, given his last paragraph:

Thirdly, and this must be said, Chad Farnan [the high school student who filed the law suit] is a self-righteously moronic creationist wanker who deserves to have his stupidity pointed out publicly, in the classroom and out of it, far and wide. Spread the word.

Where am I going with this?

Earlier this week, California health authorities released a statement declaring an epidemic of whooping cough which has killed 5 babies already this year. There were several articles including New York Times and Consumer Reports.

Being particularly sensitive to antivaccination propaganda and remembering Dana McCaffery, I was immediately interested and read several articles. I could find nothing to indicate that refusals to vaccinate were the primary culprit.

Then I saw a link to an article on my Facebook feed and these were the first few comments:

“Thanks, Jenny.”

The next post was a link titled “Jenny McCarthy kills another five infants in California.”

*sigh*

I am as anti-anti-vax as they come, but I find it distressing when my side sounds like the other side. I felt much as I do when a pharmaceutical company commits fraud.

The comment about ignorance is particularly troubling, since the commenter is ignorant themselves – ignorant of the fact that Whooping Cough did not simply “appear” anywhere. Vaccinations have done a lot to eradicate diseases like small pox, but Whooping Cough has never been close to being stamped out.

I wondered if anyone actually read the article, because there was nothing in it about vaccine fear.

There are many, many sources to blame for the increases, and vaccinations are certainly the best approach we have to stop the spread of infectious disease, but fear of vaccinations is not likely to be among the most important factors in undervaccination in this case. In 2005, the year of the last abnormal outbreak, research showed that it was not even a significant factor:

Studies have yet to show any obvious reason for the increases. No significant outbreaks have been traced to the children of parents who oppose childhood vaccination. “We’ve looked, and we can’t see any obvious connection,” Woodfill said.

There are many things which much be taken into account in this case, including (some of this information is included in the same 2005 article quoted above):

• Outbreaks of pertussis (Whooping Cough) are cyclical and spikes are normal. As we all know from climate change research, data points and bits are often misleading. “Four times as many cases as last year” is very scary, but it is taken out of context.
• The rate of pertussis has been increasing since at least the 1980s. This may or may not be a “true” increase because diagnosis for pertussis has improved with the availability of a better test in 1995 and with better awareness. Many feel that pertussis was likely to have been underdiagnosed in the past because the symptoms (especially in adults) are similar to bronchitis or flu.
• Immunity often fades without boosters, something we have not known until fairly recently. Pregnant women are often tested for antibodies to some diseases (I discovered I needed to be re-innoculated for Rubella myself when I was pregnant with my first child), but most adults don’t bother with vaccines because they think their childhood vaccines were sufficient.
• A booster is recommended for children entering middle school and many states require it for school, but not California.
• California has a large population of immigrants. The 5 children who died were latino. The areas with the highest numbers of cases are areas with large segments of migrant farm workers – poor and uneducated, with little access to quality health care. Although vaccines are free to children on Medical and through other programs, how are parents to know that these vaccines are needed if they do not have proper preventative care? How will they know where or how to receive them?
  Furthermore, many of these immigrants are undocumented. The are unlikely to seek these services or information if they are afraid of deportation.

A few articles, particularly in newspapers for affluent counties like Marin, have cited vaccine refusal as the culprit. They may be correct, but in each case their evidence was the hunch of a single doctor they interviewed. That’s just not enough evidence for a reasonable skeptic.

My point here is not that anti-vaccination propaganda is acceptable. My point is that this issue is complicated, and knee-jerk reactions without even taking the time to read the the information on which one is commenting are irresponsible and damaging.

I left a comment on that thread and the first response was disheartening:

Ok. So the article doesn’t represent the world JMcC has made, but rather the one she wants. Abortion clinic bombings aren’t universal either, but I’m still comfortable using a bit of hyperbole when telling Focus on the Family to fuck off. It’s hard to fit entire position papers into FB threads, and “Immigration Body Count” would be a bit of a red herring.

First, I do not believe that Jenny McCarthy wants children to die. I have no problem calling her a moron as I have in the past. People who have had the truth explained to them as often and as simply as she has and still insist on moving forward with their deadly campaign have earned that much. However, ignorance, arrogance, and incompetence are not wishing children dead, nor is it a reason to wish someone dead as one of the other commenters did.

But what really strikes me is the odd reference to a red herring which is backwards. There is no “hyperbole” here and going off on Jenny McCarthy in reaction to this announcement is a bit like shooting someone for dropping their cigarette butt into the lake while watching a barge dump 3 tons of trash into it 50 yards away.

This is a fallacy of relevance and it chips away at the credibility of anyone who tries to argue that such propaganda is harmful.

Not long after reading this, I saw a link to Bad Astronomy. It is a passionate and certainly heart felt tyrade over the anti-vaccination efforts of people like Jenny McCarthy and Meryl Dorey, but it starts with a discussion of the Whooping Cough epidemic as its title suggests. Phil is one of the more rational among well-known skeptics and I expected more from him.

So I left a comment.

Phil did reply to my comment, and at some point he added a caveat to his post. I do not generally reread posts unless asked, but part of his reply did not sit well with me. In fact, it angered me.

But it’s also true the movement has been making footholds all over the country, and I will take opportunities to point that out when I can.

I felt dismissed, and my anger is clearly reflected in my response:

Defensive much?

By this logic, I should write nice long rant about subtle sexism every time women only make up 20% of a company’s employee list, regardless of how many women applied for those jobs.

And have you thought about what this does to your credibility? To our credibility?

Yes, the anti-vax movement SUCKS. But I’ll repeat what I said because I think it’s important enough to say again: knee-jerk reactions don’t help.

Neither does a defensive response when someone points out that you may have reacted emotionally rather than rationally. You’re human. People will actually admire you more if you admit that.

Just sayin’.

He rebutted that I had overreacted.

Knowing how Phil feels about this topic, I believe quite strongly that, if the shoe were on the other foot, he would have lectured me on the seriousness of the matter. Crying wolf while the sheep are dying of starvation does more than harm Little Boy Blue’s credibility. It prompts people to devote resources to a wolf hunt which are better spent feeding the sheep.

And he would be right to do so. How can we call these people ignorant if we knowingly incite others to make statements of ignorance, like “The disease is making a come-back” and “Thanks, Jenny”? … and if we never correct them? or ourselves?

If I had spent a little more time editing, I probably would have left out the two words of sarcasm, but I cannot say for certain that I would. I am a little person with fewer than 100 readers and I felt slapped down for criticizing someone with millions who is hero-worshipped by most of the community. If you doubt his reach, try Googling “Bad Astronomy” sometime. It took every ounce of courage I have to post a comment in the first place, knowing that many people I consider friends would not take kindly to their friend being criticized.

I felt that this issue is extremely important and it is one of many examples in which passion and polarization get in the way rational thinking. Perhaps another is the double-standards we apply to so many situations.

Just as none of us are completely free of sacred cows, none of us are completely objective. And none of us are perfect, either. For my part, I promise that my intent is always to be the best person that I can be; it is never my intent to be mean unless I (or a friend) am under attack.

I also promise that I will insult people. Sometimes that will be intentional. Sometimes it will be because I was thoughtless and insensitive. Most of the time, however, it will either be my own knee-jerk reaction to feeling dismissed, ignored, or slighted. Every time it will be me being me, for better or for worse.

I think I would rather be disliked for being honest with a touch of sarcasm than be liked for having nothing except praise. The latter also means that I have little which is constructive to say.

No related posts.

Totally unrelated lizard image. If you close your eyes, you might be able to imagine a beautiful poster with a couple of people standing in front of it.

Now for the science.

In my experience, most teachers, particularly college instructors, believe that entitlement attitudes, student expectations, study strategies, work habits, and aptitude have changed dramatically in recent years. As students spend more time working in addition to school, they miss more class and devote less time to studying. In addition, because modern technology makes it possible to use (and provide a copy to students) slide show presentations and distribute study guides. As the proportion of courses taught by adjunct faculty, whose teaching load is greater than tenure-track faculty, increases, so does the proportion of exams given in multiple choice format. This, along with outcomes-based learning which shaped students’ habits in elementary and secondary school, promotes rehearsal study strategies.

We hypothesize that the result is a cycle of incompetence as an increasing proportion of college students who believe that memorization of material is an effective way to study and that they are entitled to be given the material and assessments which maximize the benefits of this strategy. In addition, these students do not understand what they memorize and are unprepared for coursework which builds on the material they should have learned. Because they then attribute their failures to outside forces, they do not change their habits and a vicious cycle continues. The literature on academic entitlement is thin, however, some recent findings suggest that academic entitlement attitudes are positively correlated with narcissism, external attribution patterns, feelings of superiority, and exploitative attitudes (Greenberger, et al., 2008; Achacoso, 2002).

To examine these variables, we asked students (N = 95) in upper-division psychology courses to complete a number of measures. Our specific predictions were:

• Entitlement attitudes are positively correlated with external attribution style, narcissism, and feelings of superiority.
• Metacognitive skills are negatively associated with rehearsal learning strategies and positively associated with entitlement beliefs.

As with previous discussions, I will minimize the amount of statistics and technical information I discuss I use to describe the study and its findings, but if you would like more specific information, please feel free to email me.

Our measures:

• The Superiority Scale (Robbins & Patton, 1985).
• Narcissistic Personality Inventory (Raskin & Hall, 1981)
• The Multidimensional-Multiattributional Causality Scale: Achievement Subscale (Lefcourt, Baeyer, Ware, & Cox, 1979). This scale measures the degree to which the participant attributes academic achievement to ability, effort, context (such as the difficulty of the course), and luck. The former two are internal attributes and the latter two are external.
• Learning Strategies Survey we developed to measure the study habits that student think work best. Scores determined relative amounts of passive, rehearsal, and active learning strategies.
1. Passive = attending lectures without taking notes, attending review sessions to study for exams, and using templates or examples to write papers.
2. Rehearsal = using instructor-provided lecture notes, memorizing terms and concepts (e.g., flash cards), using study guides, and studying from sample questions or past exams.
3. Active = taking notes in class, active reading from learning objectives, and drafting & revising papers incorporating feedback.
• An Academic Entitlement Survey we developed which encompassed expectations about the source of grades, what students believe should be expected of them, what they believe instructors should provide, etc.
• A Metacognitive Measure: Participants evaluated the validity of ten syllogisms (all invalid), then estimated the percentage they answered correctly. Performance in this task is fairly difficult for most people to judge.
• In addition, we asked participants to indicate the number of hours studying each week outside of class they believed was reasonable to do well in the course and how many class meetings per semester it was reasonable to miss.

The number of variables and the complex relationships we hypothesized make the findings a bit confusing, but it can be simplified to a series of strong correlations.

Not at all surprising was that the more class meetings students thought it was acceptable to miss, the less time studying they felt should be needed to to do well. What is surprising is that the more missed class meetings they thought were acceptable, the more they felt that academic achievement is determined by luck. Attribution to luck was also positively correlated with rehearsal learning strategies.

The more students attributed achievement to context, the more entitleed they felt.

In addition:

• Entitlement attitudes were positively correlated with narcissism and superiority, a finding which is consistent with most studies on entitlement.
• The greater the entitlement attitude, the more likely students were to use rehearsal learning strategies and the less likely they were to use active strategies.
• The more entitled students felt, the more they attributed academic achievement to external causes (context and luck) and the less they attributed it to effort (attribution to ability was not correlated with any variable).
• Superiority attitudes were positively correlated with attributions to context.
• The most telling finding and the strongest correlations: Overestimation of performance was positively correlated with estimated performance, but negatively with actual performance. In other words, the better students thought that they had done on the argument judgments, the worse they actually performed and more they overestimated their performance.

The worse students performed, the better they thought they'd performed.

These findings are consistent with those of Kruger and Dunning (1999), who found that incompetence is perpetuated by ignorance of incompetence. (Dubbed “The Dunning-Kruger Effect“)

Rehearsal learning strategies were correlated with entitlement and external attributions, suggesting that students who believe that rehearsal strategies work best are more likely to feel entitled to use them and less likely to attribute their failures to those strategies or their own efforts. Instead, they will attribute them to external forces such as luck, instructors, and other situational factors. As a result, they continue to use the same failed strategies.

Student use rehearsal strategies which are highly ineffective, but since they attribute failures to external factors such as context and luck, they do not recognize that they do not understand the material. Thus they are stuck in a cycle of metacognitive ignorance and rehearsal strategies ensuring that they continue with poor strategies and poor outcomes, remaining ignorant of the need for change.

Before we publish these findings, we intend to test the validity and reliability of our original measures and use structural equation modeling to map the complex relationships among the variables. This should be completed in the fall with new participants and I fully expect these findings to be replicated.

References:

Anchacoso, M.V. (2006). “What do you mean my grade is not an A?”: An investigation of academic entitlement, causal attributions, and self-regulation in college students. Dissertation Abstracts International, 67 (6-A) : 2006-99023-155

Greenberger, E., Lessard, J., Chen, C., & Farruggia, S.P. (2008). Self-entitled college students: Contributions of personality, parenting, and motivational factors Journal of Youth and Adolescence, 37, 1193-1204 : 10.1007/s10964-008-9284-9

Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments Journal of Personality and Social Psychology, 77 (6), 1121-1134 : 10.1037/0022-3514.77.6.1121

Lefcourt, H., von Baeyer, C., Ware, E., & Cox, D. (1979). The multidimensional-multiattributional causality scale: The development of a goal specific locus of control scale. Canadian Journal of Behavioural Science/Revue canadienne des sciences du comportement, 11 (4), 286-304 DOI: 10.1037/h0081598

Raskin, R., & Hall, C. (1981). The Narcissistic Personality Inventory: Alternative Form Reliability and Further Evidence of Construct Validity Journal of Personality Assessment, 45 (2), 159-162 DOI: 10.1207/s15327752jpa4502_10

Robbins, S.B., & Patton, M.J. (1985). Self-psychology and career development: Construction of the Superiority and Goal Instability Scales Journal of Counseling Psychology, 32 (2), 221-231 : 10.1037/0022-0167.32.2.221

Related posts:

1. Know Not Only What You Know, But Why and How You Know It
2. Reaching Out and Geeking Out