The use of self-reported data
In many psychological studies, data is collected by means of interviews, surveys and/or questionnaires. These research methods are heavily dependent on self-reported data. Often this is the only reasonable type of data that can be collected; however, there are several limitations to the use of self-reported data. It is important that when we evaluate research we recognize these limitations. These limitations include:
- Expectancy effect
- Social desirability effect
- Conformity effects
- Reconstructive memory
- The availability heuristic
- The peak-end rule
As part of any research in psychology, demand characteristics may play an important role in undermining our results. Expectancy effect is when a participant thinks that they know what a researcher is hoping to achieve and then tells the researcher what s/he wants to hear. Another demand characteristic is the “social desirability effect”, where a participant says what s/he believes will make them look good. For example, when people are asked how they feel about a social issue, they may simply say what they believe is “socially appropriate” because they do not want to seem to be uneducated, racist, sexist, or old-fashioned. Participants may also simply conform to what others are saying, even though they will believe that what they are saying is, in fact, their own opinion.
Another problem with self-reported data is that it is often retrospective in nature - that is, people are asked to think back on moments in their lives and give the researcher information. This may include a description of health habits, time in school or childhood experiences. Many of these memories may have been distorted over time because of the reconstructive nature of memory. In addition, the availability heuristic plays a key role in how we assess the past. This bias means that we form an opinion based on what information readily comes to mind. So, if you are able to more quickly recall the times when a teacher scolded students rather than praised them, then the perception would be that the teacher was rather mean and unfair to students. When asking people about their eating habits, if healthy meals come to mind, they will report healthy eating habits, even if this is not necessarily representative of their regular diet.
Finally, there is the peak-end rule. According to the peak-end rule, we judge our experiences almost entirely on how they were at their peak and how they ended. Other information is not lost, but it is not used. This includes net pleasantness or unpleasantness and how long the experience lasted. Daniel Kahneman argues that people seem to perceive not the sum of an experience but its average. Here is an example of a study done by Kahnemann et al (1993).
Research in psychology: Kahnemann et al (1993)
Researchers were asked to hold their hands in painfully cold water. They used their free hand to indicate their level of pain on a keyboard. Each participant was asked to carry out two tasks. The first task was to immerse their hand in ice-water for 60 seconds. In the second task, they were asked to immerse their hand in ice-water for 90 seconds. The first 60 seconds were identical to the short task. At the end of 60 seconds, the researcher opened a valve that allowed slightly warmer water to flow into the tub. The temperature rose by just 1-degree Celsius. The pain ratings for the two tasks were the same for the first 60 seconds.
The participants were then told that they would do one more task. They could either repeat the first task or the second task. Rationally, you would see that the second trial was worse. It was longer and the participant was exposed to more pain. However, 80% of the participants chose to redo the second trial, simply because the trial ended with a slightly more pleasure feeling, discounting the overall pain of the experience.
The peak-end rule is often a problem in self-reported data. Suppose a couple goes on a holiday and has a great 12 days in the sun on a spectacular beach. For the last two days of holiday, not only does in rain incessantly, but their hotel room was robbed while they were at the all-you-can-eat buffet for dinner. When asked about the holiday, it is highly likely that the couple will judge the holiday as a “disaster,” even though the vast majority of the holiday was excellent. In the study of why relationships fall apart, a 20-year marriage may be judged as horrible and unfulfilling, even though it was only the last two years which were difficult for the couple. Comments like, “I don’t think that I ever loved him” may reflect peak-end rule rather than fairly represent the overall marital experience.
When evaluating a study and commenting on the use of “self-reported data,” there are different reasons why this type of data may be problematic for a particular study. Always try to clarify why the use of this data is limited in order to show the highest levels of critical thinking.