Good Measures

In This Text

Learn about the different methods, strengths, and limitations of self-report data
Learn about the different methods, strengths, and limitations of observational data.

Hard Data in a Soft Science

One of the challenges psychologists face in their attempts to be a REAL SCIENCE ™ is that their data is particularly hard to collect. Unlike physical variables like temperature or mass, psychological variables are often internal to people, and defined by mental states that are difficult to observe. As we saw in lecture, even a “simple” expressed behavior like an interruption can be very difficult to measure with high degrees of reliability and validity that we would hope. For a more complex variable such as depression, the task might seem impossible. Indeed, a large part of psychological research is engaging in debate and scholarship about how to best operationalize variables of interest (e.g., how should we define or measure depression?)

A full discussion of the different types of methods psychologists use to collect data is beyond the scope of this author. However, below I’ve tried to outline a few different approaches psychologists take, commenting on their benefits and limitations so you can begin to critically think about whether these methods are, in fact, getting at “THE TRUTH” of what people (or non-human animals) are like.

Self-Reports

One of the simplest ways to collect data on an individual is just to ask them what they are like, and have the person report on themselves (a self-report). There are two different approaches to getting self-reports - survey methods and qualitative interviews.

Qualitative Interviews

One way to get individuals to tell you what they are like is through a structured interview where researchers ask open-ended questions. One such example of this is the McAdams Life Narrative¹. In this structured interview, a trained research assistant asks a set of broad questions to participants over the course of 1 to 3 hours. The research assistant is advised to “feel free to skip some of these questions if they seem redundant or irrelevant, and should follow up with other questions as needed “ but also to “not adopt an advisory or judgmental role, but should instead serve as an empathetic and encouraging guide and an affirming sounding board.”

¹ McAdams talks about his work in this popular press interview and writes about it in this scientific journal review article.

Below is an excerpt from the first part of the interview.

Survey Methods

Qualitative interviews are not very common in psychological research, because they take a lot of time to conduct, and then more time to convert people’s open-ended responses into data (a form of behavioral coding, described in more detail below).

Instead, the majority of self-reports come from surveys. Read about these below.

Definition	A questionnaire where individuals answer specific questions about themselves on a structured rating scale.
Example	“On a scale from 1 (Strongly Disagree) to 5 (Strongly Agree), how satisfied with your life are you right now?”
Benefits	Easy to collect : It only takes a few minutes for people to answer a survey, and the data come in a clean and organized format that requires little effort to analyze. In Part 2 below, you’ll learn how to clean and organize the results of a likert scale. Self-Knowledge Validity : People know things about themselves, often this knowledge is based in “reality”, and sometimes a self-report is the only way to get this knowledge. For example, only you know what was your happiest moment in life, and you probably have an accurate awareness about how anxious you are about the final project in this class.
Limitations	Self-Enhancement / Self-Diminishment Bias : People are often motivated to either enhance or diminish their accomplishments when asked. For example, no student has ever come up to me at the end of the semester and told me, “Professor, just so you know - I cheated on the exam.” Even though they may know they cheated, they don’t want to admit that because our society has norms or guidelines about when cheating is appropriate². Other times, a person may not want to highlight their accomplishments (self-diminishment) because they don’t want to seem like they are bragging. Self-Insight Bias : Sometimes, people also really don’t know what they are like. A person may not know, for example, whether they snore when they sleep (a behavior), how anxious they really are in a situation (an affect), or their patterns of unconscious bias (a cognition).

² Indeed, our society decides what forms of cheating are acceptable.

Self-reports have a bad reputation in psychology, particularly because of the ability for people to engage in self-enhancement or self-diminishment, or have self-insight bias. However, they are a very powerful, and very commonly used method of assessment, even for studies where researchers are able to observe other types of data.

For example, despite all the behavioral data that powerful technology companies like Facebook/Instagram/Meta, TikTok, Twitter/X, etc. collect, you’ve probably seen them ask you to answer some survey. They care about you³, and know that asking you questions about yourself is an important way to show that level of care.

³ …and your clicking on advertisements; hey, it’s hard to distinguish the two really…

Below is one example from some survey when I used to be on Facebook. If Facebook changed its graphic design since I took this screenshot, it’s probably because of some survey feedback they received.

Observations

Observational methods refer to ways in which another individual generates data about a person of interest (often called “the target”). Observational data is important to use when researchers are studying individuals who cannot give reliable and valid self-report data, such as infants, people with cognitive or motor disabilities, or non-human animals.⁴

⁴ If I was a billionaire, I’d fund a team of psychologists to train monkeys to answer surveys. this is maybe why I am not a billionaire. that and the whole “intergenerational wealth” thing / chosen teaching career / lack of a desire to crush others and extract as much wealth from them…hard to know which factor is at play. Life is complex! Let me know if you are a billionaire and want to fund some other ideas / subscribe to my newsletter.

Definition
Examples	Informant Reports. Informant reports are a special form of surveys, where researchers ask friends, family, or strangers to answer survey questions about another person. This is technically observational data, since the people answering the surveys are basing their judgments on their observations of the individual. Behavioral Data. When the variables of interest are physical, then researchers can use measurement tools to directly observe the behavior. For example, researchers wanting to measure stress might measure cortisol by taking samples of saliva from the cheek; researchers wanting to understand the brain look to voxel activation with fMRI, or cortical neuron activation with EEG. Behavioral Coding. Sometimes, it’s easier to have research assistants observe the physical behaviors of interest. For example, research assistants will observe real-life interactions and observe variables such as time spent talking, distance between participants, or provide ratings of how much emotion or anxiety the person seemed to be expressing (using a rating scale). The “strange situation” task (where a parent leaves the room and researchers observe what a child does) is another example.⁵
Benefits	“External”. Researchers like observational methods because they, by definition, require some outward expression of the underlying psychological processes. Professor could go on a rant here and won’t, but psychology has been increasingly fixated on behavior since Watson kicked off the behaviorism movement in the 1920s, and these data are often prioritized, since our field loves predicting people’s actual behavior (so they can exert power over it). Less influenced by self-report biases. Observers are considered to be less biased than individuals, who may be particularly prone to self-enhancement, self-diminishment. And observers may be able to fill in the gaps left behind by self-insight bias. For example, a person’s close friends may know more about the individual’s reputation than they themselves do. More reliable (multiple sources). While there is only one self to provide a self-report, there can be multiple observers. Indeed, observational methods almost always leverage this power and require multiple swabs of spit to get a reliable estimate of cortisol, or have multiple research assistants rate the same behavior to check and make sure they are each getting a similar answer.
Limitations	Time-Intensive. It can be incredibly costly to collect observational data. Yale charges over $600/hour for use of their fMRI machine, and conducting a study to measure a naturally occurring behavior could take years between designing the study, recruiting and training the research assistants to observe the behavior, taking the time to collect the data, and then doing the behavioral coding necessary to convert the observations into numbers. Whew. The Hawthorne Effect. The act of observation can often change behavior. Couples may be less likely to fight if they know they are being monitored by a psychologist, and even light changes its behavior depending on the way it is observed. Researchers can take steps to address this change in behavior - and sometimes the power of the phenomenon is so large that it doesn’t matter if a researcher is there. In couples studies, for example, researchers can get couples to fight by having each person write out a list of things that cause conflict in the relationship, and then having the people trade lists and talk about it. Often however, individuals habituate to the presence of observation - do you think about the fact that your online interactions are being harvested by tech companies every day, or just kinda get used to it? Not all aspects of psychology are observable. A person’s subjective experience matters, and sometimes a self-report is the only way you can get this data. If you look like you are smiling and have the neurotransmitter levels that suggest you are happy and our $600/hour test says that your brain is activated in a way consistent with happiness, but you FEEL and SAY you are miserable….are you happy?

⁵ You can compare this child’s reaction to this child’s reaction. What do you observe? How would you quantify these observations and turn them into cold, hard data?