1.4 Scientific Research Methods: Data Analysis, Theorizing, and Replication

Recall that in an experiment, dependent variables are the measurements that could be influenced or depend on levels of the independent variable. As one might imagine, there are many ways of taking measurements in psychology. We could have people solve problems and measure the percentage of questions that were answered correctly. We could measure heart rate, brainwave activity, or the level of a stress hormone in a person’s saliva. Researchers use well-tested surveys where participants answer questions. Survey questions often use Likert (pronounced LICK-ert) scales, which measure the degree of response. For example, how would you answer the following question using the Likert scale ranging from strongly agree to strongly disagree?

I would like to get, on average, more sleep.

Statistics is the science of collecting, analyzing, and interpreting data, and it plays a significant role in psychological research. Psychological research often tries to convert dependent variables (behavior, attitudes, physiological responses, answers on a Likert scale, etc.) into numbers; in other words, quantify the measurements. Creating ways to quantify the measurements helps reduce the problem of subjectivity (personal interpretation), but it also allows for an analysis of data to determine if the hypothesis can or cannot be supported. 

If a lot of data are collected, a way to describe the data is to look at values that represent middle points, or what statisticians call central tendencies of the data set. The mean Also known as an average, it is a measure of central tendency where the scores are summed and divided by the number of measurements or participants. is the average of all the responses, which is the sum of the responses divided by the number of recordings. Suppose we asked 10 people our Likert scale question about sleep, and we get the results shown in Table 1.1. We can see that the mean score is 2.10, which gives some indication that our group, on average, wants more sleep. The median Used to describe the middle number in a list of numbers, where there is an equal number of values above and below the median number. Often used instead of mean if there are outliers in the data set. is often used when the values get extreme. For example, suppose we asked 10 people how many minutes it usually takes them to fall asleep. We find nine out of 10 take between five and 20 minutes, which is typical, but one takes two hours (120 minutes). The mean time would be 24 minutes, which would not be a good representation of the time it takes most people to fall asleep. The median is the number where there is an equal number of values above and below in the data collected, which in our example is 13 minutes (refer to Table 1.1).

Another critical aspect of a data set is the variance. The standard deviation (SD) A statistical term that represents the variability in a data set. is a measure of variance—the greater the SD, the more spread out the data. This can be visualized with a bell curve or normal distribution curve (refer to Figure 1.16). In a bell curve, the mean and median are at the peak of the curve, and as we go to the right or left there are fewer values in the data set. The wider the bell curve, the more variance in the data set.

Statistics can compare the mean and variance of two or more normally distributed data sets to determine if they meet the standards of being different. Let’s use the example of the sleep and memory test we talked about earlier. Suppose we have 100 participants. Half of the participants are given a set of facts to remember at night and recall in the morning (Group M), and the other half learn the facts in the morning and are tested at night (Group N). Our dependent variable is the percentage of facts that they remember (0 to 100 percent). Our hypothesis is that the two groups will differ in the percentage they remember. However, in our data analysis, we start with an assumption of the null hypothesis The first assumption in statistics that there are no differences between groups or variables. A statistic may allow the researcher to reject the null hypothesis. , which assumes that the data from the two groups come from the same bell curve; in other words, they do not differ. If statistics give us a reason to reject the null hypothesis, we support our hypothesis that the groups are different. Statistics can help us determine if we can reject the null hypothesis to a certain confidence level. Let’s take two possible outcomes. 

Figure 1.17 shows a possible result with Group M, which averaged 53 percent remembered correctly, and Group N, which averaged 50 percent remembered correctly. One can see that there is not a great deal of variance between the two groups. Figure 1.18 shows another possible result, with the two groups having the same mean percentage correct but much greater variance. Statistical tests consider not only mean differences but also the level of variance within each group. Therefore, a statistical test might produce a finding that the two groups are statistically different in Figure 1.17 because the variance is low but find that the two groups represented in Figure 1.18 are not statistically different because there is such high variance. If that is true, the statistics allow us to reject the null hypothesis for the results shown in Figure 1.17 but not for the results shown in Figure 1.18. There are also statistical tests to determine if two measurements show positive or negative correlations, which also take into account the variance of the data. 

There can be ethical problems associated with statistics. Graphs are often presented in magazines, newspapers, social media, and news programs that are designed intentionally to misrepresent data and mislead the public for the purpose of persuasion (refer to “How to Spot a Misleading Graph—Lea Gaslowitz”). One such misrepresentation is a misleading or uneven y-axis. For example, if Figure 1.18 were presented without the error bars and a narrowed y-axis, it would appear as if there were a much larger difference between the groups (refer to Figure 1.19).

Researchers often feel pressure to publish their research to keep their jobs in universities, receive promotions, or receive grants. Also, scientific journals are often reluctant to publish articles that have non-significant or ambiguous results; this is known as publication bias A trend where only research that produces results that support their hypothesis is published. This might encourage unethical data manipulation. (Song et al., 2010). Unfortunately, the pressure to publish may lead some researchers to manipulate their data and alter the way they show their results to make it appear that they have significant findings worth publishing. Sometimes researchers collect many different data sets (i.e., many dependent variables) and then only show positive results and disregard others. Data manipulation and poor research practices can be extremely dangerous: the first study to claim that vaccinations caused autism was based on poor research design and manipulated data. The study was eventually retracted by the journal that first published it, but this one study helped launch an anti-vaccination movement even though numerous well-run and highly controlled studies show no link.

So, what are some ways to determine if you’ve run a good experiment? A mark of a good research design is the extent to which it produces consistent results when variations on the experiment are run again. This is known as the reliability A measure of the strength of a research design or assessment tool by the degree that the same or similar results will be observed if conducted again at a different time. of an experimental design or assessment tool (McLeod, 2007). For example, if a personality test had good reliability, a person would expect the same results if they took it several times. There are different ways of evaluating reliability. One could run the experiment again to see if the same or similar results are produced. Another way is to see if the same results are produced by the same measurements but evaluated by two different people. Suppose a psychologist has two research assistants separately watch a video of people's behavior and follow guidelines for quantifying what they observe. The psychologist could then compare the results of both research assistants to look for consistency in how they rate the videos. 

An experiment is considered to have validity A measure of the strength of a research design or assessment by the degree to which it accurately tests what it was intended to test. if it is accurate in testing what it was designed to test. A way to test validity is by looking at the results in the context of other similar research. Internal validity refers to whether changes to the dependent variable are due to levels of an independent variable and not some other factor. Internal validity is improved by running well-controlled experiments, having accurate and quantifiable measurements, actively reducing subjectivity, having good control groups, and reducing extraneous or confounding variables. 

External validity is the extent to which the results of a study can generalize to other settings or participants. If someone is conducting research on methods of therapy to help people with depression, then one hopes these methods could be used for people outside of the study. The problems in sampling we discussed reduce the validity of an experiment. For example, suppose a researcher wants to conduct an experiment on a method for reducing stress in people but only collects data on 18- to 20-year-old college students. In that case, the results may not generalize to all age groups, education and income levels, and people from other countries and cultures. 

We now come to the fourth stage in the scientific method, theorizing. A theory A statement of explanation of some aspect of the natural world that is well supported by evidence—the better the evidence, the better the theory. Theories are modified with new evidence. is a statement of explanation of some aspect of the natural world that is well-supported by evidence—the better the evidence, the better the theory. Often the word theory is misrepresented in public discourse as someone’s subjective idea without evidence. The theory of natural selection to explain the evolution and change of organisms has tremendous support from over a century of research in biology, geology, chemistry, genetics, physics, mathematics, and psychology. The theory that our climate is changing because of human activity has overwhelming evidence. A theory from our sleep and memory experiment might be that quality sleep helps convert short-term to long-term memory. However, a theory is not an absolute fact. Theories are dynamic and adaptable as new evidence is acquired. While theories serve as a consensus for explaining behavior or mental processes, some of the best theories create new questions.

The fifth stage of the scientific method is replication The important but often overlooked final step in the scientific method where the complete or a part of the experimental design is conducted again to support results. . This is one of the most essential and yet often underemployed parts of the scientific method. In the past, replicating one’s own or other people’s findings has not been a high priority as journals tend to want to publish novel and exciting research instead of doing what others have already done. There are other disincentives for researchers to spend time, effort, and resources replicating other people’s work. There’s not a lot of prestige in saying, “I just spent six months diligently working and discovered what someone else already discovered.” Also, if one fails to replicate a finding, it might be that the second attempt was done incorrectly. 

The problem is that when researchers make a concerted effort and rerun experiments, results are often not replicated. The lack of replicability is a problem in many sciences. For example, in 2012, researchers at the biotechnology firm Amgen attempted to replicate the research described in highly referenced journal articles about cancer treatment. The research team at Amgen could not replicate the results of 47 out of 53 “landmark” cancer studies (Begley & Ellis, 2012; Baker, 2016). Research in psychology and sociology has had a particular problem with replication to the point that it has been dubbed a “replication crisis” (for review, refer to Yong, 2012, 2018). Recently, a large group of researchers in psychology have been attempting to replicate the findings of one hundred published research papers in three psychology journals as part of the Reproducibility Project: Psychology (refer to the Reproducibility Project website). They found that only 39 percent of the newly run experiments replicated the original results (Baker, 2015a, 2015b). Whether the findings were replicated per se was not completely straightforward. For example, of the 61 studies that did not replicate, about twenty-four produced “moderately similar” results to the original experiments (refer to Figure 1.20).