Why the Way a Study Is Conducted Matters
Suppose a report finds that people who exercise more often tend to have lower resting heart rates. Another report compares patients receiving a medication with patients not receiving it. Both reports compare outcomes across groups, but the groups may have been formed in very different ways. In the exercise study, people may have chosen their own activity levels. In the medication study, researchers may have assigned patients to take the medication or a comparison substance.
That distinction is about the study’s design, not just what variables were measured. To classify a study, ask: Did the researchers assign the explanatory condition, or did they observe conditions that arose without their assignment? The answer distinguishes an experiment from an observational study. A further question—whether assignment was made by chance—distinguishes a randomized experiment from other experiments.
A study unit is the person or other unit about which measurements are collected. The explanatory variable describes a condition that may help explain differences in a response variable. In an experiment, researchers assign the condition represented by at least one explanatory variable; in an observational study, they do not. For example, if researchers assign a study technique and measure quiz scores, technique is the assigned explanatory condition and quiz score is the response.
A Reliable Classification Procedure
Read the study description for what researchers did, rather than relying on words such as “study,” “compare,” or “group.” The fact that a report compares two groups does not tell you how those groups came to be. Look for evidence that investigators assigned a condition, such as instructing one group to follow an exercise plan or allocating patients to medication conditions. If the researchers instead record participants’ existing habits or care, the design is observational.
What condition, exposure, or behavior is being compared—for example, medication received, exercise habits, or study technique?
If researchers assigned it, the study is an experiment. If participants already had it or it arose without researcher assignment, the study is observational.
If chance determined which units received which assigned condition, the experiment is randomized. Assignment by choice or another nonrandom rule makes it a nonrandomized experiment.
This order prevents two common classification errors. First, a study is not an experiment merely because researchers compare groups. Second, an experiment is not necessarily randomized: researchers can assign conditions according to a fixed rule or practical arrangement. “Randomized” describes the method of assignment, not whether the participants or study units were selected randomly from a larger population.
Worked Examples
Worked Example: Existing Exercise Habits
In a hypothetical study, researchers recruit 120 adults and ask each person to report how many minutes they exercised during the previous week. They measure resting heart rate and compare the average heart rates of people reporting less than 60 minutes of exercise with those reporting at least 150 minutes. Researchers do not tell anyone how much to exercise.
Explanatory condition: the amount of exercise reported. Response: resting heart rate, measured in beats per minute. The two groups arise from participants’ existing exercise habits, not from an exercise amount assigned by researchers. Therefore, this is an observational study, not an experiment.
The numerical comparison between groups does not change that classification. Even if the average heart rates differ, the design describes an association between reported exercise and heart rate. The study design alone does not establish that assigning a particular amount of exercise would produce the observed difference.
Worked Example: Randomly Assigned Medication
In a hypothetical trial, 90 study units are assigned by a chance-based procedure: 45 receive a specified medication dose and 45 receive a different assigned condition. After four weeks, researchers record each unit’s symptom score on a scale from 0 to 10. The study protocol specifies the assignments before symptom scores are collected.
Researchers determine which medication condition each study unit receives, so this is an experiment. Because a chance-based procedure determines the assignments, it is specifically a randomized experiment. The symptom score is the response variable, measured in points on the stated scale.
The classification follows from the assignment procedure, not from whether the observed average scores differ. If the researchers had only recorded which medication patients received as part of their usual care, without assigning it, the study would instead be observational—even if it included the same number of patients and measured the same symptom score.
Worked Example: Study Techniques Assigned by Class
In a hypothetical school study, a teacher assigns one existing class of 24 students to use flashcards and another existing class of 26 students to use rereading for a week. The teacher then gives both classes the same quiz and records scores out of 20. The teacher chose the techniques for the classes; no chance-based assignment was used.
Because the teacher assigns a study technique and measures quiz scores afterward, this is an experiment. It is not a randomized experiment, because the conditions were assigned by class choice rather than by chance. Quiz score, in points out of 20, is the response.
The fact that students selected into the classes earlier, or that the classes already existed, does not by itself make this observational. The classification question is whether the study’s explanatory condition—the assigned study technique—was assigned by the researchers. Here it was, but not randomly.
Why Random Selection Is Not Random Assignment
Random selection and random assignment refer to different decisions. Random selection concerns which units enter a study from a broader group. Random assignment concerns which study condition each included unit receives. A study can use either, both, or neither. In particular, randomly selecting participants does not turn an observational study into an experiment if researchers merely record their existing conditions.
The distinction can be justified by tracking the two separate decisions. Suppose researchers use a chance procedure to select 100 people from a city, then ask whether they already exercise regularly and measure their resting heart rates. Chance affected who was observed, but it did not determine who exercised. The exercise groups still reflect existing behavior, so the study remains observational. By contrast, if researchers assign the selected people to exercise conditions using chance, the design is a randomized experiment.
| What chance determines | What researchers do with the explanatory condition | Classification |
|---|---|---|
| Which units are included | Record conditions that already exist | Observational study with random selection |
| Which condition each included unit receives | Assign conditions using chance | Randomized experiment |
| Neither selection nor assignment | Assign a condition by a nonrandom rule | Nonrandomized experiment |
| Neither selection nor assignment | Record naturally occurring conditions | Observational study |
What the Classification Does and Does Not Tell You
The classification tells you how the explanatory conditions arose. It does not, by itself, tell you whether the study units represent a larger population, whether measurements are accurate, or whether the results are precise. Those are separate questions about the study’s sampling and measurement. Likewise, calling a design a randomized experiment does not guarantee that every practical feature of the study was handled well.
The distinction also guides how cautiously to interpret a comparison. In an observational study, people with different exposures may differ in other ways, so a response difference cannot automatically be attributed to the exposure. Assignment in an experiment makes the design more directly suited to studying the effect of an assigned condition, but the strength and scope of any conclusion still depend on how the experiment was conducted. A design label is not a substitute for examining the evidence.
Common Mistakes
- “The researchers compared groups, so it was an experiment.” Comparison alone is not enough. If the groups reflect existing behaviors or circumstances that researchers only recorded, the study is observational.
- “The participants were chosen randomly, so the study was a randomized experiment.” Random selection chooses study units; random assignment allocates conditions. Only the latter makes an experiment randomized.
- “The study involved a medication, so it must be an experiment.” Researchers may observe medication use in usual care without assigning it. The presence of a medication does not establish the design.
- “Every experiment is randomized.” Researchers can assign conditions using a nonrandom rule. Such a study is an experiment, but not a randomized experiment.
- “An observational association proves that changing the exposure will change the response.” An observed relationship does not, by itself, show that assigning the exposure would produce the same difference.
Check Your Understanding
For each situation, identify the design and briefly explain which detail determines the classification.
- Researchers ask 300 adults how often they exercise and measure their resting heart rates. They do not assign exercise amounts. Is this observational or experimental?
- A clinic assigns patients to one of two medication conditions using a chance-based procedure and records symptom scores after three weeks. What is the study design?
- A teacher assigns one existing class to use flashcards and another to use rereading, then compares quiz scores. No chance-based assignment is used. How should the study be classified?
- Researchers randomly select students from a school and record which study technique each student already uses. Does random selection make this a randomized experiment? Why or why not?
- Two reports compare exercise groups. In one, participants choose their exercise habits; in the other, researchers assign exercise conditions by chance. Explain why the designs differ.