Why Study-Design Labels Matter
A study can report a clear difference between two groups and still leave open what caused that difference. The key is not simply whether researchers compared groups or used a sophisticated analysis. It is how the groups came to receive, or experience, the conditions being compared. Misreading that feature can turn a sound description into an unsupported causal claim.
This tutorial focuses on three recurring errors: treating statistical adjustment as if it were random assignment, calling every group comparison an experiment, and interpreting an observational association as proof of cause and effect. These errors are related, but they are not identical. A careful reading separates what researchers did from what they later calculated, and separates a study’s findings from the conclusions those methods justify.
The experimental units are the people, animals, objects, or other units to which treatments are assigned. In some experiments, assignment is to groups or clusters rather than individuals. The design label depends on the actual assignment process, not on whether a paper uses words such as “treatment group,” “control,” “intervention,” or “trial.”
Comparison Is Not the Same as Experiment
Researchers can compare groups in an observational study. For example, they might compare health outcomes among people who chose different diets, or compare schools with different policies. The existence of two groups and an outcome difference does not show that researchers assigned the diet or policy. To classify the design, ask what happened before outcomes were measured: did the researchers assign the condition, or did they observe a condition that already existed or was chosen by someone else?
A study may also include an intervention without using random assignment. Researchers might assign a program according to clinic, schedule, or participant choice. Such a design is not a randomized experiment; the assignment rule matters. Natural experiments and quasi-experiments can have features that support causal reasoning, but their labels do not mean that every comparison in them is automatically causal. The assignment mechanism and the assumptions supporting the inference still need to be examined.
Worked Example: A Comparison That Is Not an Experiment
Suppose a hypothetical research team compares average sleep duration among 120 students who independently chose to join an evening study group and 140 students who did not join. The researchers measured sleep at the end of a month. They report that the groups differ by 0.6 hours per night. What kind of study is this, and what conclusion is supported?
Step 1: Find the condition being compared. The condition is joining the evening study group. The outcome is sleep duration, measured in hours per night.
Step 2: Identify who assigned the condition. The students chose whether to join. The researchers observed that choice; they did not assign students to join or not join. This is an observational comparison, not an experiment.
Step 3: State the result without adding a cause. The data show a 0.6-hour difference in average sleep between the observed groups. Since the difference is defined as one group’s average minus the other’s, the report should also specify which group slept longer. The difference alone does not establish that joining the study group changed sleep.
Students who chose to join might differ in workload, motivation, schedules, or other characteristics. Those differences could be related to sleep as well as to joining. A suitable conclusion is: “In this hypothetical sample, students who chose to join the study group had a different average sleep duration from students who did not.” The design does not by itself justify saying that the study group caused the difference.
Adjustment Does Not Turn Observation into Random Assignment
In observational analyses, researchers may adjust for recorded characteristics such as age, baseline health, or income. Adjustment can compare groups after accounting statistically for measured variables, according to a chosen model. It may help address confounding by variables that were measured and appropriately included. But it does not change how people entered the groups, and it does not guarantee that the groups are comparable in every relevant way.
An adjusted estimate can remain affected by variables that were not measured, were measured poorly, or were handled inadequately. Statistical adjustment is also not a general repair for problems such as unclear timing or selection into the study. Therefore, “adjusted for several factors” is not a synonym for “randomly assigned,” and an adjusted association is not automatically a causal effect.
A short algebraic decomposition makes the issue precise. Let \(T=1\) denote units receiving an exposure and \(T=0\) those not receiving it. Let \(Y(1)\) and \(Y(0)\) represent the outcome each unit would have under exposure and no exposure, respectively. Only one of these outcomes is observed for each unit. The observed difference in mean outcomes can be written as a difference in causal outcomes for the exposed group plus a selection difference:
To see this, the observed outcome is \(Y(1)\) when \(T=1\) and \(Y(0)\) when \(T=0\). Add and subtract \(E[Y(0)\mid T=1]\) from the left side, then group the terms. The first term on the right is the average effect among exposed units; the second is the difference those groups would have had even without exposure. In an observational study, that selection difference need not be zero. Random assignment is designed to make treatment groups comparable on average through chance; adjustment can address selected measured differences but cannot guarantee that every source of selection difference has been removed.
Worked Example: What Adjustment Can and Cannot Establish
In a hypothetical observational study, adults choose whether to attend a weekly exercise class. After one year, the researchers compare average resting heart rate between attendees and nonattendees. They adjust for recorded age and smoking status, and the adjusted comparison favors attendees. A report says, “After adjustment, the class caused a lower resting heart rate.” Evaluate that statement.
What the researchers did. Adults chose whether to attend; the researchers did not assign attendance. The study is observational. The analysis adjusts for two measured variables, age and smoking status.
What the adjustment supports. If the model and measurements are suitable, the adjusted result describes a difference after accounting statistically for the recorded age and smoking variables. The phrase “after adjustment” tells us how the analysis was conducted; it does not say that the groups were randomized.
What remains uncertain. Attendance could also be related to other characteristics, such as baseline fitness, health conditions, or time available. If such factors affect heart rate and differ between attendees and nonattendees, they may help explain the observed association. Adjustment for age and smoking alone does not show that these other factors are balanced.
Corrected conclusion. The study found an adjusted association between class attendance and resting heart rate, accounting for recorded age and smoking status. The design and adjustment described do not, on their own, establish that the class caused the difference.
Random Sampling and Random Assignment Answer Different Questions
A common interpretation error is to treat “random” as one general guarantee. Random sampling and random assignment are different procedures. Random sampling selects units from a population and can support generalization to that population when the sampling design and response process justify it. Random assignment allocates study units to conditions and can support a causal comparison by making groups comparable on average. One can occur without the other.
| Design feature | What is randomized? | Main question it helps address |
|---|---|---|
| Random sampling | Which units enter the sample | How well can the sample represent a target population? |
| Random assignment | Which condition sampled or recruited units receive | Can outcome differences be attributed to assigned conditions? |
| Neither | No random selection or assignment is described | What do the observed units show, and what limitations affect broader or causal conclusions? |
Worked Example: A Random Sample Is Not a Randomized Experiment
A hypothetical city health office randomly selects 400 adults from its resident register and records whether each person uses a cycling commute and their weekly minutes of moderate activity. People decide their own commuting habits. The office finds that cycling commuters report 45 more activity minutes per week on average. What does randomization apply to here?
Sampling step. The office randomly selected residents for the study. This is random sampling. Subject to coverage and nonresponse limitations, it can support describing or generalizing the association to the population represented by the register.
Condition step. No one was randomly assigned to cycle. Participants’ commuting habits were observed, so the study is not an experiment. Random sampling does not make the commuting groups comparable in the way random assignment is intended to do.
Conclusion. In the sampled adults, people who cycled to work reported 45 more weekly activity minutes on average than those who did not. The random sample may help with population representation, but it does not establish that cycling caused the difference. People who cycle could differ in other ways related to activity.
Association Is Not Proof of Causation
An association means that the distribution or average of an outcome differs across values or groups of an explanatory variable. It is a description of a relationship in the observed data. A causal claim goes further: it says that changing the explanatory condition would change the outcome, compared with what would have happened otherwise. That stronger statement requires a design and assumptions that support the comparison.
For an observational association, consider at least three possible explanations before making a causal claim:
- Confounding: another variable is related to both the exposure and the outcome, helping produce their association.
- Reverse direction: the outcome, or an earlier version of it, may influence the exposure rather than the exposure causing the outcome.
- Selection or measurement: who is observed, how groups are formed, or how variables are measured may distort the comparison.
These are possibilities to investigate, not automatic proof that a reported association is false. An observational study can provide useful evidence, and some carefully designed observational analyses can support causal arguments under explicit assumptions. The error is claiming that an association alone proves causation or overlooking what the design leaves unresolved.
Worked Example: Correcting a Causal Overstatement
A hypothetical records-based study finds that patients who regularly attend a nutrition program have lower average blood pressure than patients who do not attend. The study did not assign attendance. A summary states, “The nutrition program reduced blood pressure.” Rewrite the conclusion to match the design and explain why.
Design classification. Attendance was not assigned by the researchers. Patients either attended or did not, so the comparison is observational.
What is directly supported. The records show an association: patients who attended had lower average blood pressure than those who did not, among the patients represented in the analysis. This description should not imply that the groups were otherwise identical.
Why the causal wording goes too far. Patients who attend may differ from nonattendees in baseline health, access to care, motivation, or other factors. The direction of influence may also be complicated: a patient’s health could affect the decision or ability to attend. Unless the study design and assumptions address such explanations, the observed difference cannot be attributed solely to the program.
Careful rewrite. “In this hypothetical records-based study, program attendance was associated with lower average blood pressure. Because attendance was not assigned, the observed association alone does not show that the program caused the lower blood pressure.”
A Practical Reading Routine
When reading a study description, separate three questions rather than jumping directly from a result to a claim. The following routine helps prevent the common errors.
Determine what each row or study unit represents, what conditions or exposures are being compared, and what outcome was measured.
Ask whether researchers assigned a condition, who made the assignment, and whether a random process was used. If participants or circumstances determined exposure, describe the study as observational unless the actual design warrants a more specific label.
Record whether the study used random sampling, random assignment, adjustment, or none of these. Do not treat one procedure as a substitute for another.
State the observed association or experimental comparison first. Make a causal statement only when the design and its assumptions justify it, and keep population generalization separate from causation.
Common Mistakes
- “The study compared two groups, so it was an experiment.” A comparison can be observational. Check whether the researchers assigned the condition.
- “They adjusted for many variables, so it was basically randomized.” Adjustment uses measured data; it does not randomize the original assignment or guarantee control of unmeasured differences.
- “The sample was random, so the exposure caused the outcome.” Random sampling concerns selection into a sample. It does not assign exposures.
- “The association proves the treatment worked.” An observed association can have alternative explanations. Describe the association unless causal interpretation is supported by the study design and assumptions.
- “An observational study can never say anything about causation.” This is too absolute. Observational evidence may contribute to a causal argument, but the assumptions and limitations must be stated; an association alone is not proof.
Check Your Understanding
For each scenario, identify what was assigned or sampled and state the strongest conclusion justified by the description.
- A hospital compares patients who chose a new app with patients who did not, then adjusts for age and diagnosis. Is this a randomized experiment? What does adjustment establish?
- A random sample of households is surveyed about home energy use and monthly bills. Does the random sample show that energy use caused any observed bill difference? Explain.
- Researchers assign volunteers to one of two study schedules, but participants choose which schedule they prefer. Is this an observational study or an experiment? Is the assignment random?
- Why can an unmeasured variable still matter after a model adjusts for several measured variables?
- Rewrite this statement cautiously: “People who attended the wellness course had lower average stress, so the course reduced stress.” The course was not assigned by researchers.