Why the Study Design Matters
Suppose people who use a particular study app earn higher exam scores than people who do not. The difference is an association: app use and scores vary together in the observed data. But did the app improve scores, or were app users already more prepared, more motivated, or different in some other way? The numbers alone do not settle that question. To decide whether a causal conclusion is supported, we must examine how people came to receive the treatment being studied.
A study may describe a pattern accurately without establishing what produced it. By contrast, when researchers use random assignment to allocate study units to treatments, the resulting comparison can support a causal conclusion, provided the experiment is carried out in a way that preserves a meaningful comparison. Even then, the conclusion must be limited to the treatment, outcome, and study units actually examined.
What Random Assignment Adds
In an observational study, researchers record which treatment or exposure each study unit has, but do not assign it. The observed groups may differ in ways related to the outcome, so their outcome difference cannot automatically be attributed to the treatment. Confounding is one reason that an association may not represent a causal effect.
In a randomized experiment, researchers assign study units to treatment groups using chance. Random assignment does not guarantee that the groups will be identical: a chance allocation can produce differences in age, prior experience, or other characteristics. Its value is that treatment allocation is not systematically chosen according to those characteristics. Across the possible random allocations, the groups are comparable on average, including for characteristics researchers did not measure.
A causal conclusion depends on what was assigned and how the study was conducted. The assigned treatment should be clear, outcomes should be measured comparably across groups, and the comparison should not be undermined by serious losses of study units or other departures from the planned design. The conclusion is about the treatment assignment as implemented; it does not automatically show that every person received or followed the treatment exactly as intended.
A Brief Justification for Random Assignment
The role of chance can be stated precisely. Imagine \(N\) study units, with \(n_1\) assigned to treatment and \(n_0=N-n_1\) assigned to comparison. For each unit \(i\), let \(Y_i(1)\) be the outcome that unit would have under treatment and \(Y_i(0)\) the outcome that unit would have under comparison. Only one of these outcomes is observed for each unit. The average causal effect for these study units is the average of their individual treatment effects.
In a random assignment with fixed group sizes, each unit has probability \(n_1/N\) of being assigned to treatment. Therefore, the expected average of the treated units’ \(Y_i(1)\) values is the average \(Y_i(1)\) across all \(N\) units. The same reasoning applies to the comparison group and the \(Y_i(0)\) values. Subtracting these expected averages gives the average of the individual treatment effects:
Here, \(\bar{Y}_{\text{treatment}}\) and \(\bar{Y}_{\text{comparison}}\) are the observed group means. This result explains why the difference in group means from a randomized experiment is an unbiased estimate, over repeated random assignments, of the average causal effect for the study units. It does not say that a particular allocation must produce perfectly balanced groups or that one observed difference is certainly the exact causal effect. Chance variation remains.
Worked Examples
Worked Example: A Randomized App Trial
In a hypothetical experiment, 40 students are randomly assigned to either use a study app for four weeks or continue their usual study routine. Each group has 20 students. At the end, the app group reports a mean of 150 minutes of study per week, while the comparison group reports 118 minutes. The researchers want to know whether using the app increased weekly study time.
Check the design. The researchers, rather than the students’ prior preferences, assigned the treatment by chance. If study time was measured in the same way in both groups and the planned groups were followed adequately, the randomized comparison supports a causal interpretation for assignment to the app in these study participants.
Calculate the observed difference. Define the difference as the treatment-group mean minus the comparison-group mean:
The app-assigned group averaged 32 more minutes of reported study per week than the comparison group. Because allocation was randomized, it is reasonable to interpret this difference as evidence that assignment to the app increased study time on average among these participants, subject to chance variation and the study’s execution. It is not proof that every student increased study time by 32 minutes, nor does the result establish the effect for all students everywhere.
Worked Example: An Observed Association Is Not a Causal Result
In a hypothetical survey, 80 adults report whether they drink coffee daily and whether they usually sleep at least seven hours. Of 40 daily coffee drinkers, 18 report at least seven hours of sleep. Of 40 adults who do not drink coffee daily, 28 report at least seven hours. Does this show that daily coffee drinking reduces sleep?
Calculate the observed proportions. Among daily coffee drinkers, the proportion reporting at least seven hours is \(18/40=0.45\), or 45%. Among the other adults, it is \(28/40=0.70\), or 70%. The difference, stated as the first group minus the second, is:
In this sample, the proportion reporting at least seven hours was 25 percentage points lower among daily coffee drinkers. That is an association between reported coffee use and reported sleep. The survey did not randomly assign people to drink coffee or not, so it does not establish that coffee caused the difference. Other differences between the groups, or other explanations, could contribute to the pattern. A suitably cautious conclusion describes the observed association rather than claiming a causal effect.
Worked Example: A Randomized Result and Its Scope
In a hypothetical study, 60 patients at one clinic are randomly assigned to a new reminder system or the clinic’s usual reminders, with 30 patients in each group. During the next month, 24 patients assigned to the new system attend a scheduled appointment, compared with 18 assigned to usual reminders. What can the study support?
Calculate attendance proportions. The proportion attending in the new-system group is \(24/30=0.80\), or 80%. In the usual-reminders group it is \(18/30=0.60\), or 60%. The observed difference is:
The new-system group’s attendance proportion was 20 percentage points higher. Since treatment assignment was randomized, and assuming attendance was recorded consistently and the comparison was not seriously disrupted, the difference supports a causal conclusion about assignment to the new reminder system for these patients in this study. The study does not by itself establish that the system would have the same effect in other clinics or patient populations. Random assignment concerns how treatments are allocated; it does not make the participants a random sample of everyone who might use the system.
Keep Assignment and Sampling Separate
Two uses of chance answer different questions. Random assignment concerns which study units receive which treatments. It helps make treatment groups comparable and supports causal interpretation. Random sampling concerns which units are selected from a larger population. It helps support generalization from the sample to that population, when the sampling process and other conditions justify it.
A study can use random assignment without random sampling, as in the clinic example. It can therefore offer evidence about a causal effect for the participants while providing limited grounds for claiming the same effect in a much broader population. Conversely, a representative random sample can describe a population well, but if people’s exposures were merely observed, representativeness does not by itself turn an association into a causal effect.
| Study feature | Main question it helps answer | What it does not guarantee by itself |
|---|---|---|
| Random assignment | Can the treatment-group difference support a causal interpretation? | That every group characteristic is exactly balanced, or that results generalize widely |
| Random sampling | Can results describe a larger target population? | That an observed association is causal |
| Neither feature | What pattern appears among the observed units? | A causal effect or broad population generalization without further support |
How to State a Study Conclusion
A sound conclusion connects the design to the result and avoids extending either beyond its support. First identify whether treatment was assigned or merely observed. Then state the measured difference in the study units, with its direction and units. Finally, describe the kind of claim the design supports and any important limit on its scope.
Ask whether researchers randomly assigned treatments or observed naturally occurring groups. Do not treat random sampling as if it were random assignment.
State which group had the larger or smaller outcome and give the calculated difference with context and units.
For a well-conducted randomized experiment, describe evidence about the causal effect of the assigned treatment. For an observational study, describe the association unless additional causal reasoning is supplied.
Specify the study units, treatment, outcome, and time period. Do not claim broader generalization merely because assignment was random.
Why Causal Claims Need Care
Causal claims can influence decisions about clinical care, education, and public policy. Overstating causation may lead people to believe that changing one factor will necessarily change an outcome, when the evidence only shows that two variables occurred together. Careful wording preserves what the data do show without pretending the design answers a question it was not built to answer.
A common mistake is to reason, “The treatment group had a better outcome, so the treatment caused the improvement.” The correction is to ask how the groups were formed. If assignment was randomized and the study was implemented adequately, the difference can support a causal interpretation, while allowing for chance variation and limiting the conclusion to the studied context. If the groups were observed rather than assigned, the difference establishes an association in the data, not a causal effect by itself.
The reverse overstatement is also a mistake: “This was not a randomized experiment, so no causal conclusion is ever possible.” Randomized experiments provide especially direct evidence for causal comparisons, but observational evidence may contribute to causal reasoning when combined with appropriate design and assumptions. The essential point here is narrower: an observed association alone is not enough to establish causation.
Check Your Understanding
For each question, distinguish what the study observes from what its design can support.
- A survey finds that people who walk to work report lower stress than people who drive. What can be concluded from this association, and what cannot be concluded from the survey alone?
- Researchers randomly assign volunteers to two exercise programs and compare their average running times. What role does random assignment play in interpreting the difference?
- Explain why randomly assigning 50 clinic patients to treatments does not, by itself, show that the patients represent all people who might receive those treatments.
- A randomized study finds a treatment-group mean 4 points higher than the comparison-group mean. What additional context should a careful conclusion include besides the number 4?
- Why does random assignment support causal interpretation on average without guaranteeing that the groups are exactly alike in one particular experiment?