Why the Study Design Comes First
Two studies can record the same outcome and compare the same two labels—such as “treated” and “untreated”—yet support different conclusions. The key is not just which calculation is available. It is how people or other study units entered the groups, how measurements were timed, and what population the units represent. These features determine whether a comparison is causal, whether it can be generalized, and whether observations should be analyzed as independent groups or as pairs.
A useful distinction is between random assignment and random sampling. Random assignment uses chance to allocate study units to conditions. It can support a causal comparison of those conditions. Random sampling uses chance to select units from a population. It can support generalizing results to that population. One does not automatically provide the benefit of the other.
What Random Assignment and Random Sampling Support
In a randomized experiment, chance determines which condition each enrolled unit receives. If the groups are otherwise treated comparably, a difference in outcomes can be attributed to the assigned conditions for the units in the experiment, subject to the experiment’s conduct and assumptions. For a quantitative outcome, the comparison may be expressed as \(\mu_1-\mu_2\), the difference between the condition-specific population means being targeted. For a binary outcome, it may be \(p_1-p_2\), the difference between the corresponding population proportions.
Random assignment does not, by itself, make the enrolled units representative of a broader population. If volunteers at one clinic are randomly assigned to two treatments, assignment helps make the treatment groups comparable, but it does not turn the volunteers into a random sample of all patients. Generalization beyond the enrolled units needs a basis such as random sampling or a justified argument that the study population is relevant to the target population.
Conversely, a random sample can support inference about the population from which it was drawn, but it does not make an observed group difference causal. In an observational study, people’s existing characteristics or choices may be related both to group membership and to the outcome. A difference between groups may therefore reflect those other factors as well as the group characteristic being compared.
| Design feature | What it can support | What it does not establish by itself |
|---|---|---|
| Random assignment to conditions | A causal comparison of assigned conditions, for the study units, when the experiment is properly conducted | Generalization to a broader population |
| Random sampling from a population | Generalization to the population sampled from | A causal explanation for a group difference |
| Both random sampling and random assignment | Potentially both population generalization and a causal comparison | That all other design and analysis conditions are satisfied |
| Neither | A description of the observed units and their outcomes | Automatic causal conclusions or population generalization |
Measurement Timing Determines the Comparison
After considering selection and assignment, ask when the outcome was measured and whether the same units contributed to both measurements. If each unit contributes one outcome to one of two separate groups, the comparison is between two groups. If the same unit is measured under both conditions, or if units are deliberately matched into pairs and both members’ outcomes are linked, the analysis should preserve those pairings. The basic comparison is then the within-pair difference, not a comparison that treats all observations as unrelated.
A baseline measurement does not automatically make a study a paired comparison. For example, a trial may measure an outcome before treatment but define its primary outcome as the value recorded after treatment. If each participant’s post-treatment value is compared between randomized arms, the main comparison is still between two independent groups. The existence of baseline data may matter to other analyses, but it does not change what the stated outcome comparison is.
By contrast, in a before-and-after study where the outcome of interest is each person’s change, the two measurements belong together. Let \(D_i\) denote the later measurement minus the earlier measurement for unit \(i\). The target can then be the population mean change, \(\mu_D\), rather than a difference between two groups of unrelated individuals.
Worked Example: Random Assignment to Two Treatments
In a hypothetical experiment, 10 volunteers are randomly assigned to a new treatment or a standard treatment, five per group. The measured outcome is hours of symptom relief after one dose. The new-treatment outcomes are 6, 8, 7, 9, and 10 hours; the standard-treatment outcomes are 4, 5, 6, 7, and 8 hours. Identify the supported comparison and interpret the observed difference.
Each volunteer receives one assigned treatment and contributes one outcome. The outcome units are people in two separate randomized groups, so the comparison is between two groups, not a set of within-person changes. For the sample, the mean outcomes are
Taking new treatment minus standard treatment, the observed difference is
Thus, in this hypothetical experiment, the new-treatment group’s mean symptom-relief time was 2.0 hours greater. Random assignment supports a causal comparison of the assigned treatments for these volunteers, assuming the assignment was implemented as intended and the experiment otherwise permits that interpretation. Because the volunteers were not described as a random sample, the design alone does not justify generalizing the difference to all people with the condition. This descriptive calculation does not establish statistical significance.
Worked Example: A Random Sample With Existing Groups
A hypothetical survey takes a random sample of 100 adults from a city. Among 50 adults who work remotely, 30 report commuting by public transit on a typical workday. Among 50 adults who work on-site, 20 report doing so. Work arrangement was not assigned by researchers. What population comparison is supported, and what conclusion is not supported?
The outcome is binary: whether an adult reports using public transit. The observed sample proportions are
The sample difference, remote minus on-site, is
The observed difference is 0.20, or 20 percentage points. Because the adults were randomly sampled from the city, the study can support inference about the corresponding city population proportions, subject to the sampling and analysis conditions. But work arrangement was not randomly assigned. The result does not show that remote work causes a greater probability of commuting by public transit; for example, location or occupation might be related to both work arrangement and commuting.
Worked Example: Measuring Change in the Same People
In a hypothetical program, five participants record their weekly exercise time before and after eight weeks. Their times, in hours, are: participant A, 2 before and 3 after; B, 4 and 5; C, 3 and 3; D, 1 and 2; E, 5 and 6. The outcome of interest is each participant’s change. Which comparison matches the design?
The same five people provide both measurements, so the before and after values are paired. Define \(D_i=\text{after}_i-\text{before}_i\). The five changes are \(1,1,0,1,1\) hours. Their mean is
The relevant sample summary is therefore an average within-person increase of 0.8 hours per week, and the corresponding population target for a one-group paired analysis is the mean change, \(\mu_D\). It would discard the design information to treat the five “before” values and five “after” values as though they came from two unrelated sets of people. Also, because no random assignment or random sample is described, the observed average change alone does not establish that the program caused the increase or that it represents a broader population.
Why Pairing Changes the Information in the Data
Pairing matters because measurements on the same person may be related. The variance of a difference between two measurements on the same unit includes their covariance. For random variables \(Y_1\) and \(Y_2\), the identity is
This follows by writing \(\operatorname{Var}(Y_1-Y_2)=E[((Y_1-EY_1)-(Y_2-EY_2))^2]\), expanding the square, and recognizing the two squared-deviation expectations as variances and the cross-product expectation as covariance. If the two measurements are positively associated, the covariance is positive, and the variance of their difference is smaller than it would be if that association were ignored. This is one reason matched or repeated measurements can be informative.
For genuinely independent groups, the covariance between outcomes from different units is zero under the independence model, so the paired-measurement formula is not the structure to use. The design—not a preference for whichever calculation gives a smaller standard error—determines whether outcomes are paired. Any selected procedure still needs its own conditions checked; identifying the design is the first decision, not a substitute for checking those conditions.
Worked Example: Baseline Data Do Not Automatically Make a Paired Outcome
A hypothetical randomized study measures participants’ resting heart rates at baseline, assigns each participant to either a new program or usual care, and measures resting heart rate again after six weeks. Suppose the stated outcome is the six-week heart rate, and the sample means are 72 beats per minute for 12 participants assigned to the program and 76 beats per minute for 12 assigned to usual care. Determine the comparison from that stated outcome.
The study has baseline measurements, but its stated outcome is the post-assignment value. Each participant contributes one such outcome to one randomized group. Therefore, the stated comparison is between the two groups’ post-study means:
The program group’s observed mean is 4 beats per minute lower. Random assignment supports a causal comparison of assigned conditions for the study participants, subject to appropriate conduct, but the result alone does not establish generalizability beyond them. If the research question instead defined each participant’s outcome as change from baseline, the analysis would use each participant’s paired change. The measurement timing and the specified outcome determine which comparison is being made.
A Practical Design-Based Workflow
Decide whether the question concerns a mean, a proportion, a difference between groups, or a change within units. State the direction of any difference.
Identify whether units were randomly sampled, randomly assigned, both, or neither. Use sampling to assess generalization and assignment to assess causal interpretation.
Determine whether each unit contributes an outcome to one group, repeated outcomes, or one member of a matched pair. Preserve the dependence created by repeated or matched measurements.
Use a one-group comparison when the target is a single group parameter or the mean of within-unit differences; use a two-group comparison for outcomes from distinct groups. Then check the assumptions and conditions required by the particular procedure.
Common Mistakes
- Confusing random assignment with random sampling. Random allocation can support a causal comparison; it does not make participants representative of a population.
- Claiming causation from a random sample. A representative sample can support population inference, but observational group membership can be associated with other explanatory factors.
- Calling before-and-after outcomes independent groups. When the question is about change in the same people, calculate or analyze within-person differences.
- Assuming any study with baseline measurements is paired. If the stated outcome is measured once after assignment and compared between arms, the comparison is between groups. Baseline data do not change that outcome’s structure by themselves.
- Generalizing farther than the selection design allows. A result from a convenience sample of volunteers does not automatically describe everyone in a wider population.
Check Your Understanding
For each question, identify the comparison structure and state what the design does or does not support.
- A random sample of households reports whether they have solar panels and their monthly electricity use. Does random sampling establish that solar panels cause any observed difference in use?
- Patients are randomly assigned to one of two medications, and each patient’s blood pressure is measured once after four weeks. Is the stated outcome a two-group comparison or a paired comparison?
- A teacher records each student’s test score before and after a tutoring program. If the outcome is score change, what quantity should be formed for each student?
- Researchers randomly assign volunteers from one neighborhood to two diet plans. What does random assignment support, and what does it not guarantee about people outside the study?
- A study measures participants at baseline and six months later, but its stated outcome is the six-month score compared across two randomized groups. Does the presence of baseline data alone make this a paired comparison?