Two Different Questions About a Study
When a study uses the word “random,” it is important to ask what was randomized. A researcher might randomly select people from a population, randomly allocate enrolled participants to treatments, do both, or do neither. These choices address different problems. Random selection concerns who enters the study; random allocation concerns which treatment each participant receives.
The distinction matters because statistical conclusions have two separate destinations. A researcher may want to compare treatments to assess whether one caused a difference in outcomes. The researcher may also want to describe or make conclusions about a larger population. A design that helps with one goal does not automatically achieve the other.
In both cases, the use of chance should be part of the actual procedure, not just a description of the researchers’ intentions. For example, choosing every participant who happens to be nearby is not random sampling. Allowing participants to choose their preferred treatment is not random assignment. A genuinely random process gives the relevant individuals or allocations known chances determined by the design.
What Each Process Helps Establish
A probability sample, such as a simple random sample from a defined population, gives each population member a known chance of selection. This can make the sample informative about that population. It does not ensure that every sample will match the population closely: chance samples vary, and nonresponse or other departures from the sampling plan can introduce additional problems.
Random assignment makes treatment allocation independent of participants’ pre-existing characteristics in the design, rather than allowing those characteristics to determine treatment. This makes systematic differences between treatment groups less likely to be an alternative explanation for an observed outcome difference. Assignment by chance does not make the groups identical in a particular experiment; it balances characteristics as a matter of the randomization process over repeated assignments.
These roles can be summarized as follows:
| Process | What is randomized? | Main purpose | Conclusion it can support |
|---|---|---|---|
| Random sampling | Which members of a population enter the study | Help the observed sample represent a defined population | Generalization from the sample to that population, with appropriate qualifications |
| Random assignment | Which treatment each study unit receives | Make treatment groups comparable by design | A causal comparison of the assigned treatments, subject to the study’s assumptions and conduct |
“Generalization” and “causation” are not interchangeable. A randomized experiment with volunteers may provide evidence about the effect of a treatment for those studied, yet give limited grounds for claiming that the same effect applies to everyone in a wider population. Conversely, a probability sample in which people simply report their existing behaviours can help describe that population, but it does not by itself show that one behaviour caused another.
Worked Examples
Worked Example: A Random Sample Without Random Assignment
A town has 1,200 adult residents. A researcher selects 120 residents by simple random sampling and asks whether each supports a proposed public transit plan. Of those who respond, 72 say yes. The survey does not assign anyone to a treatment. What can the design support?
Step 1: Identify the random process. The researcher randomly selected residents from the town. The 120 people are a sample from the population of 1,200 adults. No treatment was allocated, so there was no random assignment.
Step 2: Summarize the responses. The observed proportion of respondents who support the plan is
So 60.0% of the responding sampled residents supported the plan. If the sampling procedure was carried out as described and nonresponse does not seriously distort the results, the random sample can help estimate support among the town’s adults. The result is not automatically exact for all 1,200 residents; random samples can differ from their populations, and the 48 sampled residents who did not respond may matter.
Step 3: Set the limit on the conclusion. Since no one was assigned to a condition, the survey cannot establish that the transit plan caused people to support or oppose it. Random sampling helps with population representation, not causal identification.
Worked Example: Random Assignment Among Volunteers
Suppose 40 volunteers agree to take part in a hypothetical experiment comparing two study schedules. The researcher randomly assigns 20 volunteers to each schedule. At the end of one week, the first group averages 6.4 hours of sleep per night and the second averages 7.1 hours. What does random assignment contribute, and what does it not establish?
Step 1: Identify the random process. The participants volunteered; the researcher did not randomly select them from all students or from any other defined population. However, the 40 participants were randomly assigned to the two schedules.
Step 2: Calculate the observed difference. Taking the first schedule’s mean minus the second schedule’s mean gives
In this hypothetical experiment, the first group slept an average of 0.7 hours less per night than the second group. Because treatment was assigned randomly, a causal comparison of the schedules among these participants is more defensible than it would be if participants had chosen their schedules. That conclusion still depends on the experiment being carried out appropriately, including measuring outcomes comparably and avoiding problems such as treatment groups losing participants in different ways.
Step 3: Limit generalization. The volunteers might differ from students who did not volunteer. Since recruitment was not a random sample of a defined student population, random assignment alone does not establish that the 0.7-hour difference would apply to all students. It addresses the treatment comparison, not who the participants represent.
Worked Example: Random Sampling and Random Assignment Together
In a hypothetical study, researchers use a simple random sample to select 200 adults from a defined population of 5,000 adults. They then randomly assign 100 sampled adults to receive a health-information program and 100 to receive usual information. After four weeks, 18 of the 100 people assigned to the program and 30 of the 100 assigned to usual information meet a specified outcome. Identify what each random process contributes and calculate the observed difference in outcome proportions.
Step 1: Separate the two random procedures. Random sampling selected the 200 adults from the population of 5,000. Random assignment allocated those sampled adults to the program or usual-information condition. These are distinct decisions made at different stages.
Step 2: Calculate the group proportions. The proportions meeting the outcome are
The observed difference, program minus usual information, is
Thus, in this hypothetical sample, the outcome proportion was 12.0 percentage points lower in the program group. Random assignment supports interpreting the group difference as evidence about the program’s causal effect, provided the study’s conduct and assumptions support that interpretation. Random sampling also gives a basis for relating results to the defined population of 5,000 adults, subject to sampling variability and any nonresponse or implementation issues.
Step 3: Keep the claims distinct. The causal comparison comes from random assignment; the potential to generalize to the defined population comes from random sampling. Neither procedure guarantees that the observed difference is exactly the effect in every person or that it applies to populations other than the one sampled.
Why Random Assignment Supports a Causal Comparison
A short justification shows the role of assignment more precisely. Imagine a fixed set of \(N\) study units. For each unit \(i\), let \(Y_i(1)\) be the outcome that unit would have under treatment 1, and let \(Y_i(0)\) be its outcome under treatment 0. Only one of these outcomes can be observed for each unit in the experiment. Suppose \(n_1\) units are assigned to treatment 1 by complete random assignment, so each unit has probability \(n_1/N\) of entering that group.
The average outcome in the treatment 1 group is the average of the \(Y_i(1)\) values for the units assigned there. Each unit is equally likely to be included, so the expected treatment-group average, over the random assignments, is the average of all \(N\) units’ \(Y_i(1)\) values:
The same reasoning applies to treatment 0. Subtracting the two expected group averages gives
The expression on the right is the average difference between the two potential outcomes across the study units. Thus, over repeated random assignments, the group-mean difference targets the average causal effect for those units. In any one assignment, chance variation can make the groups differ, and the experiment does not reveal both potential outcomes for any individual. The derivation explains why random assignment supports a causal comparison; it does not imply that every observed difference is entirely caused by treatment.
The analogous point for simple random sampling is about representation. Each population member has the same selection chance, so the expected sample mean of a quantitative variable equals the population mean. In a particular sample, however, the sample mean may be above or below the population mean. Random selection supports estimation; it is not a promise that every sample exactly reproduces the population.
Four Possible Design Combinations
The two processes can occur in any combination. Thinking of them as separate questions prevents the mistaken assumption that “randomized” automatically means representative, or that a representative sample automatically provides causal evidence.
| Sampling | Assignment | Typical strength and limitation |
|---|---|---|
| Random | Random | Can support a causal comparison and, with care, generalization to the sampled population |
| Random | Not random | Can support population description or association, but not a causal claim from assignment |
| Not random | Random | Can support a causal comparison for study participants, but population generalization may be limited |
| Not random | Not random | Neither random selection nor random allocation supplies its usual protection; conclusions require particular caution |
These are not guarantees of good evidence. A random sample can suffer from substantial nonresponse, and an experiment can have poor outcome measurement or unequal loss to follow-up. The design tells us what the random mechanisms can support; the quality of implementation and the study’s assumptions still matter.
Common Mistake: Treating One Kind of Randomness as Both
A common mistake is to say, “The experiment was randomized, so its participants represent the whole population.” That reasoning confuses random assignment with random sampling. Random assignment determines who receives each condition among the people already in the study; it does not determine who entered the study in the first place. The correction is to name each process separately: ask whether participants were randomly selected from a population, and then ask whether they were randomly assigned to treatments.
The reverse mistake is also possible: a randomly selected survey sample does not become an experiment merely because it reflects a population. If respondents report their own existing exposures or behaviours, no treatment was allocated. The data can describe patterns in the sampled population, but random sampling by itself does not make those patterns causal.
Check Your Understanding
For each scenario, identify what was randomized and state the main kind of conclusion that procedure can support.
- A researcher draws 300 names at random from a city directory and asks each selected person about weekly exercise. No treatment is assigned. What does the random process contribute?
- Patients who volunteer for a hypothetical trial are randomly allocated to one of two treatments. Does this random assignment establish that volunteers represent all patients? Explain.
- A probability sample of adults is asked whether they use a particular app and whether they sleep at least seven hours. Can the sampling process alone establish that app use causes longer sleep?
- In the derivation, why is each unit’s treatment outcome \(Y_i(1)\) equally likely to appear in the treatment-group average?
- A study randomly selects people from a target population and then randomly assigns them to treatment or control. Which process supports a causal comparison, and which supports generalization to the target population?