Two Chance Processes, Two Different Conclusions
In Statistically Significant Differences and Chance Variation, you learned how to judge whether an observed treatment difference would be unusual under chance assignment alone. The next question is what that difference lets you conclude. The answer depends partly on how the experimental units entered the study and how they were placed into treatment groups.
Two chance processes have distinct roles. Random selection uses chance to choose individuals from a population for a sample. Random assignment uses chance to place experimental units into treatment groups. Random selection helps support generalizing from a sample to a population; random assignment helps support a cause-and-effect conclusion about treatments. One process does not substitute for the other.
Random assignment does not, by itself, show that a treatment had an effect. It makes a treatment comparison fairer by using chance to create groups that are comparable, on average, with respect to potential lurking variables. The observed results must still provide evidence of a difference. As discussed in the previous tutorial, statistical significance can be evidence against a chance-only explanation; it is not proof of an effect.
Likewise, random selection does not make treatment groups comparable. It helps a sample represent a defined population, but it cannot remove confounding caused by how treatments were given. Keep these two questions separate: Can the study support a causal conclusion? and Can the study’s results be generalized to a population?
A Framework for the Conclusions
Consider the presence or absence of each chance process. This framework summarizes the usual reach of a study’s conclusions. In every case, the conclusion also depends on how well the study was carried out, how the population and sample were defined, and whether the results provide evidence relevant to the question.
| Random selection? | Random assignment? | Conclusion the design can support |
|---|---|---|
| Yes | Yes | A treatment difference can support a cause-and-effect conclusion, and results can be generalized to the population represented by the selection process. |
| No | Yes | A treatment difference can support a cause-and-effect conclusion for the experimental units studied, but not automatically for a broader population. |
| Yes | No | Results can support generalizing an association to the population represented by the selection process, but they do not establish that a treatment caused the difference. |
| No | No | Conclusions are limited to describing the observed units and their results. Neither broad generalization nor a causal conclusion is supported by these chance processes. |
“Can support” is important wording. Neither chance process guarantees a perfect sample, eliminates every possible source of error, or proves a scientific claim. For generalization, consider whether the sampling frame covered the target population and whether nonresponse or later dropouts could change who is represented. For a causal conclusion, consider whether the treatments were implemented as planned and whether the response was measured consistently.
When both chance processes are used, they support different parts of the final statement. Random assignment helps justify saying that the treatment caused a difference. Random selection helps justify saying that the effect may apply to the population from which the sample was drawn. A careful conclusion names both the treatment comparison and the population it concerns.
Worked Example: A Random Sample and Random Assignment
Worked Example: A New Reminder for Community Gardeners
In a fictional experiment, researchers use a simple random sample of 120 registered community gardeners from a well-maintained roster of 900 gardeners in a county. The sampled gardeners are randomly assigned to receive either a text reminder about watering or the usual email notice. After four weeks, the percentage of plants reported as watered on schedule is higher for the text group. A randomization analysis finds the difference statistically significant.
State: The experimental units are the 120 sampled gardeners. The treatments are the text reminder and the usual email notice, and the response is whether plants were watered on schedule during the study period. The question is whether the reminders affected on-schedule watering and whether the finding can apply beyond the study participants.
Plan: Check the two chance processes separately. The simple random sample was selected from the county roster, so random selection supports generalizing to the county’s registered gardeners represented by that roster. Random assignment to the reminder groups supports a fair treatment comparison. The conclusion also assumes the roster is suitable for that target population, participation and follow-up do not substantially distort the sample, and the reminders and response measurement were carried out consistently.
Do: The reported difference favors the text reminder, and the randomization analysis says a difference at least this extreme would be unusual under the no-effect model and the assignment process. Thus, the results provide evidence that the text reminder caused an increase in on-schedule watering among the gardeners in this experiment. The random sample gives a basis for extending that causal conclusion to registered county gardeners covered by the roster, subject to the limitations just noted.
Conclude: The experiment provides evidence that, for registered gardeners in this county, a text reminder rather than the usual email notice increases the percentage who water their plants on schedule over four weeks. This conclusion uses random assignment for cause and effect and random selection for generalization. It does not automatically apply to gardeners in other counties, people who are not registered, or different reminder systems and time periods.
Notice the conclusion does not merely say “the text reminder worked.” It identifies the treatments, response, direction of the finding, and population. It also avoids claiming that every individual gardener responded in the same way.
Worked Example: Random Assignment Without Random Selection
Worked Example: Volunteers Try Two Hydration Plans
A fictional research team recruits 80 volunteers through a notice at a city recreation center. The volunteers are randomly assigned to follow either a scheduled-water-break plan or their usual routine during a week of recreational exercise. The scheduled-break group reports a higher average hydration score, and a randomization analysis finds the difference statistically significant.
State: The experimental units are the 80 volunteers. The treatments are the scheduled-water-break plan and the usual routine, and the response is hydration score after one week. The questions are whether the plan affected scores and whether the finding applies to a wider group.
Plan: The volunteers were not randomly selected from all people who exercise, or from all city recreation-center users. They responded to an invitation, so the study lacks random selection from a defined population. However, volunteers were randomly assigned to the two plans. Assuming the assignment was followed and outcomes were measured comparably, that assignment supports a causal comparison for these experimental units.
Do: The observed higher average score, together with the statistically significant randomization result, provides evidence that the scheduled-break plan caused higher hydration scores for the volunteers in this experiment. The random assignment does not make the volunteers representative of people who did not volunteer. It cannot, on its own, justify generalizing the result to all people who exercise.
Conclude: The results provide evidence that the scheduled-water-break plan increased hydration scores for the volunteers studied during this week. Because participants were volunteers rather than a random sample from a target population, the researchers should not claim that the plan will have the same effect for all exercisers. The results may be useful for considering similar volunteers, but applying them elsewhere requires judgment and additional evidence.
This example shows why “randomized experiment” and “random sample” are not interchangeable descriptions. The experiment is randomized because treatment assignment used chance. Its participants were not randomly selected, so the basis for broad generalization is limited.
Worked Example: Random Selection Without Random Assignment
Worked Example: Shift Schedules and Sleep in Factory Workers
In a fictional study, investigators use a simple random sample of 150 workers from a factory’s current employee roster. They compare workers on a rotating night-shift schedule with workers on a fixed daytime schedule. Managers, not a chance process, determine each worker’s schedule. The sampled night-shift workers report less sleep on average, and the difference is statistically significant.
State: The observational units are the 150 sampled workers. The explanatory variable is work schedule, and the response is reported sleep duration. The question is whether the observed sleep difference can be described for the workforce and whether the schedules can be said to cause that difference.
Plan: The simple random sample from the employee roster supports generalizing an observed association to the factory’s workers represented by that roster, provided the frame covers the target population and nonresponse does not substantially distort the sample. But workers were not randomly assigned to schedules. Other factors—such as job duties, family responsibilities, or workers’ preferences—could be related to both schedule and sleep. The design therefore does not isolate the effect of schedule.
Do: The significant difference is evidence of an association between work schedule and reported sleep among the sampled workers. Because the sample was selected at random, the association may be generalized to the factory workforce represented by the roster, subject to the sampling limitations. Because schedules were assigned by managers rather than randomly, the difference cannot be attributed to the schedule itself: other variables may help explain it.
Conclude: In this fictional study, sampled workers on rotating night shifts reported less sleep on average than sampled workers on fixed daytime shifts, and the data provide evidence of an association that may apply to the factory workforce. The study does not establish that rotating night shifts caused the lower sleep. Random selection supports the population claim, but without random assignment the causal claim is not justified.
A statistically significant association is still an association. Statistical significance does not repair the lack of random assignment or remove possible confounding. As in Confounding in Experiments, a background variable that differs systematically between groups can offer an alternative explanation for their different outcomes.
Common Mistakes and AP Exam Tips
- Claiming that random assignment makes a sample representative. Assignment concerns treatment groups, not how participants entered the study. State whether the people were randomly selected before making a population claim.
- Claiming that a random sample proves causation. A representative sample can support generalizing an association, but it does not eliminate confounding when treatments were not randomly assigned.
- Saying random assignment alone proves an effect. Random assignment allows an observed treatment difference to support a cause-and-effect conclusion; the outcome evidence is needed to support the existence of an effect. It does not prove that every individual responds the same way.
- Generalizing beyond the actual sampling frame. A random sample from a roster supports claims about the population that roster adequately represents, not automatically about everyone with a broadly similar characteristic.
- Confusing “not supported” with “false.” If a study lacks random assignment, say it cannot establish a causal effect. Do not claim that the treatment definitely had no effect.
- Using “proves” in either conclusion. Random processes support particular kinds of inference, but study limitations remain. Prefer “provides evidence that,” “supports generalizing to,” and “does not establish that” when those statements fit the design.
For a strong AP response, identify the population and sample, then name whether random selection and random assignment occurred. Explain the role of each process and connect it to the specific conclusion. If the result is statistically significant, state what difference the data support; do not let the significance result stand in for a discussion of the design.
Check Your Understanding
For each question, distinguish the role of random selection from the role of random assignment.
- A randomly selected group of residents is surveyed about two existing neighborhood programs. Why can the sample support generalizing an association but not necessarily a cause-and-effect claim?
- Researchers randomly assign volunteers to two exercise plans. What conclusion can the assignment support, and what does it not establish about the population?
- An experiment randomly selects students from a school roster and then randomly assigns them to two study schedules. What role does each random process play?
- A study reports a statistically significant difference, but treatment groups were formed by participants’ own choices. Why does statistical significance not establish causation?
- Complete this statement: Random assignment concerns _______; random selection concerns _______.