Two Kinds of Randomness, Two Different Questions
A study may be designed to answer two questions: Do its findings apply to a wider population? And did a treatment cause a difference in the outcome? These are separate questions. As you learned in Generalizing Results to a Population and Association Versus Causation in Study Conclusions, random selection and random assignment serve different purposes.
Random selection uses chance to choose individuals or units from a population to be in a sample. It addresses how far findings can be generalized. Random assignment uses chance to place study participants into treatment groups in an experiment. It helps make the groups comparable, supporting a cause-and-effect conclusion about the treatments.
A study can use both methods, just one, or neither. This gives four combinations. The most useful first step is to check each feature separately rather than deciding that a study is simply “random” or “not random.” Ask whether the sample was randomly selected from a named population, and then ask whether participants were randomly assigned to treatments.
The Four Combinations
The table summarizes what each combination supports. “Generalize” means using sample findings to draw a conclusion about the population from which the sample was randomly selected. “Claim cause” means concluding that a treatment caused a difference in the response, based on a properly conducted randomized experiment.
| Random selection? | Random assignment? | What the design supports |
|---|---|---|
| Yes | Yes | Generalizing to the sampled-from population and a cause-and-effect conclusion about the treatments |
| Yes | No | Generalizing to the sampled-from population, but not a cause-and-effect conclusion |
| No | Yes | A cause-and-effect conclusion for the participants in the experiment, but not automatic generalization to a wider population |
| No | No | Neither automatic generalization to a wider population nor a cause-and-effect conclusion |
The first combination has both links needed for the two types of inference: chance selects the sample from a population, and chance assigns the sampled participants to treatments. In the second combination, the random sample can represent its population, but if researchers only observe existing conditions, other variables may help explain an association. A random sample alone does not establish cause.
In the third combination, random assignment supports comparing the treatments for the people who took part. But if those people were volunteers or a convenience sample, they might not represent the broader population a researcher wants to discuss. In the fourth combination, a nonrandom sample and no random assignment provide neither of these design-based supports. The study may still describe the participants or show an association among measured variables, but its conclusions should be limited accordingly.
These are conclusions supported by study design, not guarantees that a result is correct or that a study has no weaknesses. A random sample can still have nonresponse or gaps in its sampling frame, as discussed in Generalizing Results to a Population. An experiment can also be affected by problems such as poor measurement or participants not following their assigned treatments. State the design’s support, and mention relevant limitations when the situation provides them.
A Four-Question Classification Method
Use the following process whenever a study description includes words such as “random,” “volunteer,” “assigned,” or “observed.” Be precise about what was randomized: selecting a sample and assigning treatments are distinct actions.
Identify the population the researchers hope to learn about and who actually provided data.
Were participants chosen by chance from that population? If so, the design supports generalizing to that population, subject to coverage and response limitations.
Did researchers deliberately impose treatments and use chance to assign participants to them? If so, the experiment supports a cause-and-effect conclusion for its participants.
State whether generalization, a causal conclusion, both, or neither is supported. Name the group and treatments precisely.
If participants simply report a behavior or condition that already exists, researchers did not randomly assign it. Such a study is observational, even if its sample was randomly selected. If researchers impose treatments and randomly assign participants, it is an experiment; whether those participants represent a wider population still depends on how they were selected.
Worked Example: A Random Sample and a Randomized Experiment
Worked Example: A Random Sample and a Randomized Experiment
A county health team wants to test whether a text reminder increases attendance at routine vaccination appointments. From a complete list of adult county residents with an appointment scheduled in the next month, the team randomly selects 400 people. It then randomly assigns 200 to receive an extra text reminder and 200 to receive the usual appointment information. The team compares whether participants attend. What conclusions can the design support?
The population is adults in the county who have a routine vaccination appointment scheduled in that month and are covered by the list. Random selection from that list supports generalizing findings to that population, assuming the list covers the intended group and participation or missing outcome data do not create substantial problems.
The team also imposed two conditions and randomly assigned the selected adults to them. That makes this a randomized experiment. A difference in attendance between the assigned groups can support a cause-and-effect conclusion about the extra text reminder for the participants in the experiment. Because the sample was also randomly selected from the stated population, the design supports extending that causal finding to the population represented by the list, with appropriate caution about study limitations.
This is the “yes” and “yes” combination: random selection supports generalization, and random assignment supports causation. A suitably scoped conclusion is: “For adults in the county with a routine vaccination appointment scheduled in the study month, the experiment supports a conclusion about whether adding a text reminder affects attendance.” It would not automatically apply to people without appointments, residents of other counties, or a different time period.
Worked Example: A Random Sample Without Assigned Treatments
Worked Example: A Random Sample Without Assigned Treatments
A fictional statewide study randomly selects 800 students from a complete list of students enrolled in the state’s community colleges. Researchers ask each student how many hours they slept on a typical school night and record the student’s current course average. They find that students reporting more sleep tend to have higher course averages. Can they generalize the association? Can they conclude that more sleep caused higher averages?
The selection method is random, and the list covers the stated population of enrolled community-college students in the state. Subject to issues such as nonresponse, the sample can support generalizing the observed association to that population. However, researchers did not assign students to sleep amounts; they measured students’ existing sleep and course averages. This is an observational study, not a randomized experiment.
Other variables could help explain the association. For instance, work schedules, course load, or health could be related to both sleep and course averages. The study therefore does not establish that increasing sleep would cause a higher course average. A careful conclusion is: “Among the state’s community-college students, the study can support generalizing an association between reported sleep and course average; it does not establish that sleep caused the difference.”
This is random selection without random assignment. It supports generalization to the population sampled from, but not a cause-and-effect claim.
Worked Example: A Randomized Experiment With Volunteers
Worked Example: A Randomized Experiment With Volunteers
A fictional software team invites people to volunteer for a test of two layouts for a budgeting app. Eighty volunteers take part. The team randomly assigns 40 to use Layout A and 40 to use Layout B, then records how many tasks each person completes in ten minutes. Suppose the Layout A group completes more tasks on average. What can the team conclude?
The participants volunteered; they were not randomly selected from all budgeting-app users. The study therefore does not provide a random-sample basis for generalizing to all users. Volunteers may differ from other users in experience, interest, or other ways relevant to task completion.
However, the team did randomly assign these participants to the two layouts in an experiment. If the study was carried out appropriately, a difference in the groups’ task completion supports a cause-and-effect conclusion about the layouts for the 80 volunteers. The design supports saying that Layout A caused more tasks to be completed than Layout B for these experimental participants, if the observed difference is convincing in the study’s analysis. It does not automatically establish the same effect for every app user.
This is random assignment without random selection. The experiment supports a causal conclusion for its participants, but broad generalization is not supported by the sampling method.
Worked Example: Neither Feature Is Randomized
Worked Example: Neither Feature Is Randomized
A school club posts an online questionnaire asking students whether they use a study-planning app and what their latest quiz score was. Students choose whether to respond. Among the respondents, app users tend to have higher scores. The club did not select students at random or assign them to use or not use the app. What can it conclude?
Students chose whether to respond, so this is not a random sample from the school’s student population. The respondents’ results do not automatically generalize to all students. Also, the club did not assign app use; it recorded a condition students already chose. The comparison is observational and does not establish that the app caused higher quiz scores. Students who use the app may differ in study habits, preparation, or other relevant ways.
The supported statement is limited to the respondents: “Among students who completed the questionnaire, app use was associated with higher reported quiz scores.” Even that association describes these respondents; it does not by itself explain why the groups differed.
This is neither random selection nor random assignment. The study supports neither broad generalization nor a causal claim.
Common Mistakes and AP Exam Tips
- Calling any study with a random step “random.” Say what was random: the selection of participants from a population, or the assignment of participants to treatments. Identify both separately when both occurred.
- Treating a random sample as proof of cause. Random selection can support generalizing findings to the sampled-from population. Without random assignment to treatments, an observed relationship does not establish cause.
- Treating randomized volunteers as representative. Random assignment can support a causal conclusion for experiment participants. It does not make volunteers representative of a broader population.
- Overstating the population. Generalize only to the population from which the random sample was selected, not automatically to other places, groups, or times.
- Leaving the conclusion vague. “The study is random” or “the results are valid” does not say what inference the design supports. Name the population, the treatment or relationship, and the supported scope.
For full credit, connect each design feature to the inference it supports. For example: “Because the researchers randomly selected participants from the stated population, the findings can be generalized to that population. Because they randomly assigned participants to treatments, the experiment supports a cause-and-effect conclusion about those treatments.” If only one feature is present, state only the inference that feature supports and explain the limit on the other.
Check Your Understanding
For each situation, identify whether random selection and random assignment were used, then state which conclusions the design supports.
- A randomly selected sample of residents reports how often they exercise. Researchers do not assign exercise amounts. What kind of inference does the random selection support, and can the study establish that exercise causes a measured outcome?
- People volunteer for a study and are randomly assigned to one of two versions of a language-learning app. What does random assignment support, and what does volunteering limit?
- A researcher randomly selects patients from a clinic list, then randomly assigns them to two treatment plans. Which two kinds of inference does the design support, and to what population is generalization directed?
- Students choose whether to answer a survey about screen time and sleep. No treatments are assigned. What are the two limits on conclusions from this study?
- In one sentence, explain why random selection and random assignment should not be treated as interchangeable.