Two Questions Set the Scope of a Conclusion
In “Interpreting Results Without Overclaiming,” you distinguished association, causation, and generalization. Here we will use those distinctions as a practical check on a written conclusion. The key is to ask two separate questions: Who do the data represent? and What does the way groups were formed allow us to claim?
Random sampling and random assignment answer different questions. Random sampling is about selecting individuals from a defined population; it can support generalizing results to that population. Random assignment is about placing participants into treatment groups; it can support a cause-and-effect conclusion about the treatments for the participants in the experiment. One does not substitute for the other.
This check does not decide whether an observed difference is statistically convincing. A study’s design determines the kinds of claims its results may support; the analysis determines how strongly the data support a particular result. As in “Writing a Complete Inference Response,” do not turn a sample difference into convincing evidence without appropriate statistical evidence.
Use a Scope Check Before Writing
A useful technique is to make a short scope ledger before drafting the conclusion. Record the population of interest, who actually participated, how they were selected, and how treatment groups were formed. Then check each intended claim against that information.
Be specific about the group the question is about, such as all current members of a named community center.
A random sample from a defined population can support generalization to that population. Volunteers or a convenience sample do not automatically represent a wider population.
If researchers randomly assigned treatments, a causal interpretation may be supported. If they only measured existing groups or behaviors, describe an association rather than claiming that one variable caused another.
Keep the conclusion within the represented population and the design’s causal limits. Separately check whether the statistical analysis supports a claim of convincing evidence.
The scope ledger helps prevent a common mix-up: saying that an experiment’s result applies to a broad population simply because participants were randomly assigned. Assignment can help support a causal claim, but it does not turn volunteers into a random sample. Conversely, a random sample can represent a population, but it does not make an observational association causal.
What Each Design Feature Adds
| Selection and group formation | Conclusion the design may support | Claim to avoid |
|---|---|---|
| Random sample; variables observed, not assigned | Generalize an observed association to the population sampled. | Claim that one measured variable caused the other. |
| Volunteers or another nonrandom group; treatments randomly assigned | Support a causal interpretation for the participants studied. | Assume the participants represent a broader population. |
| Random sample; treatments randomly assigned | Support a causal conclusion for the population represented by the sample, if the evidence warrants it. | Extend the result beyond the sampled population without justification. |
| Nonrandom group; variables observed, not assigned | Describe the observed pattern among the people studied. | Claim broad generalization or causation based on these features alone. |
The word may matters. Random selection and assignment support particular kinds of inference, but do not guarantee that the study was carried out well or that the result is convincing. For example, substantial nonresponse can make a random sample less representative than the original selection plan suggests. Participants who do not follow their assigned treatment, or who leave the study at different rates, can also complicate interpretation. State relevant limitations when the description provides them.
Worked Example: A Random Sample Survey
Worked Example: Screen Time and Bedtime
Scenario. A fictional district has 1,800 students in its middle schools. Researchers randomly select 180 students from the enrollment list. Each student reports whether they usually use a screen during the hour before bed, and the researchers record each student’s usual bedtime. In the sample, the 108 students who report screen use have a mean bedtime of 10:18 p.m.; the other 72 have a mean bedtime of 9:54 p.m.
What the sample shows. The observed difference is 24 minutes: 10:18 p.m. is 24 minutes later than 9:54 p.m. In this sample, students reporting screen use before bed had a later mean bedtime than students not reporting it. This describes an association between reported screen use and bedtime; the researchers measured screen use rather than assigning it.
Conclusion that fits. “Among the 180 students randomly selected from this district’s middle-school enrollment, those who reported using a screen during the hour before bed had a mean usual bedtime 24 minutes later than those who did not report that use. The survey shows an association, not that screen use caused the later bedtime. Because the students were randomly selected from the district’s enrollment, the association may be generalized to that enrollment, subject to limitations such as nonresponse.”
Why this wording fits. Random selection supports generalization to the population on the enrollment list, not to all middle-school students everywhere. Because screen use was observed rather than assigned, the result does not establish that it caused a later bedtime. The difference between sample means describes the data; without an inference analysis, it does not establish convincing evidence of a difference in the population.
Worked Example: A Volunteer Experiment
Worked Example: A Hydration Reminder
Scenario. Sixty adults volunteer for a fictional study of a phone reminder intended to encourage regular water breaks during a workday. Researchers randomly assign 30 volunteers to receive the reminder and 30 to follow their usual routine. At the end of two weeks, 21 people in the reminder group and 15 in the usual-routine group meet a stated daily water-break goal.
Compare the sample proportions. The proportion meeting the goal is \(21/30=0.70\), or 70%, in the reminder group. In the usual-routine group, it is \(15/30=0.50\), or 50%. The observed difference is \(0.70-0.50=0.20\), or 20 percentage points.
Conclusion that fits. “Among these 60 volunteers, the proportion meeting the water-break goal was 20 percentage points higher for those assigned to receive the reminder than for those assigned to follow their usual routine. Because the researchers randomly assigned the two conditions, the experiment can support a cause-and-effect interpretation for these participants if the study was carried out as planned. The volunteers were not a random sample of adults, so the result does not automatically generalize to all adults.”
Why this wording fits. Random assignment supports causal reasoning about the comparison for the people in the experiment; it does not establish that the reminder will have the same effect for other adults. The 20-percentage-point difference is an observed sample result. Because no inference results are provided, it would be too strong to say that the experiment gives convincing evidence of a treatment effect.
Worked Example: Both Random Sampling and Random Assignment
Worked Example: Testing a Library Pickup Message
Scenario. A fictional public library system has 2,400 adult cardholders. Researchers randomly select 200 cardholders for a study and then randomly assign 100 to receive a new pickup reminder and 100 to receive the library’s usual message. By the end of the study, 68 people in the new-message group and 54 in the usual-message group pick up a reserved item.
Compare the sample proportions. The pickup proportion is \(68/100=0.68\), or 68%, for the new message and \(54/100=0.54\), or 54%, for the usual message. The observed difference is \(0.68-0.54=0.14\), or 14 percentage points.
Conclusion that fits. “In this sample of cardholders, the pickup proportion was 14 percentage points higher among those assigned the new reminder than among those assigned the usual message. Random assignment supports a causal interpretation of the message comparison, and random selection supports generalizing to the library system’s adult cardholders. Whether there is convincing evidence that the new message increases pickup in that population depends on the statistical analysis.”
Why this wording fits. The study has both features, so its design can support a population-level causal conclusion about adult cardholders, provided the analysis supports that conclusion and the study was conducted as planned. Its scope remains the library system’s adult cardholders. Neither randomization feature justifies a claim about cardholders in other systems or about people who do not have library cards.
Worked Example: Neither Feature Is Present
Worked Example: A Convenience Sample at a Farmers’ Market
Scenario. A student interviews 45 shoppers who happen to visit a fictional farmers’ market on Saturday morning. Shoppers report whether they bring a reusable bag and how often they shop for groceries. The interviewed shoppers who report bringing a reusable bag also report shopping more frequently, on average. The student did not select shoppers randomly and did not assign anyone to use a particular type of bag.
Conclusion that fits. “Among the 45 shoppers interviewed at the market that Saturday, those who reported bringing a reusable bag also reported more frequent grocery shopping on average. This convenience sample describes an association among the people interviewed. It does not establish that bringing a reusable bag causes more frequent shopping, and it does not by itself support generalizing the association to all market shoppers or residents of the area.”
Why this wording fits. The observed difference may be reported as a feature of these interviewees, but shoppers who visit at another time or shop elsewhere may differ. Since no treatment was randomly assigned, the design does not support a causal claim. Since no random sample was selected, there is no design-based reason to treat the interviewees as representative of a larger population.
Common Mistakes and AP Exam Tips
- Using “random” without saying what was randomized. State whether people were randomly selected, randomly assigned, or both. Full-credit wording explains what each feature supports.
- Generalizing to the wrong group. A random sample from a school supports generalization, at most, to the population represented by that school’s sampling frame—not automatically to all students. Name the population explicitly.
- Calling an observed difference causal. If researchers measured behavior or existing group membership, describe an association. A causal claim requires a design with random assignment to treatments, not merely a difference between groups.
- Assuming random assignment makes the sample representative. It does not. Random assignment governs treatment groups; it is random selection that supports generalization to a population.
- Assuming a design feature guarantees a strong result. Random sampling and random assignment do not show that a difference is large, important, or statistically convincing. Report the sample result and use “convincing evidence” only when the statistical analysis warrants it.
- Ignoring the study’s actual population and setting. Even when both random sampling and assignment are used, a conclusion should not quietly expand from one defined population to everyone. Keep the setting and population visible.
On an AP response, make the design logic explicit. A strong answer might say, “Because the sample was randomly selected from the defined population, the results may be generalized to that population; because treatments were randomly assigned, the comparison can support a cause-and-effect conclusion.” Then add the result and qualify the strength of the evidence. This is more precise than simply saying that a study was “randomized.”
Check Your Understanding
For each situation, identify the design feature that matters and write a conclusion whose scope fits.
- A random sample of residents from one town is asked about weekly cycling and resting heart rate. Can the study support generalizing an association? Can it establish that cycling caused a lower heart rate?
- Thirty volunteers are randomly assigned to one of two study-planning routines. What can random assignment support, and what does it not establish about students generally?
- Researchers randomly select members of a neighborhood association and randomly assign them to receive one of two garden-watering instructions. What does each kind of randomization contribute?
- A convenience sample of shoppers is surveyed about reusable bags and grocery spending. What can be described, and which broader or causal claims should be avoided?
- A randomized experiment reports that the treatment group’s sample proportion is 12 percentage points higher, but gives no inference results. What can the conclusion state, and what should it not claim?