Who Can an Observational Study Represent?
In Checking Conditions for Data from an Experiment, you saw that random assignment supports a different conclusion from random sampling. Here the central question is: when researchers observe outcomes without assigning treatments, how does the way they select people limit whom their results can represent?
An observational study records information about individuals without imposing a treatment or assigning them to groups. A survey about residents’ habits and a review of existing clinic records can both be observational studies. They may describe a group or provide evidence about a population, but observing an association alone does not show that one variable caused another.
For generalization, focus on how individuals entered the data. A random sample can support conclusions about the population from which it was selected, provided the sampling frame and response process are suitable. A convenience sample or a voluntary-response sample does not give every member of the target population a known chance of selection. Its results may describe the people who provided data, but they do not automatically represent everyone the researchers hoped to study.
The distinction is not simply “random means good, nonrandom means useless.” A nonrandom sample can provide useful descriptions of its respondents or suggest questions for further study. The important point is to match the claim to the evidence: the method determines whether the results support generalization beyond the observed individuals.
Random Sampling, Generalization, and Causation
As discussed in Recognizing Random Sampling Methods in Study Descriptions, a simple random sample, stratified random sample, or cluster sample uses chance to select units. When the frame appropriately covers the target population and the study is carried out well, these designs support generalizing from the sample to that population. The details still matter: a random sample of one school’s students does not represent every student in a state simply because the selection was random.
An observational study can use a random sample. That randomness supports population generalization, not a causal conclusion. For example, a random sample of adults might show that people who exercise more often also report better sleep. The random sample can help the researchers estimate how common those characteristics are in the population represented by the frame. Because exercise was not randomly assigned, the observed association does not establish that exercise caused better sleep.
The reverse distinction matters too. Researchers may randomly assign volunteers in an experiment, but that assignment does not make the volunteers a random sample of a broader population. As in the previous tutorial, assignment can support a causal comparison among the study participants; sampling is what supports generalization to a population. Always name which process occurred.
Even a probability sample does not guarantee perfect representation. Some selected people may not respond, some may be missing from the frame, and recorded answers may be inaccurate. In Conditions for a Survey With Nonresponse, you learned to distinguish random selection from who actually responds. A low response rate or a response pattern related to the characteristic of interest can weaken the case for generalizing to the full population.
Worked Examples
Worked Example: A Random Sample of Town Residents
A town has 12,000 adults listed in its municipal residential register. A researcher randomly selects 300 adults from that list and asks whether they received a flu vaccine this season. All 300 respond, and 186 say yes. The researcher wants to estimate the proportion of adults in the town who received the vaccine. Assess the scope of the conclusion and, assuming a 95% one-proportion \(z\)-interval is requested, check its conditions and interpret the interval.
Let \(p\) be the proportion of adults listed in the town’s municipal residential register who received a flu vaccine this season. The study is observational: it records vaccination status and does not assign a vaccine or another treatment.
Use a one-proportion \(z\)-interval if the sample is random, the 10% condition is met for sampling without replacement, and the observed success and failure counts are each at least 10. Check whether the frame and responses support generalization to the population named in the parameter.
The 300 adults were randomly selected from the register, and all responded. The sample is no more than 10% of the 12,000 listed adults because \(300\leq0.10(12{,}000)=1{,}200\). There are 186 successes and \(300-186=114\) failures, both at least 10. Thus, the random, 10%, and Large Counts conditions are met for the stated interval. The sample proportion is \(\hat{p}=186/300=0.62\). Using \(z^*=1.96\) for 95% confidence, the interval is:
The estimated standard error is \(\sqrt{0.62(0.38)/300}\approx0.0280\), and the margin of error is \(1.96(0.0280)\approx0.0549\). Rounded to three decimal places, the interval runs from 0.565 to 0.675.
We are 95% confident that between 56.5% and 67.5% of adults listed in the town’s municipal residential register received a flu vaccine this season. The random selection and complete response support generalization to adults represented by that register, not automatically to people missing from it or to adults in other towns. Since this is observational, the result does not show that any factor caused residents to receive or not receive a vaccine.
Worked Example: A Voluntary Online Poll
A school posts an optional online poll asking whether students support extending library hours. Of the 1,360 students who choose to answer, 884 select “yes.” The school has 9,000 enrolled students, and a staff member claims the result shows that 65% of all enrolled students support the change. Evaluate the claim.
The reported proportion among poll respondents is:
This calculation describes the 1,360 respondents. But students chose whether to open the poll and submit an answer; they were not randomly selected from all 9,000 enrolled students. Students with strong views, or those who noticed the post, may have been more likely to respond. That creates a plausible route for the respondents to differ from nonrespondents in their opinions.
The result therefore does not, by itself, support the claim that 65% of all enrolled students support extending library hours. The large number of responses does not remove the selection problem: a large self-selected group can still differ systematically from the wider student population. The data can be reported as 65% of the students who answered the online poll said “yes.” A stronger generalization would require a sampling method that gives students a chance to be selected, along with attention to nonresponse and the accuracy of the answers.
This is also an observational poll, not an experiment. It records opinions and does not assign students to different library-hour policies. The result neither establishes the opinion of the whole student body nor shows that extending hours would cause a change in student behavior.
Worked Example: Randomly Sampled Clinic Records
A clinic maintains a registry of 2,400 enrolled adult patients. To study whether patients completed a recommended screening last year, a researcher randomly selects 240 records and finds that 156 patients completed it. The records contain screening status for all 240 selected patients. The clinic plans to describe the result as the proportion of all adults in the region who completed screening. Explain what population the data can represent and what the result cannot establish.
The sample proportion is:
Because the records were randomly selected from the clinic’s registry, they can support estimating the screening-completion proportion among the 2,400 enrolled adult patients represented by that registry, assuming the records are accurate. The 10% condition is met: \(240\leq0.10(2{,}400)=240\). There are 156 patients who completed screening and \(240-156=84\) who did not, so both observed counts are at least 10. These checks would support using a one-proportion \(z\)-interval for the clinic’s registry population if an interval were requested.
The sample does not automatically represent all adults in the region. Adults who are not enrolled at this clinic are outside the registry and had no chance to be selected from it. The clinic could accurately report that the sample estimates the proportion among its enrolled adult patients; to generalize to all regional adults, it would need a sampling frame that represents those adults, not just this clinic’s patients.
The study is observational because the researcher reviewed existing records rather than assigning people to a screening program. Even if the records show that patients receiving more reminders were more likely to complete screening, that association would not establish that reminders caused higher completion. Other differences between patients could help explain the association.
Match the Conclusion to the Sampling Method
A useful way to evaluate a claim is to trace the path from the target population to the actual data. Ask who was eligible to be selected, what list or source supplied the sample, how chance was used, and who ultimately provided usable information. The answer identifies both the strength and the boundary of generalization.
- Random sample from a suitable frame: Generalization may be appropriate for the population represented by that frame, provided response and measurement concerns are not serious.
- Random sample from a limited frame: Generalization is limited to the group represented by that frame. Random selection does not extend coverage to people absent from the list.
- Voluntary-response sample: Results describe the people who chose to respond. Self-selection may make them unlike the intended population.
- Convenience sample: Results describe the people who were easiest to reach. Without a suitable random selection process, broad population generalization is not justified.
- Random assignment in an experiment: Assignment supports a causal comparison among study participants, but does not by itself establish that those participants represent a larger population.
These labels are not substitutes for explaining the study. “Random sample” needs a stated population and a sampling frame; “voluntary response” needs a clear account of how people decided to participate. In a written response, identify the method, say what population it represents, and state a limitation that follows from how the data were collected.
Common Mistakes and AP Exam Tips
- Generalizing to the intended population rather than the sampled population. A survey of one clinic’s patients does not automatically represent all people in the region. Name the frame and the group it covers.
- Assuming a large sample fixes selection bias. A large voluntary poll may estimate its respondents’ opinions precisely while still failing to represent nonrespondents. Sample size does not make self-selection random.
- Calling an observational association causal. Random sampling can support generalization, but it does not control other explanations for an association. Do not claim that one observed variable caused another.
- Using “random” without specifying the process. Say whether researchers randomly sampled people or randomly assigned treatments. As explained in Checking Conditions for Data from an Experiment, these processes support different conclusions.
- Ignoring coverage or nonresponse. Even a random sample from a list may omit people outside the frame, and selected people who do not respond may differ from those who do. State these limits when the study description raises them.
- Confusing a sample description with a population estimate. “65% of respondents said yes” describes the observed group. A claim about the larger population requires a sampling design that supports that generalization.
Key Takeaway
In an observational study, the sampling method determines how far a result can be generalized. Random selection from a suitable frame can support inference to the population represented by that frame; voluntary-response and convenience samples usually support conclusions about respondents or participants only. None of these sampling methods, by itself, turns an observed association into a causal effect.
Check Your Understanding
For each situation, identify the population the results can represent and explain any important limit on the conclusion.
- A random sample of 180 households is selected from a city’s 6,000 listed households. All respond to a survey about recycling. What population does the selection support generalizing to?
- A news site invites readers to vote in a poll about a proposed transit route. Why might the poll’s result fail to represent all residents, even if thousands respond?
- A researcher uses a random sample of patients from one medical practice to study an association between sleep and headaches. Does random sampling prove that poor sleep causes headaches? Explain.
- A random sample is drawn from a school’s student list. Can its results automatically be generalized to students at every school in the district? Why or why not?
- An observational study uses records from a clinic registry and finds that 72% of sampled patients received a screening. Write a sentence that appropriately describes the population the result can represent, assuming the sampling and records are suitable.