Why the Study Sample Is Not the Whole Story
A regression analysis is calculated using observed data, but the question it is meant to answer may concern people or other units beyond those observed. For example, a study may analyze records from patients at two clinics while its motivating question concerns adults with a particular condition more broadly. Whether the result can inform that broader group depends partly on how the study group was defined and assembled.
A useful generalization begins by stating exactly who or what the result is intended to inform. That group is not automatically the same as the people who were invited, enrolled, or included in the final analysis. Recruitment, eligibility rules, the study setting, and the time period can each narrow the scope of a defensible conclusion.
A population must be described in terms that make membership understandable. Depending on the question, this may require specifying the type of unit, relevant characteristics, location, and time period. “Patients” is usually too vague: it could mean patients at one clinic, people with a defined diagnosis, or everyone receiving care in a health system. The unit must also match the study question. If a study analyzes one record per clinic, for instance, the population relevant to that analysis consists of clinics rather than individual patients.
From Target Population to Analytic Sample
Between a target population and an analytic sample there may be several narrower groups. A study might be able to contact only people listed in certain records; it might invite some of them; and only some invited people may participate. Eligibility rules may also exclude people who otherwise belong to the broader group named in the original question. These distinctions help reveal the actual reach of the data.
| Group | Meaning | Question to ask |
|---|---|---|
| Target population | The group the study aims to inform | Which units, in what place and period, are included in the intended claim? |
| Reachable or source population | The part of the target group the study could approach using its records, locations, or recruitment channels | Who could realistically be contacted through the study’s chosen routes? |
| Recruited sample | The units that actually enroll or provide study data | Who participated, and who did not? |
| Analytic sample | The units whose observations are used in a particular analysis | Were any recruited observations missing or excluded? |
These groups need not be identical. A study can define a broad target population but recruit from a much narrower source population. Even if every recruited participant is analyzed, the analytic sample may still differ from the target in important ways. Naming each group makes the study’s scope visible; it does not, by itself, determine whether a regression calculation is correct.
Worked Example: A Clinic-Based Sample
Suppose a hypothetical study asks how a health measurement is associated with age among adults with a particular condition across a state. The investigators recruit adults attending three clinics in one city. The clinics’ records identify 800 potentially eligible adults; 240 are invited, 150 enroll, and complete records are available for 132. Identify the groups and assess the scope of a possible conclusion.
Target population. As stated, the target is adults with the condition across the state. A clearer definition would also specify the time period and any other intended eligibility criteria.
Reachable group. The 800 adults identified in records at the three clinics are reachable through the investigators’ chosen sources. They do not necessarily represent every adult with the condition in the state: people who receive care elsewhere, for example, are not in these records.
Recruitment and analysis. The invited group contains 240 adults, the recruited sample contains 150, and the analytic sample contains 132. The analytic sample is \(132/150=0.88\), or 88% of those enrolled. This arithmetic describes data availability among enrollees; it does not show that the 132 people represent all adults with the condition statewide.
A conclusion about the 132 analyzed adults is directly tied to the data. Extending it to all adults with the condition across the state requires additional justification, because the study used only three clinics and not everyone invited enrolled or had complete records. The data could still be informative, but the broad population claim should acknowledge these limits.
How Recruitment and Eligibility Set Boundaries
Recruitment determines which members of a reachable group enter a study. Eligibility criteria determine who may participate. Both choices can make a sample less suitable for a broader target population, even when they are reasonable for the study’s purpose. A study of people who volunteer through an online advertisement, for example, directly describes those volunteers; it does not automatically describe every person who meets the health criteria in the advertisement.
The key question is not simply whether the sample “looks representative” in a general sense. It is whether the route into the study and the eligibility rules could leave out groups relevant to the intended conclusion. A sample may resemble its target in one observed feature but differ in another that matters to the study question. Conversely, some differences between sample and target may be unimportant for a particular question. Generalization therefore requires reasoning about the question and context, not a universal checklist.
A large sample can make a calculated result more precise for the sample and still leave the target-population question unresolved. Increasing the number of participants recruited through the same narrow channel does not, on its own, bring people outside that channel into the data. Size and scope are different issues.
Worked Example: A Volunteer Recruitment Route
Imagine a hypothetical study of the relationship between weekly exercise and a health measure among adults aged 40 to 60. Recruitment is conducted through a fitness app, and 1,200 app users complete the survey. The researchers would like to discuss adults aged 40 to 60 generally. What can they reasonably say about the population represented?
The 1,200 respondents are the study sample. The reachable group consists of people the app-based recruitment route could reach, and the participants are those who chose to respond. Adults aged 40 to 60 who do not use the app could not enter through this route. App users who chose not to respond are also absent.
The sample size is substantial, but the route still defines a boundary. If app use or willingness to answer is related to exercise habits or the health measure, the observed association among respondents may not carry over unchanged to all adults in the age range. The study has direct evidence about its respondents. A broader claim needs an argument that the recruitment differences do not undermine the particular conclusion; the count of 1,200 alone is not that argument.
A careful description would state that the analysis concerns adults aged 40 to 60 who responded to app-based recruitment. It could discuss broader relevance cautiously, but should not present the respondents as a random cross-section of all adults in that age group.
Setting and Time Are Part of the Population Definition
A target population is not defined only by personal or organizational characteristics. Place and time matter too. A finding based on one hospital system may not apply in the same way to a system with different referral practices or access to care. A study conducted during a particular period may not describe a later period if services, policies, or the composition of the population have changed.
This does not mean that results from one setting are useless elsewhere. It means that moving from the observed setting to another is a generalization that needs support. Ask whether the new setting has relevant features in common with the study setting, and whether important differences could change the relationship being studied. If the answer is unknown, describe the uncertainty instead of silently treating the settings as interchangeable.
Worked Example: Two Settings and a Narrow Eligibility Rule
Consider a hypothetical study of 300 adults receiving care at urban clinics. To participate, a person must speak the study language and attend an in-person appointment during a six-month recruitment period. The researchers want their finding to inform adults with the same condition throughout the country, including rural residents. Assess the boundaries step by step.
Eligibility boundary. Adults who do not speak the study language are excluded, even if they have the condition. The analyzed group therefore cannot directly describe that excluded group without further evidence.
Setting boundary. Recruitment at urban clinics gives no direct sample of rural clinics or people who receive care through other routes. Differences in access or care context could matter, so national generalization is not guaranteed.
Time boundary. The data cover a six-month recruitment period. A claim about that period is more directly tied to the observed data than an unqualified claim about all future periods.
The most defensible description identifies the studied group: eligible adults attending the participating urban clinics during the recruitment period. The wider national population may be a sensible intended audience, but extending the result there requires considering the language, location, and time restrictions rather than ignoring them.
A Practical Generalization Check
Before describing a regression result as relevant beyond its analytic sample, work through the chain from the intended audience to the data actually analyzed. This is a study-design judgment, not a mechanical test that produces a yes-or-no answer. The purpose is to make the claim’s reach explicit and to identify where supporting evidence would be needed.
Name the units, relevant characteristics, setting, and time period that match the study question.
Identify the records, sites, invitations, or other recruitment routes through which participants could enter.
Note eligibility criteria, nonparticipation, and any reduction from the recruited group to the analytic sample.
Ask which parts of the target population or context are not represented and whether those differences could matter to the question.
Describe the analyzed group directly; qualify any extension to a broader group and say why it is plausible or uncertain.
Why Generalization Is an Argument, Not a Sample-Size Rule
The logic is straightforward. Data directly describe the units whose observations were analyzed. A claim about a larger group adds a step: it assumes that the observed units provide relevant information about units not observed. If some members of the target could not enter the study, or if participants differ from nonparticipants in ways relevant to the question, that extra step needs justification. More observations can improve the detail of what is known about the observed sample, but they cannot by themselves establish that the missing groups behave similarly.
This distinction is useful when reading or reporting regression results. Instead of saying “the regression applies to everyone,” specify the group represented by the data and then identify any broader group the study hopes to inform. A transparent account of limits is not a claim that the study has no value; it tells readers how far the evidence can reasonably travel.
Common Mistakes
- Equating the sample with the target population. The sample is observed; the target is the intended group. State both rather than using “the population” ambiguously.
- Treating a large sample as automatically generalizable. A large number does not repair a recruitment route that excludes relevant parts of the intended population.
- Ignoring eligibility rules. If people are excluded by language, age, setting, or another criterion, conclusions should acknowledge that boundary.
- Assuming one setting stands for every setting. Results from particular locations may inform other locations, but that extension requires a reasoned comparison.
- Making an unlimited claim from a time-limited study. Include the study period when changes over time could alter who is represented or the context of the finding.
Check Your Understanding
Try these questions without looking back.
- A study aims to inform all adults with a condition in a state but recruits only from one specialist clinic. Identify the target population and one important boundary on the sample.
- Explain the difference between a recruited sample and an analytic sample.
- Why does increasing a convenience sample from 100 to 1,000 people not automatically justify generalizing to everyone in the target age group?
- A study excludes people who cannot attend weekday appointments. What should a reader consider before applying its finding to the broader target population?
- Give two ways that the setting or time period can limit the scope of a study’s conclusions.