Who Is Represented in the Analysis?
A study can have a clear research question and carefully measured variables yet still analyze a group that differs from the people its conclusions are meant to describe. This can happen before any measurements are taken: eligibility rules exclude some people, some eligible people are never reached, some decline, and some enrolled participants do not provide the outcome measurement. Each step affects who can appear in the final analysis.
It is useful to trace these steps separately. “The study had 200 participants” does not tell us how many people were eligible, how many were invited, who declined, or who was missing from the analysis. Those details help assess whether a sample-based summary or regression relationship is informative for the intended population.
These stages have different denominators. An invitation rate is the number invited divided by the number eligible; a participation rate is the number who enroll divided by the number invited; and a retention rate is the number measured at follow-up divided by the number enrolled. Researchers may use different conventions for a “response rate,” so a rate should always be accompanied by its numerator and denominator.
Eligibility also sets a boundary on what the study directly covers. If a study enrolls only adults with a particular diagnosis, its analyzed data cannot directly describe children or adults without that diagnosis. A broader conclusion would need a separate justification. Even within the eligible group, however, participation and retention can further change the composition of the analyzed sample.
Selection and Sample-Based Conclusions
Selection is not automatically a problem. If people are selected by a process that preserves the relevant characteristics of the intended population, the analyzed sample may support useful conclusions about that population. In particular, a properly conducted random sample from a defined population gives a principled basis for generalization, subject to sampling variability and the study’s other limitations. By contrast, a group of volunteers is not automatically representative merely because it is large.
Selection becomes a concern when inclusion is related to the quantity being estimated or to variables relevant to the analysis. For a mean, for example, if people with higher outcome values are more likely to participate, the mean among participants may differ from the mean among all eligible people. For a regression relationship, selection can change the distribution of the predictor, the outcome, or both, and may consequently change the observed relationship.
A simple identity makes one part of this issue precise. Let \(Y\) be an outcome and let \(R=1\) indicate that a person is included in the analyzed sample; \(R=0\) indicates non-inclusion. Write \(p=P(R=1)\), and suppose \(p>0\). The covariance between \(Y\) and the inclusion indicator is
The equality follows because \(E[R]=p\) and \(E[YR]=pE[Y\mid R=1]\). Therefore,
If inclusion is unassociated with \(Y\), the covariance is zero and the selected and full-population means are equal in expectation. If inclusion tends to occur more often for people with larger \(Y\), the covariance can be positive and the selected mean can exceed the population mean. This identity concerns a mean; equality of means alone does not establish that every feature of a population distribution, or a regression relationship, is preserved.
Worked Example: Following the Recruitment Funnel
Suppose a hypothetical study defines its eligible population as 120 adults in a service area: 60 who work remotely and 60 who do not. All 120 are invited. Of the remote workers, 30 enroll, and 24 of those provide the required measurement. Of the non-remote workers, 20 enroll, and 12 of those provide the measurement. Calculate the rates and compare the analyzed sample with the eligible group.
Participation. Among remote workers, the participation rate is \(30/60=0.50\), or 50%. Among non-remote workers, it is \(20/60=0.333\), or 33.3%. Overall, \(50\) of \(120\) invited people enroll, so the overall participation rate is \(50/120=41.7\%\).
Retention. Among enrolled remote workers, \(24/30=80\%\) provide the measurement. Among enrolled non-remote workers, \(12/20=60\%\) provide it. Overall, \(36\) of the \(50\) enrolled participants are measured, giving a retention rate of \(36/50=72.0\%\).
Composition of the analyzed sample. The eligible group is 50% remote workers. In the analyzed sample, \(24\) of \(36\) people are remote workers, or \(24/36=66.7\%\). The difference, calculated from the exact fractions, is \(100(24/36-60/120)=16.7\) percentage points.
Remote workers are therefore more common in the analyzed sample than in the eligible group. This comparison does not by itself show that a particular outcome mean or regression slope is biased: that depends on how remote-work status relates to the study’s measured variables. It does show that the analyzed sample has a different composition on a potentially relevant characteristic, so conclusions about all 120 eligible adults need care.
Selection Can Change a Regression Relationship
For a regression question, the key concern is not only whether the sample has the same proportions of different kinds of people as the intended population. The joint pattern of predictor and outcome values matters too. If inclusion depends on either variable, or on a characteristic related to them, the relationship among included cases can differ from the relationship among all eligible cases.
A small example makes this concrete. With a binary predictor \(X\), the least-squares slope from a simple regression of \(Y\) on \(X\) equals the difference between the mean outcome for \(X=1\) and the mean outcome for \(X=0\). This follows because the fitted value at \(X=0\) is the \(X=0\) group mean and the fitted value at \(X=1\) is the \(X=1\) group mean; the slope is the change between those two fitted values. Thus, if selection changes either group’s outcome mean, it can change the slope.
Worked Example: Volunteers and a Binary-Predictor Slope
In a hypothetical eligible group, \(X=0\) means not enrolled in a weekly activity and \(X=1\) means enrolled. The outcome \(Y\) is a hypothetical score, in points. Among ten eligible people with \(X=0\), five have \(Y=0\) and five have \(Y=10\). Among ten with \(X=1\), five have \(Y=10\) and five have \(Y=20\). Suppose all five people with \(Y=0\) in the \(X=0\) group volunteer, while all ten people in the \(X=1\) group volunteer. Calculate the group means and the slopes for the eligible group and the volunteers.
Eligible group. For \(X=0\), the outcome mean is
For \(X=1\), it is
The slope for this binary predictor is \(15-5=10\) points. It describes a 10-point difference in mean outcome between the two \(X\) groups in this eligible group.
Volunteers. The volunteers with \(X=0\) are the five people scoring zero, so their mean is \(0/5=0\) points. The ten volunteers with \(X=1\) have mean \(15\) points, as calculated above. The slope among volunteers is therefore \(15-0=15\) points.
The volunteer slope is 5 points larger than the slope in the eligible group, even though the eligible group and volunteers come from the same hypothetical setting. Here, selection is directly related to \(Y\) within the \(X=0\) group. The example illustrates how selection can alter a regression relationship; it does not establish that every volunteer sample will do so, or that the relationship is causal.
Loss Before Measurement
Selection does not stop at enrollment. Participants may be unavailable, withdraw, or miss a follow-up measurement. For an analysis using that follow-up outcome, people without the measurement cannot contribute an observed outcome value. If loss is related to the outcome or to the predictor–outcome pattern, the remaining cases may no longer reflect the enrolled group for the question being asked.
It is important to distinguish the number lost from the reason or pattern of loss. A high overall retention rate does not guarantee that the observed cases preserve the relevant relationship. Conversely, a lower retention rate does not by itself prove that the analysis is distorted. The key question is whether those measured and those not measured differ in ways relevant to the estimate. Often the unmeasured outcomes are unknown, so a study cannot settle that question simply by comparing outcome means.
Worked Example: Follow-Up Loss and an Observed Difference
Consider a hypothetical cohort of 100 enrolled adults. Let \(X=1\) denote a baseline characteristic and \(X=0\) its absence; let \(Y=1\) indicate a later outcome. For illustration, suppose the outcome is known for everyone, including those who were not measured at follow-up. In practice, the outcomes of people lost to follow-up would generally not be known. The counts are:
| Baseline group | Enrolled | With outcome among enrolled | Retained and measured | With outcome among measured |
|---|---|---|---|---|
| \(X=1\) | 40 | 24 | 24 | 12 |
| \(X=0\) | 60 | 18 | 45 | 15 |
Difference among all enrolled. In the \(X=1\) group, the outcome proportion is \(24/40=60.0\%\). In the \(X=0\) group, it is \(18/60=30.0\%\). The difference is \(60.0\%-30.0\%=30.0\) percentage points.
Difference among those measured. In the retained \(X=1\) group, the outcome proportion is \(12/24=50.0\%\). In the retained \(X=0\) group, it is \(15/45=33.3\%\). The difference is \(50.0\%-33.3\%=16.7\) percentage points, rounded to one decimal place.
The observed difference is smaller than the difference among all enrolled adults in this hypothetical example. The retention rates also differ: \(24/40=60.0\%\) for \(X=1\) and \(45/60=75.0\%\) for \(X=0\). More importantly, retention differs by outcome status within each group: for \(X=1\), \(12/24=50.0\%\) of those with the outcome and \(12/16=75.0\%\) of those without it are retained; for \(X=0\), \(15/18=83.3\%\) of those with the outcome and \(30/42=71.4\%\) of those without it are retained. These patterns help explain why the measured cases show a different contrast. If the unobserved outcomes were genuinely unknown, the enrolled-group proportions could not be calculated this way from the observed data alone.
How to Assess Selection in a Study
A practical assessment follows the recruitment process from its starting population to the cases used for analysis. The aim is not to label every difference as bias, but to identify which conclusions are supported and which require caution.
State which people or units the research question concerns. Compare this group with the population allowed by the eligibility rules.
Record how many were eligible, reached or invited, enrolled, retained, and included in the specific analysis. State the denominator whenever reporting a rate.
Where information is available, compare participants with nonparticipants and retained participants with those lost on characteristics relevant to the research question.
Describe the analyzed sample accurately. Extend conclusions beyond it only when the recruitment process and available evidence support that extension.
A useful report gives enough information to evaluate the funnel: eligibility criteria, recruitment source, numbers at each stage, and the amount of missing follow-up measurement. It should also say whether the analysis includes all enrolled participants or only those with complete measurements. These details are part of interpreting the regression result, not administrative trivia.
Common Mistake: Treating a Large Sample as a Representative One
Wrong reasoning: “The study enrolled thousands of people, so the sample must represent the intended population and its regression relationship must apply to that population.”
Correction: Sample size alone does not show how people entered the study or who was lost before measurement. A very large group of volunteers can still differ systematically from eligible nonparticipants. First identify the target and eligibility rules, then examine recruitment and retention. A large sample may make estimates more precise for the analyzed group, but it does not by itself remove selection concerns or justify broader generalization.
Check Your Understanding
Answer each question by identifying the relevant stage and denominator.
- A study has 250 eligible people, invites 200, enrolls 120, and measures 90 at follow-up. Calculate the invitation, participation, and retention rates.
- Why does a high participation rate not, by itself, establish that the analyzed sample represents the intended population?
- In a simple regression with a binary predictor, explain why changing one predictor group’s mean outcome among selected cases can change the fitted slope.
- A study’s participants have a different predictor distribution from the eligible population. What additional issue must be considered before concluding that the regression relationship is distorted?
- Why can researchers usually not compare follow-up outcomes for retained participants with those lost to follow-up using observed outcome data alone?