Why the Analysis Sample Matters
A regression model describes an association among the observations included in the analysis. That association need not match the association in the population researchers meant to study. People may be recruited, enrolled, retained, or included in the final analysis according to factors related to both an exposure and an outcome. Restricting the analysis to those selected can then create an association that was absent in the source population, distort an existing association, or even reverse its direction.
This is selection bias. The word “bias” here refers to a systematic difference between the association estimated in the selected sample and the association targeted in the source population; it does not simply mean that a sample is small or unrepresentative in some general sense. The key question is how inclusion in the analysis depends on the variables and causes relevant to the association.
A directed acyclic graph makes the mechanism visible. For example, if the exposure affects selection and an outcome-related factor also affects selection, the structure is \(X \rightarrow S \leftarrow U \rightarrow Y\). Here \(U\) is a factor that predicts the outcome. In the source population, the path from \(X\) to \(Y\) through \(S\) is blocked at the common effect \(S\). Restricting the analysis to \(S=1\) conditions on that common effect and can open the path. The resulting association between \(X\) and \(Y\) through \(U\) is not necessarily a causal effect of \(X\).
The same logic applies when the outcome itself helps determine selection, as in recruitment of people with symptoms as well as people with a particular exposure. A selection diagram describes a possible mechanism; it does not by itself establish that a particular study has selection bias. The investigator must use knowledge of recruitment, follow-up, and the target population to assess whether the diagram is plausible.
How Conditioning on Selection Changes an Association
The basic arithmetic follows from conditional probability. For binary \(X\) and \(Y\), the probability of the outcome among selected people with exposure value \(x\) is
The numerator weights the source-population probability of \(Y=1\) by the probability of being selected in that exposure–outcome group. The denominator accounts for selection among both outcome groups. If selection probabilities differ by outcome and exposure, the selected-sample risk can differ from the source-population risk. In particular, selection can make outcome status informative about exposure status among those included, even if the two were unrelated before selection.
This is a sampling mechanism, not a special property of a particular regression equation. A linear, logistic, or other regression fitted to \(S=1\) data summarizes an association in that selected data, subject to the model and its assumptions. Adding covariates does not automatically undo the way participants came to be selected.
Worked Example: Selection Creates an Association From None
Imagine a hypothetical source population in which exposure \(X\) and outcome \(Y\) are independent binary variables, each with probability \(0.5\) of being 1. A study recruits a person if the person is exposed or has the outcome: \(S=1\) whenever \(X=1\) or \(Y=1\). This idealized rule represents, for example, recruitment through either an exposure-focused program or a symptom-based service.
In the source population, independence gives probability \(0.25\) to each of the four \(X,Y\) combinations. The source-population outcome risk is \(0.5\) in both exposure groups, so the risk difference is \(0.5-0.5=0\).
The rule excludes only the \((X=0,Y=0)\) group. The selected population therefore has total probability \(0.75\). For \(X=0\), selected people must have \(Y=1\), so
For \(X=1\), both outcome values are selected and each had source-population probability \(0.25\). Thus
The selected-sample risk difference, exposed minus unexposed, is \(0.5-1=-0.5\), despite a source-population risk difference of zero. Conditioning on recruitment has created a negative association. This calculation does not show an effect of exposure; the exposure and outcome were independent in the source population by construction.
Follow-Up Can Select Participants Too
Selection is not confined to initial recruitment. In a longitudinal study, the analyzed sample may include only participants who complete follow-up or have an outcome recorded. If continued participation depends on exposure and on a factor related to the outcome, the retained participants can form a selected sample. The relevant indicator \(S\) then means “has usable follow-up data,” rather than “was originally recruited.”
Worked Example: Outcome-Related Retention Distorts Follow-Up Results
Consider a hypothetical cohort in which exposure \(X\) and a baseline risk factor \(U\) are independent, each with probability \(0.5\) of being present. Let the follow-up outcome be \(Y=U\), so the risk factor determines the outcome in this simplified example. Suppose every exposed participant is retained, while an unexposed participant is retained only if \(U=1\). This is the rule \(S=1\) when \(X=1\) or \(U=1\).
In the source population, \(P(Y=1\mid X=0)=P(Y=1\mid X=1)=0.5\), because \(X\) and \(U\) are independent and \(Y=U\). The source-population risk difference is zero.
Among retained unexposed participants, the rule requires \(U=1\), and therefore \(Y=1\). Their outcome risk is \(1\). Among retained exposed participants, both values of \(U\) remain possible in equal proportions: the \(X=1,U=0\) and \(X=1,U=1\) groups each had probability \(0.25\), and both are retained. Their outcome risk is
The observed risk difference is \(0.5-1=-0.5\), rather than zero. The mechanism is that retention \(S\) is a common effect of \(X\) and \(U\), while \(U\) predicts \(Y\). The numerical rule is deliberately simple; real follow-up may depend probabilistically on exposure, health, access, or other factors rather than following an all-or-none rule.
Selection Can Reverse an Existing Association
Selection bias is not limited to inventing an association from nothing. When an exposure is associated with an outcome in the source population, selection can alter the size of that association or change its direction. The next example makes this possibility explicit.
Worked Example: Recruitment Reverses a Positive Risk Difference
Suppose a hypothetical source population contains 1,000 people, half unexposed and half exposed. The outcome risk is \(0.2\) among unexposed people and \(0.6\) among exposed people. The corresponding expected cell counts are:
| Exposure | Outcome \(Y=1\) | Outcome \(Y=0\) | Total |
|---|---|---|---|
| \(X=0\) | 100 | 400 | 500 |
| \(X=1\) | 300 | 200 | 500 |
In the source population, the risk difference is \(0.6-0.2=0.4\), or 40 percentage points. Now suppose recruitment includes anyone who is exposed or has the outcome. In this simplified example, the only excluded people are unexposed people without the outcome: the 400 people in the \(X=0,Y=0\) cell.
Among selected unexposed people, the 100 included people all have the outcome, so their risk is \(100/100=1\). Among selected exposed people, all 500 are included, including 300 with the outcome, so their risk is \(300/500=0.6\). The selected-sample risk difference is therefore
The direction has reversed: the source-population risk difference is \(+0.4\), while the selected-sample difference is \(-0.4\). This is an arithmetic illustration of what can happen when selection depends on exposure and outcome status. A real study’s recruitment mechanism and its target population must be assessed on their own merits; this example is not evidence that any particular study has such a mechanism.
Reasoning Carefully About a Restricted Sample
Before interpreting an analysis based on a restricted sample, distinguish the population of analysis from the target population. If the scientific question concerns only the people who meet a well-defined eligibility condition, an association within that group may be the intended target. If the question concerns a broader population, recruitment or retention may have selected a subset whose association differs from the one of interest.
A useful assessment follows the selection process from the start to the final analysis. Ask which people could enter, which could remain under observation, and which had complete measurements. Then ask whether exposure, outcome, or their causes affected any of those steps. The final restriction can occur through several stages, not just an explicit eligibility rule.
State the population and exposure–outcome association the analysis is meant to describe.
Define what \(S=1\) means and identify factors that affect recruitment, retention, or availability of outcome data.
Use a causal diagram and subject-matter knowledge to check whether selection is a common effect of exposure-related and outcome-related factors.
Describe the association as applying to the analyzed sample unless the design and analysis justify a broader target.
A restriction does not guarantee selection bias. If selection is unrelated to the relevant variables, or if the causal and probability structure happens not to distort the association, the selected and source associations may agree. Conversely, the direction and size of distortion cannot generally be determined from the phrase “selection bias” alone. They depend on how selection probabilities vary across exposure and outcome groups and on the causal structure.
Nor does the word “selection” mean that every analysis should try to adjust for \(S\). Within a dataset restricted to \(S=1\), the selection indicator is constant and cannot be included as an ordinary varying predictor. More importantly, recognizing the problem requires defining the target and selection mechanism; a regression coefficient from the selected data does not by itself establish that the source-population association has been recovered.
Common Mistake
- Wrong reasoning: “The sample is large and I adjusted for several covariates, so the association must represent the broader population.”
- Correction: Large sample size can make a selected-sample estimate more precise without making it representative of the target association. Adjustment is not a general remedy for conditioning on a common effect. First explain how participants were selected and whether that mechanism could connect exposure-related and outcome-related factors.
Check Your Understanding
Use the selection mechanism and target population to guide each answer.
- In the first worked example, why is the source-population risk difference zero, and why is the selected-sample risk difference negative?
- A cohort retains participants based on both exposure and a baseline factor that predicts the outcome. Draw or describe a directed acyclic graph showing a possible selection-bias path.
- In the recruitment example with 1,000 people, calculate the source-population and selected-sample risk differences. What changes in direction?
- Does a large sample necessarily remove selection bias? Explain the difference between precision and distortion of the target association.
- What information about recruitment and follow-up would you seek before deciding whether a restricted-sample association applies to a broader target population?