Why Study Inclusion Matters
A causal comparison can be distorted not only by how treatment is assigned, but also by who is included in the analysis. Participants may be recruited, retained, or have their outcomes recorded at different rates depending on treatment and on factors related to the outcome. If the analysis is restricted to those included, the treated and control groups can become systematically different even when treatment was initially randomized.
This is a distinct problem from unmeasured confounding. In the example below, treatment is randomized and has no effect on the outcome in the full population. The bias arises because inclusion is affected by both treatment and outcome. Adjusting for ordinary pretreatment covariates does not necessarily undo the association created by restricting the analysis.
Use \(T\in\{0,1\}\) for treatment assignment, \(Y\) for the observed outcome, and \(Y(1)\) and \(Y(0)\) for the potential outcomes under treatment and control. The full-population average causal effect is \(E[Y(1)-Y(0)]\). In a randomized study, assignment is independent of the potential outcomes. But that fact alone does not guarantee that the same independence holds after restricting the analysis to \(S=1\).
How Conditioning on Selection Creates an Association
Suppose treatment can affect inclusion, and outcome status can also affect inclusion. The structure can be represented as \(T\rightarrow S\leftarrow Y\). Here, \(S\) is a common effect of treatment and outcome. Before conditioning on \(S\), randomized treatment may be independent of the outcome. After restricting to \(S=1\), knowing that a person was included can provide information about treatment status and outcome status: among included people, one factor can partly explain why the other person was included. This can induce an association between \(T\) and \(Y\), even without a causal treatment effect.
The relevant condition is not merely that the selected sample is smaller or differs demographically from the source population. The concern is that the selection process changes the relationship being used to estimate an effect. A selection variable may be a direct consequence of treatment and outcome, or a consequence of other variables related to them. Either way, restricting to selected units can open a noncausal path.
Worked Example: A Spurious Effect in a Randomized Study
Consider a hypothetical randomized study with a binary outcome. Half the participants are assigned treatment and half control. Treatment has no effect: in each group, the outcome is positive for 50% of participants. Inclusion depends on both treatment and outcome, as shown by these probabilities:
| Assignment | Outcome | Probability of inclusion |
|---|---|---|
| Treatment | Positive | 0.90 |
| Treatment | Negative | 0.30 |
| Control | Positive | 0.30 |
| Control | Negative | 0.90 |
In the full population, the treatment risk is \(0.50\) and the control risk is \(0.50\). The causal risk difference is therefore \(0.50-0.50=0\).
Among selected treated participants, the probability of a positive outcome is proportional to the original outcome probability times its inclusion probability. Thus,
Among selected controls, the corresponding probability is
The selected-sample risk difference is \(0.75-0.25=0.50\), or 50 percentage points, although the true effect in the full population is zero. Randomization balanced the groups before selection; conditioning on inclusion did not preserve that balance.
Why Ordinary Covariate Adjustment May Not Help
Adjustment works only to the extent that the variables included in the adjustment address the relevant source of bias. In the example, suppose \(X\) is a measured pretreatment characteristic, such as age group, and the same selection pattern applies within each level of \(X\). Then the selected treated risk is 0.75 within each stratum of \(X\), and the selected control risk is 0.25 within each stratum. A standardized contrast that averages the stratum-specific differences over any common distribution of \(X\) is still 0.50.
The reason is structural: \(X\) does not block the path \(T\rightarrow S\leftarrow Y\). Including \(X\) in a regression or stratified analysis cannot, by itself, undo the dependence created by conditioning on \(S\). This is not an argument against adjustment in general. It is a reminder that covariate adjustment is not a universal correction for every form of bias.
Worked Example: Adjustment Within Baseline Covariate Strata
Suppose the randomized population has two equally common age groups, \(X=A\) and \(X=B\). Within both groups, the outcome risk is 0.50 under each assignment, and the selection probabilities are exactly those in the preceding example. The selected risks are therefore:
| Age group | Selected treatment risk | Selected control risk | Risk difference |
|---|---|---|---|
| A | 0.75 | 0.25 | 0.50 |
| B | 0.75 | 0.25 | 0.50 |
Standardizing to equal weights for the two age groups gives
The adjusted selected-sample contrast remains 50 percentage points. In this constructed example, age is measured and balanced in the target population, but it does not account for the selection mechanism. Adjusting for age is therefore not a remedy for this bias.
The result does not mean adjustment can never help when selection is present. If selection depends on measured pretreatment variables and, conditional on those variables, does not further depend on potential outcomes, then appropriate adjustment or weighting may address the resulting differences. But that is a substantive assumption about the selection process, not a consequence of merely adding covariates to a model.
A Selection-Weighting Illustration
When inclusion probabilities are known or can be credibly estimated, inverse-probability weighting can sometimes reconstruct the distribution that selection distorted. For a selected unit with treatment \(T\) and observed outcome \(Y\), the basic selection weight is the reciprocal of its inclusion probability, \(1/P(S=1\mid T,Y,X)\). To justify this approach, selection probabilities must be correctly specified, positive for the relevant groups, and the variables used to model selection must be observed for selected units. Recovering a target-population contrast also requires that the selection model capture the variables through which inclusion relates to the outcome.
Worked Example: Correcting the Selected Risks With Known Probabilities
Return to the hypothetical study. For selected treated participants, the observed positive-outcome fraction is 0.75 and the negative-outcome fraction is 0.25. The inclusion probabilities are 0.90 for positive outcomes and 0.30 for negative outcomes. Weighting each selected outcome category by the reciprocal of its inclusion probability gives the reconstructed positive-outcome risk:
For controls, the selected positive-outcome fraction is 0.25 and the negative-outcome fraction is 0.75. The corresponding calculation is
The weighted risk difference is \(0.50-0.50=0\), matching the full-population effect in this example. The calculation relies on the known selection probabilities for each treatment-outcome group. In real applications, the probability of inclusion may not be known, and when outcomes are missing for nonselected participants, the selection model generally cannot be learned from selected data alone without additional information or assumptions.
Diagnostics, Alternatives, and Limitations
Selection bias is difficult to diagnose from the analyzed sample alone: the people omitted from the analysis may have outcomes that are unavailable. Still, the study process can provide warning signs. Ask whether inclusion, follow-up, or outcome recording could be affected by treatment assignment, treatment receipt, symptoms, prognosis, or the outcome itself. Compare inclusion rates across treatment groups and measured baseline characteristics, and inspect whether reasons for noninclusion differ by group. Similar observed rates do not prove the absence of selection bias, since the selection process may depend on unmeasured or unrecorded factors.
The main alternatives address different mechanisms. Retaining complete follow-up or improving outcome ascertainment can prevent some selection problems at the design stage. Covariate adjustment can address selection related to measured variables when the relevant conditional assumptions hold. Selection weighting can reweight observed participants to represent a specified target population if inclusion probabilities are available and the required positivity and modeling assumptions are credible. Sensitivity analysis can show how conclusions change under plausible departures from those assumptions. None of these approaches guarantees correction when important determinants of selection and outcome are unavailable.
| Approach | What it can address | Key limitation |
|---|---|---|
| Improve follow-up and outcome collection | Prevents some outcome-dependent loss before analysis | May not be feasible; does not remove all selection that has already occurred |
| Adjust for measured covariates | Differences explained by measured variables under appropriate assumptions | Cannot automatically close a path created by conditioning on selection |
| Weight by inverse inclusion probability | Selection represented by a correct, sufficiently rich selection model | Needs positivity, correct probabilities, and adequate information about selection |
| Sensitivity analysis | Shows dependence of conclusions on unverified assumptions | Does not identify the bias from data alone |
Selection bias also limits interpretation beyond the numerical contrast. A selected-sample association is not automatically a causal effect for selected people, because selection may depend on treatment and potential outcomes. Nor should a corrected estimate be treated as reliable merely because a weighting procedure was used. Its interpretation depends on the target population and the assumptions that connect the observed selected sample to that population.
Specify the source population, the inclusion rule, and the point in the study at which selection occurs.
Ask whether treatment, outcome, or their causes affect selection; distinguish pretreatment variables from post-assignment variables.
For adjustment or weighting, state which measured variables are sufficient to account for selection and whether all relevant groups have positive inclusion probability.
Report the population represented by the estimate and discuss how unmeasured selection factors could change it.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- If \(T\) is randomized but inclusion is affected by \(T\) and \(Y\), why might the selected treated and control groups differ in outcome risk?
- In a hypothetical study, the selected treated risk is 0.60 and the selected control risk is 0.40. What is the selected-sample risk difference, and does that alone establish a causal effect?
- Suppose a measured baseline covariate has the same selected-sample distribution in both treatment groups. Does that establish that selection bias is absent? Why or why not?
- In the weighting example, what is the selected treated positive-outcome fraction after inverse-probability weighting, and what is the weighted risk?
Solutions. (1) Conditioning on inclusion, a common effect of treatment and outcome, can induce an association between them. (2) The difference is \(0.60-0.40=0.20\), or 20 percentage points; it is not by itself a causal effect because selection may have distorted the comparison. (3) No. Balance on one measured covariate does not rule out selection depending on outcome or other variables, measured or unmeasured. (4) The weighted positive contribution is \((0.75)(1/0.90)=0.833333\ldots\), and the weighted negative contribution is \((0.25)(1/0.30)=0.833333\ldots\). The weighted risk is \(0.833333\ldots/(0.833333\ldots+0.833333\ldots)=0.50\).