Why Retention Can Change a Treatment Comparison
A treatment comparison at a follow-up visit is often calculated using the biomarker values that were actually recorded. But if participants with different health trajectories are retained at different rates, the observed values may represent different subsets of the treatment groups. The resulting contrast can be smaller than, larger than, or even opposite in direction to the contrast for the groups originally assigned to treatment.
This is not simply a loss of precision from having fewer observations. When the chance of remaining observed is related to health or to the biomarker itself, the retained participants can differ systematically from those who dropped out. If that selection also differs between treatment groups, comparing observed means can be especially misleading.
Here \(Y^*\) denotes the outcome that would be recorded at the chosen time if it were available. This framing requires the biomarker to be defined for everyone in the target population at that time. If a terminal event makes the biomarker undefined, that is not automatically ordinary dropout; the target question must address the event explicitly. The earlier material on dropout, right censoring, and terminal events distinguishes these situations.
Which Contrast Is Being Estimated?
Suppose treatment is randomized and the scientific question concerns the mean biomarker at a fixed visit for everyone assigned to each group. The target contrast is the difference between the treatment-group means in the randomized populations, \(E(Y^*\mid A=1)-E(Y^*\mid A=0)\). Under randomization, consistency, and no interference, this contrast can be interpreted as an average causal effect of assignment if the endpoint is well defined for all participants.
A complete-case comparison instead uses only participants with \(R=1\): \(E(Y^*\mid A=1,R=1)-E(Y^*\mid A=0,R=1)\). It compares the groups that happened to remain observed, not necessarily the groups originally randomized. Randomization balances treatment groups at baseline on average; it does not ensure that the retained subsets remain comparable after dropout.
A key condition for using the observed mean in arm \(a\) as the target mean is that retention does not select on the outcome within that arm, at least after conditioning on variables used in the analysis. For example, if retention is independent of \(Y^*\) within each treatment group, then \(E(Y^*\mid A=a,R=1)=E(Y^*\mid A=a)\). If instead retention depends on observed baseline health, this equality may hold within baseline-health strata but fail without conditioning.
How Selection Shifts an Observed Mean
A simple identity makes the source of the shift explicit. Within treatment group \(a\), suppose \(R\) is binary and the outcome \(Y^*\) is defined for everyone. The observed mean is \(E(Y^*\mid A=a,R=1)\), while the target mean is \(E(Y^*\mid A=a)\).
The second line follows because \(Y^*R=Y^*\) for retained participants and is zero otherwise, so \(E(Y^*R\mid A=a)=P(R=1\mid A=a)E(Y^*\mid A=a,R=1)\). Dividing the covariance identity by the retention probability gives the final line. Thus, when higher biomarker values are associated with lower retention, the covariance is negative and the observed mean is below the target mean. A treatment contrast based on retained participants includes the selection shift in each arm; the two shifts need not cancel.
Worked Example: Dropout Reverses the Observed Contrast
Consider a hypothetical randomized study in which higher biomarker values indicate worse health. At the fixed follow-up time, 60 of 100 participants assigned to treatment are retained. Their mean is 8.0 units. The other 40 have a hypothetical mean of 11.75 units at that time. In the control group, 80 of 100 are retained with a mean of 8.5 units; the other 20 have a hypothetical mean of 11.0 units. Assume these values are defined for everyone, including those who dropped out.
Target treatment-group means. For treatment, the full-group mean is \((60\times8.0+40\times11.75)/100=(480+470)/100=9.5\) units. For control, it is \((80\times8.5+20\times11.0)/100=(680+220)/100=9.0\) units. The target treatment-minus-control contrast is \(9.5-9.0=0.5\) units, indicating a higher, worse mean in the treatment group.
Observed means. Among retained participants, the treatment mean is 8.0 units and the control mean is 8.5 units. The observed contrast is \(8.0-8.5=-0.5\) units, pointing in the opposite direction. In both arms, those who dropped out had worse mean outcomes than those retained, so retention is health-related. The different retention proportions (60% versus 80%) and outcome distributions yield selection shifts that do not cancel.
The hypothetical dropout outcomes show what the target means would be if the endpoint were available for everyone; in an actual study those outcomes are not observed. The example therefore illustrates the potential distortion, not a direct recipe for recovering missing outcomes.
A Difference Can Appear Even With No Treatment Effect
Informative retention can also create an apparent treatment difference when the full randomized groups have the same outcome distribution. Consider two baseline health strata, each with 50 participants per treatment group. Suppose the mean follow-up biomarker is 6 units in the healthier stratum and 10 units in the more vulnerable stratum in both groups. The full-group mean in each arm is therefore \((50\times6+50\times10)/100=8\) units.
Worked Example: A Spurious Difference From Changing Group Composition
Suppose 40 healthier and 10 more vulnerable treatment participants are retained. Their observed mean is \((40\times6+10\times10)/50=(240+100)/50=6.8\) units. In the control group, 30 healthier and 20 more vulnerable participants are retained, giving \((30\times6+20\times10)/50=(180+200)/50=7.6\) units.
The observed treatment-minus-control contrast is \(6.8-7.6=-0.8\) units, even though the full-group means are both 8 units and the treatment effect in this example is zero. In each arm the vulnerable stratum has a lower retention proportion than the healthier stratum: 10/50 versus 40/50 in treatment, and 20/50 versus 30/50 in control. But the degree of selection differs by arm, changing the mix of health strata among those observed.
This calculation depends on the stated stratum-specific outcomes and retention counts. In real data, the relevant strata may not be fully measured, and the observed stratum-specific means may not represent outcomes among dropouts.
Adjustment Requires Assumptions
One possible adjustment uses inverse-probability weighting. It can recover a population mean when retention is conditionally independent of the outcome given measured information, the retention probabilities are correctly estimated, and every relevant subgroup has a positive chance of being retained. The first condition is often called missing at random conditional on the measured variables. It is an assumption, not something that can generally be verified from observed outcomes alone.
Worked Example: Weighting for Measured Baseline Health
Use the two health strata from the previous example. For this calculation, assume the stratum-specific follow-up mean is 6 units for healthier participants and 10 units for more vulnerable participants, including those who dropped out. Assume retention is independent of the follow-up outcome within each stratum. In the treatment group, 40 of 50 healthier participants and 10 of 50 vulnerable participants are retained; in control, 25 of 50 are retained in each stratum.
The treatment retention probabilities are \(40/50=0.8\) and \(10/50=0.2\). Weight each retained participant by the inverse of their stratum-specific retention probability. The weighted treatment mean is \[ \frac{(40/0.8)\times6+(10/0.2)\times10}{40/0.8+10/0.2} =\frac{50\times6+50\times10}{100} =8 \] units. The unweighted treatment mean was 6.8 units.
For control, each stratum has retention probability \(25/50=0.5\), so the weighted mean is \[ \frac{(25/0.5)\times6+(25/0.5)\times10}{25/0.5+25/0.5} =\frac{50\times6+50\times10}{100} =8 \] units. Under the stated assumptions, weighting gives a treatment-minus-control contrast of \(8-8=0\) units, matching the full-group contrast. In practice, the weights and uncertainty must be estimated from data; very small retention probabilities can produce highly variable estimates.
Diagnostics, Alternatives, and Limits
Begin by reporting retention by treatment group and examining whether retention varies with baseline health, prior biomarker values, and other recorded predictors. Compare the timing and stated reasons for dropout across groups. These checks can reveal patterns that make an unadjusted comparison questionable, but they cannot establish that dropout is noninformative: outcomes after dropout are unobserved, so dependence on those unobserved values cannot be ruled out by a test of observed data.
The main approaches answer different questions and rely on different assumptions:
| Approach | What it uses | Key limitation |
|---|---|---|
| Complete-case comparison | Observed outcomes among retained participants | Generally describes retained groups; it targets the randomized groups only under suitable nonselective-retention conditions. |
| Likelihood-based longitudinal analysis under missing at random | Observed longitudinal outcomes and their modeled distribution | Requires a suitable outcome model and conditional independence of dropout and unobserved outcomes given included information. |
| Inverse-probability weighting | Retained outcomes weighted by estimated retention probabilities | Requires adequate predictors, correct probability estimation, and positivity; unstable weights can inflate uncertainty. |
| Joint modeling of outcomes and dropout | A model linking the longitudinal outcome process with the dropout process | Requires specifying and checking both processes; conclusions can depend on modeling assumptions not identified by observed data alone. |
A practical warning is that adjustment cannot repair information that was never measured without additional assumptions. If unobserved current health affects both dropout and the missing biomarker after conditioning on all recorded variables, a missing-at-random analysis may still be biased. Sensitivity analyses can examine how conclusions change under plausible departures from that assumption; they do not make the unobserved outcomes known.
Common mistake: “The trial was randomized, so comparing the observed means preserves the randomized comparison.” Randomization applies to assignment at baseline, not necessarily to the selected subsets with observed follow-up values. The correction is to state the target estimand, report retention by arm, and justify why the chosen analysis recovers that estimand under its assumptions. Also avoid treating death or another terminal event as routine missingness when the biomarker is not defined after that event.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If higher biomarker values are associated with lower retention within an arm, what is the sign of the covariance between the biomarker and the retention indicator, and how does the observed mean compare with the full-group mean?
- In the dropout-reversal example, calculate the retention percentages and the observed treatment-minus-control contrast.
- Why does baseline randomization alone not guarantee that the retained treatment and control groups are comparable?
- What assumption allows the weighting example to use retained participants to represent all participants within each health stratum?
- If no vulnerable participants are retained in one treatment group, can inverse-probability weighting recover their mean outcome from the observed data? Why or why not?
Solutions. (1) The covariance is negative. By the selection identity, the observed mean is below the full-group mean. (2) Retention is \(60/100=60\%\) for treatment and \(80/100=80\%\) for control; the observed contrast is \(8.0-8.5=-0.5\) units. (3) Dropout occurs after randomization and can select participants differently according to health or outcome, so the retained subsets need not preserve baseline balance. (4) Retention must be independent of the follow-up outcome conditional on the measured health stratum, with correctly estimated retention probabilities and positive retention probability in each stratum. (5) No. The retention probability is zero, violating positivity; there are no observed vulnerable participants to represent that subgroup by weighting.