Diagnosing and Sensitivity-Analyzing Differential Attrition

Why Differential Attrition Needs Diagnosis

Random assignment protects comparisons at the point of assignment, but it does not ensure that outcomes will be observed for everyone. If follow-up is less complete in one group, or if observation depends on participants’ baseline characteristics differently across groups, the observed-outcome comparison can be misleading. The first task is therefore to describe who has an outcome recorded; the next is to show how conclusions depend on plausible values for outcomes that are missing.

An unequal observation rate is a warning sign, not a verdict. Equal rates do not guarantee that the observed participants are comparable to those with missing outcomes, and unequal rates do not by themselves establish the direction or size of bias. The missing outcomes cannot be diagnosed from the observed data alone. Sensitivity analysis makes that limitation explicit by showing what follows under specified assumptions.

For causal interpretation, the usual randomized-experiment conditions include genuine random assignment, consistency between assigned treatment and the corresponding potential outcome, and no interference between individuals. The finite-population average treatment effect is

This target averages both potential outcomes over the full randomized population. Attrition does not change its definition; it makes some outcomes needed to estimate it unavailable. In the absence of attrition, the difference in assigned-group means is a randomization-based estimator of \(\tau\). With missing outcomes, the observed-only difference is generally not that complete-data estimator. Throughout this tutorial, sensitivity calculations for group means are presented as completed assigned-group contrasts and thus as estimators of \(\tau\), not as direct identification of \(\tau\).

Compare Observation Rates First

For assigned group \(z\), let \(n_z\) be the number assigned, \(m_z\) the number with observed outcomes, and \(q_z=m_z/n_z\) the observation rate. Report the group denominators and rates, their difference in percentage points, and the missing counts. A rate difference is easy to interpret, while the counts reveal how much information is absent.

The rate difference is descriptive. It is not automatically a test of a causal effect or a test of the missingness mechanism. Randomization can support a formal comparison of rates if that is the question, but a non-significant comparison does not prove equal rates or rule out selective missingness. For a sensitivity analysis, the size and pattern of the missingness matter even when a conventional test is not significant.

Use Baseline Covariates to Locate the Pattern

When baseline covariates are recorded for all randomized participants, compare observation patterns within covariate groups as well as overall. For a categorical covariate, show the number assigned and the number observed in each category and arm. For a continuous covariate, useful summaries can include observation rates across pre-specified ranges and the covariate distributions among those with and without outcomes. Covariates should be measured before assignment; post-assignment variables may themselves be affected by treatment and should not be treated as baseline diagnostics.

These comparisons can reveal, for example, that the overall rate difference is concentrated among participants with high baseline risk. They can also reveal that the observed participants differ from the full assigned group in ways that are relevant to the outcome. Such patterns help motivate sensitivity assumptions or follow-up efforts; they do not establish that missingness is ignorable after adjustment. A regression of \(R_i\) on assignment and baseline covariates can summarize conditional associations, but it cannot recover unobserved outcomes or prove a missingness assumption.

A Pattern Is Not a Missingness Assumption

The missing-data terminology helps separate what is observable from what must be assumed. Outcomes are missing completely at random if observation is unrelated to outcomes and covariates; missing at random if, conditional on observed information such as treatment and baseline covariates, observation is unrelated to the unobserved outcome; and missing not at random if observation still depends on the unobserved outcome after conditioning on observed information. The observed data alone generally cannot distinguish the latter two explanations.

In particular, finding that observation rates differ by assignment or baseline risk does not prove missing not at random. Conversely, similar observed rates do not establish missing completely at random. Analyses under a missing-at-random assumption may be useful, but that assumption is not verified just because observed covariates have been included. Sensitivity analysis asks how results vary when the unobserved outcomes depart from a reference assumption.

Sensitivity Analysis With Mean Shifts

A transparent sensitivity model specifies, separately by randomized group, how the mean outcome among participants with missing outcomes differs from the mean among participants with observed outcomes. Let \(\bar Y_{z,\mathrm{obs}}\) be the observed mean in assigned group \(z\), and define \(\delta_z\) as the mean outcome among missing members of that group minus \(\bar Y_{z,\mathrm{obs}}\). The units of \(\delta_z\) are the outcome’s units. A negative value says the missing outcomes are assumed lower, on average, than the observed outcomes in that arm.

For this calculation, assume each arm has at least one observed and one missing outcome, and that the sensitivity parameter is a meaningful difference in the arm’s outcome means. The completed assigned-group mean implied by a chosen \(\delta_z\) is a weighted average of the observed and assumed missing means. This is an arithmetic identity under the chosen mean-shift assumption, not a claim that the assumption is true.

The completed assigned-group contrast, treatment minus control, is therefore

This formula shows why a fixed shift has more influence in a group with a larger missing fraction. It also shows why specifying the direction of shifts in both arms matters: a lower assumed mean among missing treated participants pulls the contrast down, while a lower assumed mean among missing control participants pulls it up.

What the Sensitivity Calculation Does—and Does Not—Establish

The mean-shift analysis is useful because it exposes the assumptions that an observed-only analysis leaves implicit. It can be reported as a table or graph over a range of \(\delta_1\) and \(\delta_0\), with clearly labeled outcome units. A one-dimensional tipping point is convenient when one arm’s shift is held fixed; a two-dimensional display is more honest when plausible departures in both arms are uncertain. If baseline covariates are important predictors of the outcome or observation, shifts can instead be specified within baseline strata and then aggregated using the assigned group’s stratum proportions. That approach requires stating the stratum-specific assumptions and weights.

A completed assigned-group contrast and the finite-population ATE are not interchangeable. The formula imputes outcomes missing for participants in their assigned arm, thereby completing each realized assigned group’s outcome mean. It does not reveal both potential outcomes for every individual, nor does it remove random assignment variation between the realized groups. Under random assignment, such a contrast is used to estimate the ATE; its sampling uncertainty must be addressed separately using an appropriate design-based method. Sensitivity to missing outcomes and uncertainty due to randomization are distinct issues.

If the outcome has known bounds, one can also substitute the lowest and highest permissible values for missing outcomes to find the range of completed assigned-group contrasts consistent with those bounds. Those are bounds on the completed arm contrast under the observed assignment and outcome range. They are not, without further assumptions or a separate derivation, identification bounds for the finite-population ATE. In particular, the unobserved potential outcomes under the assignments individuals did not receive are a separate issue.

Common Mistakes

• Treating a rate difference as proof of bias. Different rates flag a potential problem but do not reveal the missing outcomes’ means. Use explicit assumptions to assess how the estimate could change.
• Calling a non-significant rate comparison evidence of no attrition problem. A test may have limited precision, and equal rates can coexist with different selection patterns. Report the rates and examine baseline patterns.
• Assuming baseline adjustment proves outcomes are missing at random. Adjustment can address observed differences under an assumption; it cannot verify that assumption or rule out dependence on unobserved outcomes.
• Calling a completed-arm contrast an identification bound for the ATE. Completing missing outcomes in the assigned groups does not supply both potential outcomes for every individual. Label the calculation as a completed contrast or estimator, and distinguish it from identification of the finite-population ATE.
• Giving a sensitivity shift without units or direction. State whether missing outcomes are assumed higher or lower than observed outcomes, by how much, and on what measurement scale.