Sensitivity Analysis for Nonignorable Attrition

Why Sensitivity Analysis Is Needed

When people leave a panel, their later outcomes are not observed. If dropout is related to those unobserved outcomes, the outcomes among respondents may not represent the outcomes that would have been observed for everyone. An analysis that assumes otherwise can be precise and still answer the wrong question. Sensitivity analysis makes the untestable part of the missing-data assumption explicit: it asks how estimates change across specified, scientifically interpretable departures from an assumption such as missing at random.

The purpose is not to identify the true unobserved outcomes from the observed data alone. Rather, the analyst varies a sensitivity parameter, recalculates the target estimate under each value, and reports where conclusions change. A useful analysis describes the parameter in the outcome’s units or in an interpretable response-odds scale, and explains which values are considered plausible.

Pattern-Mixture Sensitivity Parameters

Let \(Y\) be an outcome measured at a follow-up wave, and let \(R=1\) indicate that the outcome is observed and \(R=0\) indicate dropout by that wave. For a group \(g\), write \(r_g=P(R=1\mid g)\) for its response fraction. A simple pattern-mixture sensitivity parameter is \( \delta_g=E(Y\mid R=0,g)-E(Y\mid R=1,g) \). Thus, \(\delta_g\) is measured in the same units as \(Y\). A negative value says that, within group \(g\), the mean outcome among dropouts is assumed lower than the mean among respondents; a positive value says it is higher.

In this simple version, the mean outcome for the full group is a mixture of the respondent and dropout means. If the respondent mean is \(\bar{Y}_{1g}\), then the assumed dropout mean is \(\bar{Y}_{1g}+\delta_g\). Therefore the full-group mean under a specified value of \(\delta_g\) is:

This formula assumes that the respondent mean and response fraction are estimated for the target group, and that the specified shift applies to the dropout mean for that group. With covariates, the shift can instead be specified within covariate strata and the resulting means standardized to a defined target population. The unstratified formula is a transparent illustration, not a claim that covariates are unnecessary in an actual analysis.

Contrasts and Tipping Points

Sensitivity analysis is often most informative for a contrast, such as a difference in follow-up means between two groups. Let groups A and B have respondent means \(\bar{Y}_{1A}\) and \(\bar{Y}_{1B}\), and response fractions \(r_A\) and \(r_B\). Define the target contrast as the full-group mean in A minus the full-group mean in B. Applying the mixture formula to each group gives:

The coefficients on the shifts are the dropout fractions. A shift in a group with little dropout has less influence on its full-group mean than the same shift in a group with extensive dropout. The equation also makes clear why a single shared shift is not always appropriate: the two groups may have different dropout processes or different reasons for attrition.

More generally, if the observed respondent contrast is \(d=\bar{Y}_{1A}-\bar{Y}_{1B}\), the zero-contrast tipping line is obtained by solving \(\Delta(\delta_A,\delta_B)=0\):

The condition \(r_B<1\) matters because if everyone in B responds, its dropout shift has no contribution to this full-group mean. A tipping line or table can show many combinations that change the conclusion, while a single tipping value requires fixing the other sensitivity parameters. State the direction of the contrast and the rule defining “tipping”; reaching zero, losing statistical significance, or changing a policy decision are different criteria.

Selection-Model Parameters

A selection model expresses the response process conditional on the outcome. For example, holding covariates \(X\) fixed, one can specify:

Here \(c\) is a reference outcome, \(\alpha(X)\) is the response log-odds at \(Y=c\), and \(\gamma\) is a sensitivity parameter. A 10-unit increase in \(Y\), at fixed \(X\), multiplies the odds of response by \(\exp(\gamma)\). A positive \(\gamma\) means higher outcomes are associated with higher response odds; a negative \(\gamma\) means lower response odds. Since \(Y\) is unobserved for dropouts, the outcome dependence represented by \(\gamma\) is generally not learned from the observed outcomes without additional assumptions or information. The parameter must be fixed over a range, or varied as part of the sensitivity analysis, with other model parameters estimated under each specification.

How to Conduct and Report the Analysis

A useful sensitivity analysis gives readers enough information to understand the assumptions and reproduce the change in the estimate. The analyst should define the target population and outcome, describe the response pattern, specify the sensitivity parameter and its units or scale, and calculate estimates across a substantively defensible range. If multiple groups have separate dropout shifts, report the combinations considered rather than silently imposing equality.

Uncertainty should be described separately from sensitivity to missingness. A confidence interval calculated under one fixed value of \(\delta\) or \(\gamma\) reflects sampling uncertainty conditional on that assumption and the fitted model. It does not account for uncertainty about which sensitivity value is correct. A report can show confidence intervals across a grid of assumptions, but should label them as scenario-specific and should not imply that the grid itself is a probability distribution over the missing-data assumptions.

Why the Analysis Matters—and a Common Pitfall

A single estimate under an assumption of ignorable missingness can conceal how strongly the conclusion depends on that assumption. Sensitivity analysis makes the dependence visible. It is especially informative when the tipping point can be translated into a concrete statement, such as how many points higher or lower the average dropout outcome would need to be for a contrast to reach zero. That statement helps readers judge plausibility; it does not prove that the assumed shift occurs.

• Pitfall: treating the sensitivity parameter as observed or estimated. The missing outcomes are not available to verify the specified dropout shift. Correction: label the shift as an assumption and show how results vary over alternatives.
• Pitfall: interpreting a tipping point without fixing the other parameters. In a two-group analysis, the tipping value for one group depends on the shift assumed for the other. Correction: state the fixed value or report the full tipping line or grid.
• Pitfall: confusing a conditional confidence interval with robustness. An interval that excludes zero under one shift does not show that the result is robust to other shifts. Correction: report both sampling uncertainty and how estimates change across sensitivity scenarios.
• Pitfall: saying that a nonsignificant result proves no effect. A crossing of a significance threshold is not the same as a zero estimate, and neither establishes absence of an effect. Correction: define the tipping rule and interpret the estimate and interval in context.