Choosing an Attrition Adjustment From Panel Diagnostics

Why Attrition Adjustment Is a Diagnostic Choice

Panel attrition occurs when some units stop providing outcome data in later waves. The resulting data are unbalanced: each unit may contribute a different number of observations. The practical question is not simply whether to calculate weights. It is whether the observed pattern of loss, the available predictors of retention, and the behavior of the resulting weights support a particular adjustment for the estimand of interest.

A retention model can adjust for differences in observed characteristics between retained and lost units, but it cannot establish that unobserved outcome differences are irrelevant to attrition. Nor does a large weighted estimate automatically correct selection. A defensible analysis therefore combines a clear description of observation patterns, an explicit retention model, weight diagnostics, and sensitivity checks. The choice should be reported as conditional on assumptions, not as a mechanical consequence of fitting a model.

Start With Observation Patterns

An observation-pattern table records how many units follow each response history. For a three-wave panel, a pattern such as \(110\) means that outcomes are observed in waves 1 and 2 but not wave 3. The table makes the timing and extent of attrition visible. It also helps distinguish monotone dropout from intermittent missingness, in which a unit misses a wave and later returns.

Pattern counts should be accompanied by comparisons of baseline covariates and, where available, earlier outcomes across eventual response patterns. Such comparisons can reveal which observed characteristics predict loss and inform a retention model. They are diagnostic, not a test that proves missingness is ignorable: similarities in observed data cannot rule out selection on unobserved outcomes.

Retention Models and Inverse-Probability Weights

For an analysis of the final-wave mean, define the retention probability \(\pi_i=P(R_{iT}=1\mid X_i)\), where \(T\) is the final wave. A response model estimates this probability from observed predictors. Under a missing-at-random condition for the target outcome, \(R_{iT}\) is conditionally independent of \(Y_{iT}\) given the modeled observed information. Positivity is also required: units in target-relevant covariate groups must have a nonzero probability of remaining observed.

The inverse-probability weight for a retained unit is \(w_i=1/\hat{\pi}_i\). A unit with a low modeled probability of retention represents more baseline units than one with a high probability. A commonly used normalized weighted mean is

For monotone attrition across multiple waves, the probability of remaining observed through wave \(T\) can be modeled as a product of conditional retention probabilities at each wave, given the observed history used in the model. The corresponding inverse weight is the inverse of that product. This construction makes the conditioning history and the assumptions at each stage important; using only a single baseline model may be inadequate when retention depends on evolving observed information.

Inspect Weight Behavior Before Choosing an Adjustment

A retention model can generate very large weights when fitted retention probabilities are near zero. Such weights can make a few observed outcomes dominate the estimate and can inflate sampling variability. Review the distribution of fitted probabilities and weights, their maximum and upper quantiles, and which units carry the largest values. Also examine whether extreme weights correspond to sparse covariate combinations, model misspecification, or a real lack of overlap.

One compact summary is the effective sample size (ESS), defined for positive weights as

ESS equals the actual number of observed units when all weights are equal, and falls as weight concentration increases. It is a diagnostic summary, not a replacement for the sample size or a guarantee of valid inference. A low ESS signals that precision and sensitivity to individual observations deserve particular attention.

Choose an Adjustment and Check Its Sensitivity

The adjustment should follow the target estimand and the evidence from the diagnostics. If attrition is limited and observed response patterns are reasonably comparable on relevant measured variables, an unweighted analysis may be a useful reference, but it still relies on a defensible account of missing outcomes. If response varies substantially with observed covariates and estimated probabilities are not too small, inverse-probability weighting may be suitable under conditional missing at random and positivity. If weights are unstable, reconsider the response model, the information used to predict retention, and whether the target can be supported by the observed overlap.

An outcome-regression adjustment can be considered as a complementary approach: model the outcome using observed predictors and use that model to account for missing outcomes. Agreement between differently structured approaches is informative, but does not prove their assumptions. The substantive question is whether the variables and model represent the relevant selection process, not which method produces the more appealing estimate.

Sensitivity analysis is essential because the observed data alone generally cannot distinguish conditional missing at random from selection depending on unobserved outcomes. Specify a meaningful range of departures, calculate how the target estimate changes, and report whether substantive conclusions change. Earlier material on selection models, attrition sensitivity analysis, and attrition bounds provides formal approaches; diagnostics here guide which assumptions and alternatives deserve attention.

Why the Choice Matters

The choice of adjustment affects both the point estimate and its uncertainty. Complete-case analysis targets the retained sample unless additional assumptions justify extending its results to the baseline cohort. Inverse-probability weighting attempts to restore representation according to modeled retention probabilities, but can become unstable when some types of units rarely remain observed. A small difference between weighted and unweighted estimates is not proof that attrition is harmless; a large difference is a reason to understand which units and assumptions drive the change.

A report should name the estimand, observation patterns, response model, weight construction, diagnostics, and sensitivity checks. It should explain why the adjustment is plausible and identify what cannot be verified from observed data. Inference should reflect the panel structure and uncertainty from estimating retention probabilities, rather than treating fitted weights as known constants when that would understate uncertainty.