Stabilized Weights, Positivity, and Attrition Diagnostics

Why Stabilize Attrition Weights?

Inverse-probability weighting addresses panel attrition by giving more influence to observed units whose measured characteristics are associated with a lower probability of remaining observed. The difficulty is that a very small estimated observation probability produces a very large weight. A few such weights can dominate an estimate, making it unstable and highly sensitive to the fitted response model.

Stabilization rescales the weights by multiplying by a marginal observation probability. This can reduce their overall scale and is useful for interpreting and reporting them. It does not remove the relative influence of units with very low response probabilities, and it does not make a lack of covariate overlap disappear. Diagnostics therefore need to examine the probabilities and the resulting weights, not just whether the weights have been stabilized.

Definition: For a target panel cohort, let $R_i=1$ if unit $i$'s outcome at a specified follow-up time is observed and $R_i=0$ otherwise. Let $X_i$ denote observed covariates measured before that outcome, and let $e(X_i)=P(R_i=1\mid X_i)$ be the conditional observation probability. The unstabilized inverse-probability weight for an observed unit is $1/e(X_i)$. A marginally stabilized weight is $$ w_i^{S}=\frac{\pi}{e(X_i)}, \qquad \pi=P(R_i=1), \qquad R_i=1. $$

In practice, both probabilities are usually estimated: $\hat e(X_i)$ comes from a response model, and $\hat\pi$ is often the observed proportion with follow-up outcomes. The estimated stabilized weight for an observed unit is $\hat w_i^{S}=\hat\pi/\hat e(X_i)$. Weights are assigned to respondents because their outcomes are observed; nonrespondents' outcomes do not enter the weighted outcome calculation.

For a normalized weighted mean, the common numerator $\hat\pi$ cancels. Thus the stabilized and unstabilized weights give the same point estimate when applied to the same respondents. Stabilization changes the scale of the weights, not their relative ratios. In other procedures, including some weighted regression specifications, the numerator can affect the estimating equation; the target and the precise weighting convention should then be stated.

Conditions Behind the Weighting Argument

The target here is the mean outcome at one specified follow-up time for the original baseline cohort, including units whose outcomes are missing at that time. The weighting argument requires that response be conditionally independent of the follow-up outcome given the observed covariates, and that every covariate pattern in the target cohort have a positive chance of being observed.

Assumptions: Conditional missing at random requires $R_i$ to be independent of $Y_i$ given $X_i$, where $Y_i$ is the follow-up outcome. Positivity requires $P(R_i=1\mid X_i=x)>0$ for covariate patterns $x$ represented in the target cohort. The response model must also estimate these conditional probabilities adequately. These conditions are substantive and modeling requirements; diagnostics can reveal warning signs but cannot verify them completely.

The basic justification is that, conditional on $X_i$, weighting a respondent by the inverse of the response probability offsets the probability of being observed. Under conditional missing at random,

E\!\left[\frac{R_iY_i}{e(X_i)}\middle|X_i\right] = \frac{E[R_iY_i\mid X_i]}{e(X_i)} = \frac{e(X_i)E[Y_i\mid X_i]}{e(X_i)} = E[Y_i\mid X_i].

Averaging over the target cohort's covariate distribution recovers its mean outcome. With the stabilized numerator, the corresponding weighted expectation is $\pi E[Y_i]$, while the expectation of the stabilized weights is $\pi$. Their ratio therefore still gives the target mean. In a sample, a normalized weighted mean divides the sum of weighted observed outcomes by the sum of weights.

Worked Example: Constructing Stabilized Weights by Risk Group

Suppose a hypothetical baseline cohort has 200 units: 100 in a lower-attrition-risk group and 100 in a higher-risk group. At follow-up, 80 lower-risk units and 40 higher-risk units are observed. Assume a saturated response model estimates the corresponding observation probabilities as $0.80$ and $0.40$.

The estimated marginal observation probability is $$ \hat\pi=\frac{80+40}{200}=0.60. $$

The stabilized weights are $0.60/0.80=0.75$ for each observed lower-risk unit and $0.60/0.40=1.50$ for each observed higher-risk unit. To see the role of the weights, suppose the observed mean follow-up outcome is 10 units in the lower-risk group and 20 units in the higher-risk group. The normalized stabilized weighted mean is $$ \frac{80(0.75)(10)+40(1.50)(20)} {80(0.75)+40(1.50)} = \frac{600+1200}{60+60} = 15.0\text{ outcome units}. $$

The unweighted respondent mean is $\{80(10)+40(20)\}/120=13.33$ outcome units. The weighted mean gives the two groups equal total weight, matching their equal sizes in the baseline cohort. This calculation illustrates the adjustment under the stated response model; it does not establish that conditional missing at random or the model is correct.

Positivity and Estimated Observation Probabilities

Positivity is a property of the target population and response process: each relevant covariate pattern must have a nonzero probability of being observed. In finite samples, the practical concern is often near-positivity, where some estimated probabilities are positive but close to zero. If $\hat e(X_i)$ is small, then $\hat w_i^{S}=\hat\pi/\hat e(X_i)$ is large, so a small number of respondents may carry much of the weighted estimate.

Inspect estimated probabilities across the entire baseline cohort, including respondents and nonrespondents. Useful summaries include the minimum and selected quantiles, and distributions within important covariate groups. A fitted probability of zero or one, predictions near those boundaries, or covariate patterns with no respondents deserve investigation. These signs can indicate sparse data, poor overlap, or a response model that is too restrictive. A model that predicts probabilities comfortably away from zero is not proof of positivity in the underlying population.

Worked Example: Diagnosing an Extreme Weight

In a hypothetical panel, suppose $\hat\pi=0.40$. Four observed units have estimated response probabilities $0.05$, $0.20$, $0.50$, and $0.80$, with follow-up outcomes of 100, 80, 40, and 20 points. Their stabilized weights are, respectively, $$ \frac{0.40}{0.05}=8.00,\qquad \frac{0.40}{0.20}=2.00,\qquad \frac{0.40}{0.50}=0.80,\qquad \frac{0.40}{0.80}=0.50. $$

The normalized weighted mean is $$ \frac{8(100)+2(80)+0.80(40)+0.50(20)} {8+2+0.80+0.50} = \frac{1002}{11.30} = 88.67\text{ points}. $$

The unit with an estimated observation probability of $0.05$ contributes $8(100)=800$ to the numerator, most of its total. This is a clear influence warning: the estimate depends heavily on a unit whose response probability is small. The value $0.05$ does not by itself prove a positivity violation, but it warrants checking the covariate pattern, the model, and the sensitivity of the result.

Weight Summaries, Truncation, and Balance

Report the distribution of estimated weights, not only their average. Relevant summaries include the minimum, median, upper quantiles, maximum, and the share of total weight carried by the largest weights. The effective sample size is another compact diagnostic:

n_{\mathrm{eff}}=\frac{\left(\sum_{i:R_i=1}\hat w_i^{S}\right)^2} {\sum_{i:R_i=1}(\hat w_i^{S})^2}.

This quantity is at most the number of respondents and decreases when weights are concentrated on a few units. It is a diagnostic of weight concentration, not the actual number of independent observations and not a replacement for an appropriate variance calculation.

Truncation, also called weight capping, replaces weights above a chosen threshold with that threshold. For example, a cap of 4 replaces each weight greater than 4 by 4. Compare the primary untruncated estimate with estimates under one or more transparently specified caps. Truncation can reduce the influence of extreme weights and improve precision, but it changes the weighting scheme and can reintroduce bias. It does not repair missing covariate support or validate the response model.

Covariate balance offers a further diagnostic. Compare baseline covariate summaries among respondents before weighting, among respondents after weighting, and in the full baseline cohort, which defines the target covariate distribution. If the response model and weighting work as intended, weighted respondent summaries should generally be closer to the baseline summaries for covariates used in the model. Balance is not a test of conditional missing at random: measured covariates may balance while unmeasured predictors of both response and outcome remain.

Worked Example: Checking Weighted Covariate Balance

Consider a hypothetical baseline cohort of 1,000 units: 600 urban and 400 rural. At follow-up, 480 urban and 160 rural units respond. A saturated model using residence estimates response probabilities of $480/600=0.80$ for urban units and $160/400=0.40$ for rural units. The overall response probability is $640/1000=0.64$.

Among respondents, the unweighted rural proportion is $160/640=0.25$, compared with $400/1000=0.40$ in the target cohort. The stabilized weights are $0.64/0.80=0.80$ for urban respondents and $0.64/0.40=1.60$ for rural respondents. Their weighted rural proportion is $$ \frac{160(1.60)}{480(0.80)+160(1.60)} = \frac{256}{384+256} = 0.40. $$

Thus weighting corrects the rural share from 0.25 to 0.40 in this example, matching the baseline cohort. The check is meaningful because the response model is saturated for the two residence groups. It diagnoses balance on residence only; it cannot establish balance on other covariates or the validity of the missingness assumption.

A Practical Diagnostic Sequence

Specify the target and response model.
State the target cohort, follow-up outcome, response indicator, and pre-outcome covariates used to estimate observation probabilities.

Inspect probability support.
Summarize fitted probabilities for the baseline cohort and investigate very small predictions and covariate patterns with sparse or absent response.

Summarize the weights.
Report upper quantiles, the maximum, weight concentration, and effective sample size so readers can see whether a few respondents dominate.

Check balance and sensitivity.
Compare weighted and unweighted baseline covariates with the target cohort, then report how clearly specified truncation choices affect estimates and balance.

Common Mistakes and Interpretation

Assuming stabilization solves extreme weights. Multiplication by the common $\hat\pi$ rescales all weights equally, so it does not change which respondents dominate a normalized mean.
Treating every positive fitted probability as adequate support. A probability of $0.001$ is technically positive but can create a weight large enough to make an estimate unstable. Examine the magnitude and the covariate pattern, not just whether the model returned a nonzero value.
Calling a truncated estimate automatically more valid. Capping can reduce variance while changing the adjustment and increasing bias. It is a sensitivity analysis or an explicitly chosen procedure, not a correction guaranteed to recover the target.
Equating covariate balance with proof of the assumptions. Balance assesses measured covariates. It cannot verify the absence of residual dependence between response and the unobserved outcome after conditioning on those covariates.

A credible attrition-weighting analysis makes its target and response model explicit, reports probability and weight diagnostics, and shows how estimates behave under reasonable alternatives. Stabilized weights are useful bookkeeping and can make weighted procedures easier to manage, but the key evidence remains adequate support, a defensible response model, and careful interpretation of sensitivity results.

Check Your Understanding

Answer each question using the target cohort and follow-up outcome as context.

If $\hat\pi=0.50$ and $\hat e(X_i)=0.25$, what is the unit's estimated stabilized weight? What does that weight indicate?
Why do stabilized and unstabilized weights give the same normalized weighted mean when the numerator is common to all respondents?
What does a very small estimated observation probability suggest, and why is it not by itself proof of a population positivity violation?
Explain why truncating a large weight can reduce instability but also change the estimator's bias.
If weighted respondents match the baseline cohort on measured covariates, what important assumption about attrition remains unverified?