Distributional Estimation With Missing Outcomes

Why Missing Outcomes Change a Distribution

When an outcome is missing for some units, the observed outcomes need not have the same distribution as the outcomes for everyone in the study. If response is more likely for people with particular observed characteristics, the respondents can overrepresent some parts of the covariate distribution. Averages are not the only quantities affected: the entire outcome distribution, including its quantiles, can be distorted.

Response-probability weighting addresses this problem by giving each observed outcome more weight when it represents units that were less likely to respond. The method does not recover missing values one by one. Instead, under explicit assumptions, it identifies distributional features of a defined population from observed outcomes and covariates.

The target population must be specified. In the basic setup here, it is the population represented by the study’s eligible units, whose covariates are observed whether or not their outcomes are observed. The assumptions say that, within covariate values, respondents represent nonrespondents with respect to \(Y\), and that every relevant covariate group has some chance of response. They do not say that response is unrelated to the outcome without conditioning on \(X\).

Why Inverse Response Probabilities Work

A respondent with response probability 0.25 receives weight \(1/0.25=4\): that respondent represents, on average, four units with those covariates. This intuition follows from a conditional expectation identity. For any fixed threshold \(y\), let \(I(Y\leq y)\) equal 1 when the outcome is at most \(y\), and 0 otherwise.

The first equality uses the fact that \(X\) and \(Y\) are fixed inside this conditional expectation. Under missing at random, \(E[R\mid X,Y]=P(R=1\mid X,Y)=P(R=1\mid X)=\pi(X)\); positivity makes the division valid. Taking expectations of both sides gives the identification result:

This argument applies to every threshold \(y\), so it identifies the marginal outcome distribution, not just one mean. If response probabilities are known, a sample version of the expectation on the right is the Horvitz–Thompson CDF estimator. If the probabilities are estimated, the same construction uses estimated probabilities \(\hat{\pi}(X_i)\), provided their estimation is sufficiently reliable.

In finite samples this estimator need not end at 1, because the sum of the estimated weights need not equal \(n\). A common alternative is the normalized, or Hájek, weighted CDF:

The normalized version is a weighted empirical CDF: it is nondecreasing and runs from 0 to 1 when the observed outcomes are finite. Both estimators target the same population CDF under the assumptions and suitable sampling conditions, though they can differ in a finite sample.

Conditional CDFs and Marginal Quantiles

For a particular covariate value \(x\), the conditional CDF is \(F(y\mid x)=P(Y\leq y\mid X=x)\). With discrete covariates, it can be estimated within the \(X=x\) group using response weights. With continuous covariates, a conditional model or a smoother is needed; for example, a kernel estimator gives more influence to observations with covariates near \(x\).

Here \(K_h(X_i-x)\) is a nonnegative smoothing weight that decreases with the distance between \(X_i\) and \(x\); \(h\) controls the degree of smoothing. This estimator requires appropriate smoothness and enough observed outcomes near \(x\). Alternatively, distribution regression models the binary indicator \(I(Y\leq y)\) as a function of \(X\), fitting a separate binary-response model at each threshold or using a suitably structured model across thresholds. The response weights can be used in the fitting procedure. Any fitted conditional CDF should be a valid nondecreasing distribution curve in \(y\).

A marginal quantile for probability level \(\tau\), where \(0<\tau<1\), is defined as the generalized inverse of the marginal CDF. Thus the response-weighted marginal quantile is obtained by inverting the estimated marginal CDF, not by averaging conditional quantiles.

Standardizing Conditional Distributions to a Target

A marginal distribution can also be assembled from conditional distributions. If the target population has covariate distribution \(F_X\), then

In a finite target sample with observed \(X_i\), a direct estimate averages estimated conditional CDFs over those target covariates. This makes the target population explicit: the same conditional outcome distributions can produce different marginal CDFs when averaged over different covariate distributions. The target needs support in the study data; if a target covariate group is absent, its conditional outcome distribution cannot be recovered from these data alone.

Overlap, Large Weights, and Truncation

Positivity is an identification condition, but very small response probabilities create a practical problem even when they are technically positive. If \(\pi(X_i)\) is close to zero, the inverse weight is large. A few such respondents can dominate a weighted CDF and make the estimate unstable. It is therefore important to inspect the response probabilities and weights, including their range, largest values, and concentration among respondents.

Truncation or capping replaces large weights with a chosen maximum. It can reduce sensitivity to a small number of observations, but it changes the estimator and generally introduces bias for the original target distribution. Truncating does not repair a true lack of overlap: if some target covariate values have zero probability of response, their outcomes are not identified by response weighting. A different estimand, a defensible restriction of the target population, or additional information may be needed.

Assumptions, Interpretation, and a Common Pitfall

The key assumptions and choices should be stated alongside a distributional estimate:

• Missing at random: after conditioning on the observed covariates used in the response model, response does not depend on the outcome.
• Positivity: the response probability is positive for every covariate pattern represented in the target population.
• Response-probability estimation: if \(\pi(X)\) is estimated, the model or estimation method must adequately recover the relevant response probabilities.
• Target definition: the covariate distribution used in the marginal estimate must match the population about which the conclusion is made.

A common mistake is to say, “The respondents are representative because missingness is at random,” and then use an unweighted respondent CDF. That reasoning is wrong when response probabilities vary with \(X\): missing at random is a conditional statement, not a claim that the marginal respondent distribution equals the target distribution. The correction is to condition on the relevant observed covariates and use response weighting, or to estimate conditional CDFs and standardize them to the target covariate distribution. If response probabilities are constant across the target population, the unweighted respondent distribution may be appropriate under the stated assumptions; otherwise, it generally is not.

An estimate should be interpreted as a feature of the stated target population, under the missingness and positivity assumptions. It is not a causal effect, and weighting alone cannot verify that the missing-at-random assumption is true, since outcomes are unavailable for nonrespondents. Inference also needs to reflect the estimation procedure: when response probabilities are estimated, their uncertainty should be accounted for, and quantile uncertainty depends on the behavior of the estimated CDF near the quantile.