Separating Selection Effects From Imputation Discrepancies

Why a Marginal Gap Has More Than One Explanation

An imputed value belongs to a case whose target variable was missing; an observed value belongs to a case whose target was recorded. These groups can differ in their measured covariates and in their missingness patterns. As a result, their marginal distributions can differ even when their conditional distributions agree within relevant covariate strata. Conversely, similar marginal distributions can conceal conditional discrepancies that point in opposite directions across strata.

The diagnostic question is therefore not simply whether the observed and imputed marginals match. It is whether measured differences in group composition plausibly account for the marginal gap, and whether meaningful gaps remain after comparing like with like. The earlier material on why imputed and observed marginals need not match motivates this distinction; here the focus is on interpreting it carefully.

A plausible selection explanation is that the groups have different distributions of measured \(X\), while observed and imputed \(Y\) behave similarly within comparable values of \(X\). Under a missing-at-random assumption, if the observed covariates adequately capture the variables associated with missingness and the target, equality of the conditional distributions is expected in the population. This is a diagnostic interpretation, not proof that missingness is at random. Unmeasured predictors, a misspecified imputation model, chance variation, or lack of overlap can also affect the comparison.

Separate Composition From Conditional Differences

For a quantitative target, write \(\mu_r(x)=E(Y\mid R=r,X=x)\) and let \(F_r\) be the distribution of \(X\) in group \(R=r\). The marginal mean in each group combines its conditional means with its own covariate composition. Thus the difference between imputed and observed means can be decomposed by adding and subtracting the observed conditional mean averaged over the missing group’s covariate distribution:

The first bracket is a composition component: it asks how much the observed conditional means would differ if averaged over the missing group’s rather than the observed group’s covariate distribution. The final term is a conditional-discrepancy component, averaged over the missing group’s covariate distribution. This is an algebraic decomposition, not a causal decomposition. Reversing the reference distribution gives a different allocation between components, although the total difference is unchanged.

In practice, \(X\) may include several variables and the conditional distributions may be complex. Stratification is most interpretable when there are a few substantively important covariates and enough cases in each stratum. Standardization instead chooses a common set of covariate weights and averages each group’s stratum-specific summaries over those same weights. For categorical targets, compare conditional proportions or full conditional category distributions, not only means.

Worked Comparisons

Use Missingness Patterns and Diagnostics Carefully

When several variables have missing values, a single \(R\) indicator may hide important structure. A missingness pattern records which variables are observed and which are missing for a case. The distribution of a target and its predictors can differ across these patterns. Comparisons within patterns can clarify whether a gap is associated with measured case composition, but they do not make imputed target values observed data. Report the pattern definition, the stratum sizes, and the extent of overlap; avoid pooling unlike patterns into a single unexplained category.

Diagnostics are most informative when strata are substantively justified, sufficiently populated, and represented in both groups. Sparse cells make proportions and means unstable; a visually striking gap in a very small cell may be mostly sampling variation. Conversely, large pooled samples do not guarantee useful comparisons if one group is absent from a key stratum. Check the distributions of important covariates, the conditional summaries within strata, and whether conclusions change under reasonable choices of common weights. For continuous covariates, coarse bins can conceal residual differences, while excessively fine bins can create sparse comparisons.

For an imputed dataset, summaries of the missing cases depend on the fitted imputation procedure and the realized imputations. Multiple imputations can reflect some uncertainty in those summaries, but agreement across imputed datasets does not independently validate the unobserved values: all may reflect the same model assumptions. A comparison with observed cases is a model diagnostic, not a direct accuracy assessment for the missing cases.

Alternatives, Limits, and Common Mistakes

A frequent mistake is to say, “The imputed marginal differs from the observed marginal, so the imputation is defective.” That conclusion ignores selection into the observed and missing groups. The correction is to compare relevant covariate compositions, examine conditional distributions, and state what the data can and cannot support. The opposite mistake is to say, “The adjusted summaries agree, so the imputations must be valid.” Agreement is reassuring only for the measured comparisons made; it cannot rule out unmeasured selection or errors shared by the imputation model.

These diagnostics have limits. Conditional agreement may arise by chance, and a conditional discrepancy may reflect sampling variability, unmodeled interactions, an unsuitable functional form, or a real difference associated with unmeasured causes of missingness. Where missingness depends on the unobserved target even after conditioning on measured covariates, the observed data alone cannot identify the missing group’s true conditional distribution. Therefore, present these comparisons as evidence that prioritizes investigation—not as a definitive test of validity or a causal claim.

A Practical Review Sequence