Stratifying Observed-Imputed Comparisons by Predictors

Why Stratify an Observed-Imputed Comparison?

A marginal comparison combines values across all predictor profiles. It can therefore conceal an important problem: observed values and imputed values may have similar overall distributions while behaving differently at particular predictor levels. The reverse can also occur. The overall distributions may differ because the two groups contain different proportions of predictor profiles, even when their distributions look similar within each profile.

Stratifying means dividing records into groups defined by predictor values, then comparing observed and imputed values separately within each group. The strata should be chosen before inspecting the comparison and should be substantively interpretable. This diagnostic adds context to the empirical CDFs, distribution plots, and summaries used in earlier comparisons; it does not replace them.

For quantitative \(Y\), one useful comparison is the empirical CDF within each group. If \(n_{O,s}\) is the number of observed values in stratum \(s\), and \(n_{I,s}\) is the number of missing values with imputations in that stratum, their empirical CDFs are

Here \(k\) indexes one of the \(K\) imputed data sets, and \(Y_i^{(k)}\) is the imputed value for record \(i\) in that data set. The observed values are the same across imputations; the imputed values can vary. For categorical variables, compare the within-stratum category proportions instead. Means, medians, spreads, and tail summaries can supplement these distributional comparisons, but a single summary cannot describe every possible discrepancy.

What the Comparison Can and Cannot Establish

The rationale for comparing these groups is conditional. If missingness is independent of \(Y\) given a sufficiently rich set of observed predictors \(W\), then the population distribution of \(Y\) among observed and missing cases is the same at a fixed value of \(W\). A correctly specified imputation model should represent the conditional distribution of missing \(Y\) using relevant predictor information. A stratum is often a practical, coarse summary of that information.

This does not mean that observed and imputed values must have identical distributions in every stratum. Equality is most directly expected when the stratum captures the relevant predictors well enough that the observed and missing cases are comparable within it. If important predictors vary inside a broad stratum, or if their distributions differ between observed and missing cases, the two groups can differ even when the imputation model uses those predictors appropriately. Conversely, apparent agreement in broad strata does not establish that a model is adequate.

In practice, use the comparison to locate patterns that merit investigation: a persistent shift, different spread, unusual tails, or discrepancies concentrated in particular predictor levels. Consider whether the stratum is too broad, whether relevant predictors have been omitted from the comparison, whether the imputation model represents interactions or nonlinearities, and whether cell sizes are adequate. These are diagnostic clues, not a formal test with a universal pass-fail threshold.

Why Marginal and Stratified Comparisons Can Disagree

A marginal distribution is a mixture of the stratum-specific distributions. Its weights depend on how many records from each stratum enter that group. For a distribution function, the observed and imputed mixtures are

where \(w_{O,s}=n_{O,s}/\sum_s n_{O,s}\) and \(w_{I,s}=n_{I,s}/\sum_s n_{I,s}\). The weights need not be equal: observed and missing records can have different predictor profiles. Even if the within-stratum distributions agree, unequal weights can produce different marginal distributions. If within-stratum discrepancies point in opposite directions, they can also cancel in the marginal comparison.

Choosing Strata and Reading Distribution Summaries

A useful stratum is defined by predictor information that matters for the imputation and has enough observed and missing cases to make a comparison interpretable. For a categorical predictor, its levels may be used directly, or scientifically defensible levels may be combined before looking at the results. For a continuous predictor, a small number of meaningful ranges can be used when the ranges are chosen in advance. Avoid creating many narrow bins after seeing an unexpected plot: that practice can turn random variation into an apparent pattern and leave too few records in each group.

Within each stratum, use the same scale and summary definitions for observed and imputed values. An empirical CDF can show differences in location, spread, and shape across the full range. For example, a higher CDF at a particular value means a larger fraction of that group lies at or below that value. For a binary variable, the category proportion is direct; for a multi-category variable, compare the full set of within-stratum proportions rather than one category in isolation.

With multiple imputed data sets, keep the imputed comparisons separated by imputation when inspecting them. Look for patterns that recur across imputations, and note whether a discrepancy varies substantially from one imputation to another. Simply concatenating all imputed values treats repeated imputations as if they were independent records and can create an unjustified impression of precision. These plots and summaries are descriptive diagnostics, not inferential analyses of independent samples.

Diagnostics, Alternatives, and Limitations

A stratified comparison is most informative when it is planned around predictors relevant to the imputation, and when the same strata contain enough observed and missing cases. Review the group counts alongside every plot or summary. A CDF based on a handful of observations has large jumps; category proportions based on a small denominator are unstable. If a stratum has no observed outcomes or no missing outcomes, the corresponding comparison cannot be made. Report that limitation rather than silently merging groups in a way chosen to improve the appearance of agreement.

Patterns worth investigating include gaps that recur across imputations, abrupt differences at a predictor level, strong changes in spread, or a discrepancy concentrated in one tail. Also check whether a coarse stratum hides variation in a continuous predictor, or whether the observed and missing records within a stratum differ on other predictors. A discrepancy can reflect such differences in composition rather than a simple failure to reproduce \(Y\) at a fixed predictor profile. Similarity can also be misleading when strata are too broad or sparse.

The main alternative is a marginal observed-imputed comparison. It is simpler and useful as an overall summary, but its mixture weights can hide conditional patterns. Comparing only means or proportions is more compact, but can miss differences in spread, tails, or category distributions. Stratifying by many predictors can provide more detail, but creates sparse cells and noisy comparisons; it is not automatically better. Choose a modest set of prespecified strata that makes the diagnostic readable and relevant.

The comparison is descriptive. It does not test the missingness mechanism, prove that the imputation model is correct, or establish that the missing values are accurate. It also compares values from records with different missingness status, not paired observed and missing values on the same record. If the comparison raises a concern, use it to guide checks of predictor information, model specification, and data quality. Agreement is reassuring only in a limited sense: important discrepancies may remain in unexamined predictors or relationships.

A Practical Comparison Sequence