Comparing Conditional Distributions Across Predictor Levels

Why Compare Within Predictor Levels?

An outcome can have different distributions at different levels of a predictor. A single overall plot can obscure that pattern: for example, an imputation may preserve the overall center while weakening the increase in outcome values across predictor groups. A conditional comparison therefore asks a focused question: within each selected predictor level, how do the observed outcome values compare with the values imputed for cases in that level?

This is a descriptive diagnostic. It can reveal visible differences in location, spread, skewness, or other distributional features across groups. It does not show what the missing outcomes would actually have been, and visual agreement alone cannot establish that an imputation model is correct.

Definition: Let \(Y\) be an outcome, \(X\) a selected predictor, and \(R_i=1\) when the outcome for case \(i\) is observed and \(R_i=0\) when it is missing. For predictor group \(g\), the observed conditional distribution is the distribution of \(Y_i\) among cases with \(X_i\) in group \(g\) and \(R_i=1\). The imputed conditional distribution in completed data set \(l\) is the distribution of the imputed \(Y_i\) values among cases with \(X_i\) in group \(g\) and \(R_i=0\).

For a numerical outcome, let \(O_g\) denote the observed cases in group \(g\), and \(I_g\) the cases with missing outcomes in that group. The empirical cumulative distribution functions (ECDFs), which give the proportion of values at or below a threshold \(t\), are

\widehat F_{\mathrm{obs},g}(t) = \frac{1}{|O_g|} \sum_{i\in O_g}\mathbf{1}(Y_i\leq t), \qquad \widehat F_{\mathrm{imp},g}^{(l)}(t) = \frac{1}{|I_g|} \sum_{i\in I_g}\mathbf{1}(Y_i^{(l)}\leq t).

Here \(Y_i^{(l)}\) is the imputed value for case \(i\) in completed data set \(l\), and \(\mathbf{1}(\cdot)\) equals 1 when its statement is true and 0 otherwise. A histogram, density display, boxplot, or ECDF can show these distributions. The key is to compare observed and imputed outcomes within the same predictor group, using a common outcome scale.

Constructing a Conditional Comparison

For a categorical predictor, use its substantively meaningful levels as groups. For a continuous predictor, grouping it into intervals can make a plot readable, but the interval boundaries should be stated and chosen with the predictor's meaning and the available sample sizes in mind. Grouping is a display decision: it does not make values within a bin identical or remove any underlying variation.

For each group, display observed outcome values and imputed outcome values separately. If there are multiple imputations, inspect the imputed distribution from each completed data set, for example in separate panels or with a consistent visual convention. Do not place all \(m\) completed data sets' imputed values in one pile and treat them as \(m\) independent sets of cases: each missing case appears repeatedly, and that display can hide variation among imputations.

Before interpreting a plot, check that the groups are correctly assigned, the observed/missing status refers to the outcome being diagnosed, and the observed and imputed values use the same units, limits, and transformations. Also check the number of observed and missing outcomes in each group. An apparently smooth distribution based on only a few cases is not strong evidence about a group's pattern.

Worked Example: Comparing a Numerical Outcome Across Two Predictor Levels

Consider hypothetical measurements of an outcome \(Y\) in units. A categorical predictor has levels A and B. In one completed data set, the observed outcomes in A are \(4,5,6,7,8\), and the imputed outcomes for missing cases in A are \(4,5,5,6\). In B, the observed outcomes are \(8,9,10,11,12\), and the imputed outcomes are \(5,6,7,8,9\). Calculate the group means to organize what a conditional plot should show.

For A, the observed mean is \((4+5+6+7+8)/5=30/5=6.00\) units. The imputed mean is \((4+5+5+6)/4=20/4=5.00\) units. For B, the observed mean is \((8+9+10+11+12)/5=50/5=10.0\) units, while the imputed mean is \((5+6+7+8+9)/5=35/5=7.00\) units.

The observed group means differ by \(10.0-6.00=4.00\) units, whereas the imputed group means differ by \(7.00-5.00=2.00\) units. A conditional display would make this weaker separation in the imputed values visible; it would also show the full distributions rather than only their means. This is a reason to inspect the pattern, not by itself a conclusion that the imputation is wrong: the observed and missing cases are not the same cases.

Why Stratification Can Reveal What an Overall Plot Hides

The basic justification is that a distribution across all cases combines information from the predictor groups. For any one collection of values divided into groups, its overall ECDF is the group-size-weighted average of the group ECDFs. If \(n_g\) is the number of values in group \(g\) and \(N=\sum_g n_g\), then

\widehat F(t) = \frac{1}{N}\sum_g\sum_{i\in g}\mathbf{1}(Y_i\leq t) = \sum_g\frac{n_g}{N}\widehat F_g(t).

The first equality counts all values at or below \(t\) by group; dividing each group's count by \(n_g\) gives the second expression. Consequently, group-specific differences can be diluted or obscured in an overall display. For example, if one group shifts upward while another shifts downward, their combined centers may look similar even though neither group's conditional pattern is preserved. This is why an overall comparison and a predictor-stratified comparison answer different descriptive questions.

The weighting identity applies to a given collection of values. It does not imply that observed and imputed groups have matching sizes or represent interchangeable samples. In particular, the diagnostic is not a test whose required outcome is identical observed and imputed distributions in every stratum. Its purpose is to inspect conditional features and identify patterns worth investigating in context.

Worked Example: Grouping a Continuous Predictor

Suppose a hypothetical study records age in years as a predictor and a numerical score in points as the outcome. For a diagnostic display, the analyst uses three stated age intervals: 20–39, 40–59, and 60–79 years. The observed and imputed scores in one completed data set are:

20–39 years: observed \(12,14,16,18\); imputed \(11,13,15\).
40–59 years: observed \(20,22,24,26\); imputed \(18,20,22\).
60–79 years: observed \(28,30,32,34\); imputed \(25,27,29\).

For ages 20–39, the observed mean is \((12+14+16+18)/4=60/4=15.0\) points, and the imputed mean is \((11+13+15)/3=39/3=13.0\) points. For ages 40–59, the corresponding means are \(92/4=23.0\) points and \(60/3=20.0\) points. For ages 60–79, they are \(124/4=31.0\) points and \(81/3=27.0\) points.

Both sets of means increase across the age intervals. The imputed means are lower in each interval, and the observed-minus-imputed differences are \(2.00\), \(3.00\), and \(4.00\) points, respectively. A plot should be inspected for these changes as well as for spread and shape. The intervals are part of the diagnostic specification: different boundaries can produce different within-bin displays, so the boundaries and group counts should be reported.

Reading the Plot and Recognizing Failure Modes

Compare features that matter for the outcome and analysis, rather than relying on a single visual impression. Within each predictor group, look for shifts in center, differences in spread, changes in skewness, gaps or clusters, and values inconsistent with plausible outcome ranges. Then compare how those features vary across groups. A conditional pattern may be visible even if the marginal distributions look alike, or an apparent group difference may be driven by a very small number of values.

The next example illustrates why counts belong beside the plot. When a stratum is sparse, its apparent shape can change substantially with one value. An empty observed or imputed stratum has no distribution to compare.

Worked Example: A Sparse Predictor Level

In a hypothetical diagnostic, a rare predictor level has two observed outcome values, \(3.1\) and \(3.5\) units, and one imputed value, \(2.2\) units. The observed mean is \((3.1+3.5)/2=6.6/2=3.30\) units; the imputed mean is \(2.2/1=2.20\) units. The observed median is also \((3.1+3.5)/2=3.30\) units, while the sole imputed value is its own median, \(2.20\) units.

The numerical difference is \(3.30-2.20=1.10\) units, but the imputed distribution here consists of just one point. A plot can display the values and counts honestly, but it cannot establish a stable distributional discrepancy from this stratum alone. The appropriate conclusion is that the comparison is highly limited by sparse data. Do not merge this level with another merely to make the plot look fuller; combining levels is defensible only if the grouping remains meaningful for the scientific question.

Common diagnostic problems include:

Using an overall plot as a substitute for stratification. An overall display can conceal opposing or uneven group patterns. Show the selected predictor groups directly when the conditional relationship is the diagnostic target.
Changing scales across panels. Independently chosen axes can make distributions look more alike or more different than they are. Use common axes when visual comparison across groups is intended.
Using too many bins for a continuous predictor. Very narrow intervals can leave little data per group. Report counts and, if necessary, use fewer substantively defensible intervals.
Pooling imputations without distinction. Repeated imputed values across completed data sets are not new independent cases. Inspect each completed data set or otherwise preserve the distinction among imputations.
Calling visual similarity validation. Missing outcomes are not observed truth. A plausible-looking plot cannot verify their values or establish that all relevant conditional features are correctly represented.

Choosing Among Diagnostic Displays

A marginal plot is compact and useful for an overall distributional view, but it does not answer whether patterns differ across predictor levels. A stratified plot answers that conditional question at the cost of smaller group-specific samples and greater visual complexity. Groupwise numerical summaries can make shifts easier to describe, but they compress shape and may miss features visible in a distribution plot. These displays are complementary rather than interchangeable.

This diagnostic is also limited by the predictor selected and the way it is grouped. A plot across one predictor cannot demonstrate that all relevant conditional structure has been preserved. Nor does a plot resolve whether a visible difference reflects the imputation procedure, the composition of cases with observed outcomes, or sparse data. Treat it as a targeted check that can motivate closer review, not as a stand-alone pass/fail rule.

Key takeaway: Compare observed and imputed outcome distributions within clearly defined predictor groups, on common scales, and with group counts visible. Interpret the conditional patterns as diagnostic evidence, while keeping the limits of sparse strata and unobserved outcomes explicit.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In a predictor group, the observed outcomes are \(2,4,6\), and the imputed outcomes in one completed data set are \(3,5\). Calculate both means and state one conclusion the comparison supports and one it does not support.
Why can an overall distribution plot conceal a conditional difference across predictor groups?
A continuous predictor is divided into many narrow intervals, leaving one or two cases per interval. What is the diagnostic concern, and what is a reasonable response?
Why should imputed values from several completed data sets not simply be pooled and treated as independent cases?
If a predictor group has no observed outcomes, what can be concluded from an observed-versus-imputed distribution comparison in that group?

Solutions. 1. The observed mean is \((2+4+6)/3=12/3=4.00\); the imputed mean is \((3+5)/2=8/2=4.00\). Their centers match in this small example, but this does not establish that the distributions or unobserved true outcomes match. 2. The overall distribution combines groups, so group-specific shifts can be diluted or offset in the aggregate. 3. The group distributions are too sparsely represented to assess reliably; use fewer substantively defensible intervals and show group counts. 4. The same missing cases recur across completed data sets, so pooling duplicates their representation and hides between-imputation variation. 5. There is no observed distribution in that group to compare; the conditional diagnostic is unavailable there.