Why Check Relationships, Not Just Distributions?
An imputation model can produce plausible values for an incomplete predictor while still distorting how that predictor relates to the outcome. That matters when the planned analysis depends on a particular relationship: for example, an interaction in which the predictor’s slope changes by group, or a nonlinear effect in which the outcome curve bends. A marginal check of the imputed predictor’s distribution may not reveal either problem.
This tutorial focuses on a targeted diagnostic: compare observed and imputed predictor–outcome relationships within covariate groups relevant to the analysis. For a proposed interaction, compare the relationship separately across the interacting covariate’s levels. For a proposed nonlinear term, examine whether the imputed cases follow the same curve pattern as the observed cases. These comparisons are diagnostic evidence, not a test that can establish model compatibility.
The comparison is most directly interpretable when \(Y\) and the grouping covariates are observed for the records being checked. Under a missing-at-random condition such as \(R_X \perp X \mid Y,Z\), the distribution of \(X\) given \(Y,Z\) is the same for records with observed and missing \(X\). A useful imputation model should reproduce that conditional relationship. If missingness still depends on unobserved \(X\) after conditioning on the variables used in the check, the observed records need not be a reliable reference for the missing records.
The check also requires enough observed and imputed records in the regions being compared. Sparse groups, narrow predictor ranges, influential observations, and uncertainty in the imputed values can make apparent differences unstable. When \(Y\) is itself missing, this direct comparison cannot use that record’s observed outcome; a more involved diagnostic is needed.
Checking an Interaction Through Group-Specific Slopes
Suppose the analysis model specifies a linear interaction between \(X\) and a binary covariate \(Z\):
For \(Z=0\), the slope of \(Y\) on \(X\) is \(\beta_1\). For \(Z=1\), it is \(\beta_1+\beta_3\). Thus, the interaction is represented by a difference between group-specific slopes. A basic diagnostic is to compare the observed and imputed-case slopes within each group, then ask whether the group contrast is retained. This is a descriptive check of the pattern, not an estimate of the final pooled analysis coefficient.
Worked Example: Comparing Slopes Across Interaction Groups
Suppose a hypothetical analysis plans to model \(Y\) using \(X\), binary group \(Z\), and their interaction. For illustration, the observed records in each group and the records with imputed \(X\) have the following pairs:
| Group and record type | \(X\) values | \(Y\) values, in outcome units |
|---|---|---|
| \(Z=0\), observed \(X\) | 1, 2, 3 | 10, 12, 14 |
| \(Z=0\), imputed \(X\) | 1, 2, 3 | 11, 13, 15 |
| \(Z=1\), observed \(X\) | 1, 2, 3 | 10, 13, 16 |
| \(Z=1\), imputed \(X\) | 1, 2, 3 | 11, 12, 13 |
Each sequence has equally spaced \(X\) values, so its slope is the change in \(Y\) divided by the change in \(X\). In group \(Z=0\), the observed slope is \((14-10)/(3-1)=2\) outcome units per \(X\) unit. The imputed-case slope is \((15-11)/(3-1)=2\) outcome units per \(X\) unit.
In group \(Z=1\), the observed slope is \((16-10)/(3-1)=3\) outcome units per \(X\) unit, while the imputed-case slope is \((13-11)/(3-1)=1\) outcome unit per \(X\) unit. The observed group slope difference is \(3-2=1\) outcome unit per \(X\) unit, but the imputed-case group slope difference is \(1-2=-1\). In this hypothetical diagnostic, the imputed cases do not show the specified direction of the interaction. That flags a potential incompatibility or data-selection issue for investigation; it does not, by itself, prove which is responsible.
The comparison is meaningful only if records are being compared on a sufficiently similar basis. In particular, \(Y\) and \(Z\) must be available for these cases, and important variables related to missingness and the \(X\)-\(Y\) relationship should be considered. The small numbers here illustrate arithmetic, not a reliable real-world estimate.
Checking a Specified Nonlinear Pattern
When the analysis specifies a nonlinear effect, the question is not whether one straight-line slope matches. Instead, check whether observed and imputed cases support a similar curve over the predictor range. A simple quadratic model has the form \(Y=\beta_0+\beta_1X+\beta_2X^2+\varepsilon\); its curvature depends on \(\beta_2\). Plotting \(Y\) against \(X\), with observed and imputed cases marked separately, can reveal whether imputed points follow the bend or instead flatten, reverse, or depart systematically from it.
A plot should be read within relevant covariate groups where the analysis relationship depends on those covariates. Binned summaries can help when points overlap: use the same \(X\) ranges for observed and imputed records, and show the number of records in each bin. Binning is a visual aid, not a substitute for the specified functional form. A flexible smoother can also be useful for exploration, but should not silently replace the analysis model’s stated curve.
Worked Example: Looking for a Curved Relationship
Consider a hypothetical analysis that specifies \(Y=10+2X+0.5X^2\), with \(Y\) measured in outcome units. Observed records at \(X=-2,-1,0,1,2\) have \(Y\) values \(8,8.5,10,12.5,16\), respectively. For example, at \(X=2\), the specified curve gives \(10+2(2)+0.5(2^2)=10+4+2=16\). The values show the curve’s upward bend.
Now suppose two records with observed outcomes have imputed \(X\) values of 0 and 1, while their outcomes are 12.5 and 16. At \(X=0\), the curve predicts \(10+2(0)+0.5(0^2)=10\), so the imputed-case point is \(12.5-10=2.5\) outcome units above that curve. At \(X=1\), it predicts \(10+2(1)+0.5(1^2)=12.5\), so the other point is \(16-12.5=3.5\) outcome units above it.
These two points are not enough to characterize a relationship, but their direction is a warning sign: the imputed \(X\) values may be too low for their outcomes relative to the specified curve. A useful follow-up is to inspect the full scatterplot and the distribution of imputed \(X\) across outcome ranges, with uncertainty across imputations shown. Do not conclude that the quadratic term is absent from only two residuals.
Why the Comparison Is Informative—and What It Cannot Establish
The logic behind this check follows from the conditional distribution used to impute \(X\). If the missingness assumption \(R_X \perp X \mid Y,Z\) is plausible, then:
The observed \(X\) values provide information about the right-hand conditional distribution; the imputation model supplies draws for records where \(X\) is missing. If those draws fail to preserve the relationship with \(Y\) within \(Z\), then the completed data may fail to reflect a feature needed by the planned analysis. This is a justification for checking relationships, not a guarantee that a visual comparison will identify every incompatibility.
A key limitation is that observed and missing-\(X\) records can differ in ways not captured by \(Y\) and \(Z\). If missingness depends on additional observed covariates, include them in the comparison or interpret discrepancies cautiously. Also, the observed relationship may be imprecise because the observed-\(X\) subset is small. A mismatch may reflect chance variation, sparse support, a poor imputation model, or a mistaken missingness assumption.
With multiple imputation, imputed values vary from one completed data set to another. Make the plot or compute the diagnostic summaries separately for each imputation, then examine their range or distribution. A single completed data set hides imputation uncertainty. Conversely, simply pooling all imputed records into one large set can make the display look more precise than the information supports.
This diagnostic serves a different purpose from checking only the marginal distribution of \(X\): a marginal check can look satisfactory while the \(X\)-\(Y\) association is distorted. It is also not a replacement for the substantive analysis or for examining missingness patterns. A pooled regression coefficient alone may obscure where a relationship differs, while a group-specific plot can show the location and form of a discrepancy. No one display proves that the imputation model is correct.
A Practical Diagnostic Sequence
Write down the analysis term that matters, such as a group-specific slope or a quadratic curve, and identify the covariates that define relevant comparisons.
Separate cases with observed \(X\) from cases where \(X\) was imputed. Confirm that the outcome and grouping variables are available, and consider additional observed predictors related to missingness.
Use scatterplots, group-specific slopes for a linear interaction, or curve-oriented displays for a nonlinear effect. Mark observed and imputed cases distinctly and report group counts.
Check whether the pattern is stable across completed data sets. Investigate consistent discrepancies rather than treating a single unusual draw as decisive.
Consider support, selection, and sampling variability before revising the imputation model or drawing conclusions about compatibility.
Common Mistakes and Failure Modes
- Checking only marginal distributions. A plausible histogram of imputed \(X\) does not show that the imputed values preserve \(X\)'s association with \(Y\) or its variation across \(Z\).
- Comparing overall slopes when the analysis includes an interaction. Opposing group-specific discrepancies can be concealed in an overall summary. Make comparisons in the groups relevant to the analysis.
- Treating observed cases as an unquestionable gold standard. Observed cases can be selected, sparse, or unrepresentative. The missingness assumption and the variables conditioned on affect how informative the comparison is.
- Ignoring support. If observed cases cover only low \(X\) values but imputed cases occupy a high range, the relationship comparison depends on extrapolation. Show ranges and counts, and avoid confident conclusions where the data do not overlap.
- Over-interpreting one completed data set. Imputed values are draws. A relationship that changes substantially across imputations is uncertain and should not be reduced to one slope or plot.
A frequent error is to reason, “The completed-data scatterplot looks roughly linear, so the imputation preserved the interaction.” That conclusion does not follow. A pooled-looking pattern can conceal different within-group slopes; moreover, a visually plausible plot cannot establish the missingness assumption or rule out systematic bias. The correction is to check the relationship in the groups and predictor ranges required by the specified analysis, and to consider the result alongside uncertainty and model assumptions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- An analysis includes an interaction between \(X\) and binary \(Z\). Why can comparing only the overall \(X\)-\(Y\) slope miss a problem?
- In one group, the observed slope is 2 outcome units per \(X\) unit and the imputed-case slope is 2.5. What is the difference, and what can you conclude from it?
- A quadratic analysis curve is \(Y=5+X+0.25X^2\). What value does it predict at \(X=2\)?
- State a missingness condition that makes the observed-\(X\) records informative about the distribution of \(X\) among records with missing \(X\), conditional on \(Y\) and \(Z\).
- Why should a relationship check be reviewed across multiple imputations rather than on only one completed data set?
Solutions. (1) The overall slope can conceal group-specific differences, even though the interaction is defined by a difference between group slopes. Compare within each \(Z\) group. (2) The imputed-case slope is \(2.5-2=0.5\) outcome units per \(X\) unit steeper. This is a diagnostic discrepancy, not proof of imputation failure; sampling variation, selection, and limited support could also contribute. (3) Substitute \(X=2\): \(5+2+0.25(2^2)=5+2+1=8\) outcome units. (4) One suitable condition is \(R_X \perp X \mid Y,Z\), meaning that after conditioning on \(Y\) and \(Z\), whether \(X\) is observed does not depend on \(X\) itself. (5) Imputed values vary across completed data sets. Reviewing only one can mistake a particular draw for a stable relationship and fails to show the uncertainty in the diagnostic.