Why Check the Imputations?
Multiple imputation creates several plausible completed data sets, but producing them is not the same as showing that the imputation setup is adequate. A model can generate values that are mathematically possible under its equations yet implausible for the variable, or its iterative algorithm can still be moving toward a stable pattern. Checking the results helps identify such warning signs before interpreting analyses based on the completed data sets.
These checks are diagnostic, not a pass-or-fail test of whether every assumption is true. A reasonable-looking distribution does not prove that the imputation model is correctly specified, and an observed difference between values recorded in the data and values supplied by imputation is not automatically a defect. Each check should be interpreted in light of the variable, its predictors, and the way missingness arose.
Compare Distributions With Context
A basic check is to compare the observed values of a variable with the values imputed for its missing entries. Useful summaries may include the minimum and maximum, mean and standard deviation, median and quartiles, or a histogram. The best summaries depend on the variable: a mean alone can hide an unusual tail, while a range alone says little about the typical values.
For multiple imputation, examine the imputed distribution in each completed data set, or summarize how the imputed summaries vary across the \(m\) data sets. Do not treat the repeated imputations of one missing entry as if they were \(m\) separate people. The goal is to notice features that deserve investigation, such as a strong shift, an unexpected pile-up at a boundary, or a shape unlike what the model and context make plausible.
The reason can be expressed in terms of a response indicator \(R\), where \(R=1\) means the variable is observed and \(R=0\) means it is missing, and a set of predictors \(X\). Under a missing-at-random model conditional on \(X\), the outcome distribution for missing and observed records can agree at the same \(X\) values while the overall marginal distributions differ. In particular,
If the conditional distributions \(f(Y\mid X,R=0)\) and \(f(Y\mid X,R=1)\) agree, the marginal distributions can still differ because \(f(X\mid R=0)\) and \(f(X\mid R=1)\) may differ. Thus, compare distributions while also asking whether the observed and missing records have different predictor profiles and whether the imputation model reflects relevant relationships.
Worked Example: Interpreting a Difference in Distributions
Suppose a hypothetical health survey records a continuous health score for 100 adults, with 80 observed scores and 20 missing scores. The observed scores have a mean of 68 points, a median of 66, and a standard deviation of 12. Across five completed data sets, the means of the 20 imputed scores are 72, 73, 74, 75, and 76 points; their average is 74 points. The corresponding imputed medians are 70, 71, 72, 73, and 74 points, averaging 72 points.
Calculation. The average of the five imputed means is \( (72+73+74+75+76)/5=370/5=74 \) points. The average of the five imputed medians is \( (70+71+72+73+74)/5=360/5=72 \) points. Relative to the observed summaries, these averages are 6 points higher for the mean and 6 points higher for the median.
That difference is worth examining, but it does not by itself show that the imputation is defective. Suppose the missing-score records tend to be older, and the imputation model includes age and a relevant predictor of the score. If those predictors imply higher scores for the missing records, a higher imputed distribution may be reasonable. The analyst should inspect the predictor patterns, conditional relationships, and the distribution in each completed data set. If the upward shift is unexplained or inconsistent with the model and context, the model needs further review.
Inspect Trace Plots and Iteration Stability
Some imputation procedures, including chained equations, repeatedly update missing values by cycling through variable-specific imputation models. A trace plot displays a quantity recorded at each iteration, such as the mean of the imputed values for a variable or a coefficient in one of the imputation models. With multiple chains, the plot shows a separate trace for each chain.
A stable trace generally fluctuates without a persistent upward or downward trend, and different chains should overlap rather than remain separated in distinct bands. This pattern is evidence that the monitored quantity is not obviously drifting and that chains are exploring similar regions. It is not proof that every feature of the imputation model is correct, nor does a trace plot alone establish that the algorithm has converged in a formal sense.
Look across the whole trace, not just its last point. A long trend, prolonged separation between chains, or very limited movement can signal that more iterations are needed, that chains have not mixed well, or that the setup is difficult for the algorithm. The appropriate response depends on the pattern: inspect the model and variables, consider whether the specifications are suitable, and rerun with justified changes before relying on the output.
Worked Example: Reading Four Imputation Traces
In a hypothetical chained-equations analysis, four chains track the mean of imputed sleep hours. The recorded values are:
| Iteration | Chain 1 | Chain 2 | Chain 3 | Chain 4 |
|---|---|---|---|---|
| 1 | 5.8 | 8.1 | 6.0 | 8.4 |
| 5 | 6.2 | 7.8 | 6.3 | 8.0 |
| 10 | 6.4 | 7.5 | 6.5 | 7.7 |
| 15 | 6.7 | 7.2 | 6.6 | 7.4 |
| 20 | 6.9 | 6.9 | 7.2 | 6.7 |
The first two chains move in opposite directions across the displayed iterations: Chain 1 increases from 5.8 to 6.9 hours, while Chain 2 decreases from 8.1 to 6.9 hours. This persistent movement suggests that the monitored quantity may not have stabilized by iteration 20. The table alone is not enough to diagnose the cause, but the full trace plot and additional iterations should be inspected.
At iteration 20, the four values are 6.9, 6.9, 7.2, and 6.7 hours. Their range is \(7.2-6.7=0.5\) hours. The range at that one iteration describes the spread among the four chain values at that moment; it is not a convergence test and should not replace inspection of the traces over time. The value 7.2 must be included when finding the maximum.
Check Plausibility and Constraints
Every imputed value should be checked against the variable’s measurement scale and substantive constraints. For example, a score defined to range from 0 to 20 cannot be negative or exceed 20. A value outside that range is a clear warning that the imputation model may not respect the variable’s form. Other warning signs can include an unexpected concentration at a boundary, extreme values unsupported by the observed data or context, or categories that are not valid responses.
Plausibility is broader than checking a minimum and maximum. Values can fall within a permitted range but still be implausible in combination with other variables. For example, a generated value may conflict with a structural rule in the data collection process. Consider both individual values and relationships among variables; a plausible marginal distribution does not guarantee plausible combinations.
Worked Example: Detecting Out-of-Range Imputations
A hypothetical questionnaire records a symptom score on an integer scale from 0 to 20. In one completed data set, the imputed scores include three values outside the allowable range: \(-1.4\), \(20.8\), and \(-0.6\), among 100 imputed entries. The observed data contain only scores between 0 and 20.
The fraction outside the permitted range is \(3/100=0.03\), or \(3\%\). Since a questionnaire score cannot be below 0 or above 20, these are not merely surprising observations; they are impossible under the stated measurement scale. A likely issue is that an unconstrained continuous imputation model was used for a bounded score.
The appropriate next step is to inspect the imputation specification, coding, and variable type, then consider an imputation model that respects the score’s scale and bounds. Simply replacing values below 0 with 0 and values above 20 with 20 is not automatically a sound fix: that operation can distort the distribution and relationships. Any adjustment should be justified by a suitable model and checked again after re-imputation.
A Practical Diagnostic Sequence
A useful review combines numerical summaries, plots, and knowledge of how the variables were measured. No single graph or threshold can certify the imputation results. The following sequence keeps the checks focused on potential problems while avoiding the assumption that observed and imputed values must match exactly.
Confirm variable coding, permitted categories, units, bounds, and known structural constraints before judging generated values.
Summarize observed values and imputed values separately using statistics and plots suitable for the variable. Inspect variation across completed data sets rather than treating repeated imputations as new independent records.
For relevant quantities and chains, look for persistent trends, chain separation, or unusually little movement across iterations.
Check predictor distributions, model specifications, coding, and conditional relationships. Revise the setup for a reasoned concern, then rerun and repeat the diagnostics.
Common Mistakes
- Demanding identical observed and imputed distributions. The missing records may have different predictor profiles, so their imputed marginal distribution can differ even when the conditional model is appropriate. Investigate the difference in context.
- Calling a trace plot proof of correctness. A stable-looking trace only addresses visible movement in the monitored quantity; it does not verify the imputation model, predictor choices, or all assumptions.
- Checking only the final iteration. A final value can look reasonable even when the chain has drifted for many iterations. Inspect the pattern over time and across chains.
- Clipping impossible values without review. Truncating out-of-range values may change distributions and associations. First identify why the model produced them and choose a defensible specification.
- Ignoring relationships between variables. Imputed values can be individually within range but implausible conditional on other recorded variables. Check combinations and relevant conditional patterns as well as marginal summaries.
Check Your Understanding
Answer these questions using the ideas in this tutorial.
- Why might the imputed marginal distribution differ from the observed marginal distribution even when the imputation model conditions appropriately on predictors?
- A trace plot shows one chain rising steadily while another falls over the displayed iterations. What does this suggest, and what should be checked next?
- A variable is defined on a scale from 0 to 20, but some imputed values are negative. Why is this a warning sign, and why might clipping the values not be an adequate solution?
- At one iteration, four chain values are 6.9, 6.9, 7.2, and 6.7. Calculate their range and explain what that single range can and cannot tell you.
- Why should repeated imputations of the same missing entry not be counted as separate independent people when summarizing the imputed distribution?