Why MICE Diagnostics Need More Than One Target
A chained imputation algorithm repeatedly updates missing values using conditional models for variables with missing entries. Because these updates involve stochastic draws, the completed data can change from iteration to iteration. A diagnostic question is therefore not simply whether the imputed values change, or whether a plotted line looks flat. Different checks address different concerns: whether summaries have settled, whether the chains explore their relevant range, whether estimates from a finite run are sufficiently precise, and whether the imputed values are credible given the observed data and the model.
These concerns are related but not interchangeable. A chain can appear stable while moving only within a narrow region. Several chains can overlap yet all be exploring an implausible distribution. A plausible-looking distribution can still be estimated imprecisely because the chain is autocorrelated. And no diagnostic based only on observed data can directly verify the values that would have been observed for cases with missing data.
A trace plot displays a chain-level quantity across iterations. The quantity might summarize the current completed data or the imputed values for a variable. A trace plot can help identify visible drift, long periods with little movement, or differences among chains. It is not a direct plot of the unknown missing values, and it is not a test that the imputation model is correct. Detailed choices of trace summaries and visual patterns are considered separately; here the focus is on what diagnostic target a plot can and cannot address.
Four Questions, Four Kinds of Evidence
The first question is whether the monitored summaries appear stable. A persistent upward or downward trend suggests that the chain may not yet be sampling from a time-invariant regime, or that the algorithm is behaving poorly. But a flat trace only indicates that the particular summary has no obvious trend at the displayed scale. It does not show that the chain has visited all important parts of its distribution.
The second question is whether the chains mix. If a chain moves readily across its relevant range, successive iterations can represent different parts of that range. If it remains near one level for many iterations, or changes only rarely between regions, it may explore slowly. Multiple chains can provide useful evidence when their starting values differ: persistent separation can be a warning. Overlap, however, is not proof of good mixing; chains may share the same restricted range.
The third question is whether a finite run gives adequate Monte Carlo precision for the summary of interest. Stability is about visible behavior over iterations; precision is about how much random simulation error remains in an estimate. Autocorrelation matters because adjacent iterations often carry less independent information than the same number of unrelated draws. The approximate calculation below illustrates why a visually steady trace need not yield a precise estimate.
The fourth question is whether the imputations are plausible. Compare imputed values with observed values, but do so in a way that respects the variables and patterns related to missingness. If cases with missing outcomes differ systematically in their observed covariates, their imputed outcomes need not have the same marginal distribution as outcomes observed in a different covariate mix. Context matters: values must also be credible in units, range, and substantive relationships.
Worked Example: Stability Is Not the Same as Mixing
Suppose three hypothetical MICE chains are run for a continuous variable. A diagnostic summary is the mean of the current imputed values. Over the last 20 iterations, Chain A’s summary stays between 49.7 and 50.3, Chain B’s between 49.8 and 50.2, and Chain C’s between 49.6 and 50.4. In each chain, the summary looks roughly level. Yet closer inspection shows that Chain A remains near 50 for nearly all 20 iterations, while Chain B repeatedly moves above and below 50. What can be concluded?
The summaries provide some evidence of stability: none shows an obvious sustained increase or decrease in this short segment. Chain B’s more frequent movement may be more consistent with mixing than Chain A’s near-constant sequence, but the range alone cannot establish that Chain B explores every relevant region. Nor can this short segment establish that either chain has adequate Monte Carlo precision.
A flat-looking Chain A is not automatically reassuring. If the chain is moving very slowly, the plot may show little visible change simply because it has not explored much. The relevant conclusion is limited: the chosen summary has no obvious drift over the displayed iterations, while Chain A raises a possible slow-movement concern. More iterations, additional summaries, and comparison with other chains may be informative; none alone proves the imputation model correct.
A Short Justification: Why Autocorrelation Affects Precision
Suppose, for illustration, that a chain-level summary \(S_t\) is stationary with variance \(\sigma_S^2\), and its lag-\(k\) correlation is \(\rho_k\). The Monte Carlo estimate based on \(T\) iterations is \(\bar{S}=T^{-1}\sum_{t=1}^{T}S_t\). Its variance depends on both the individual variances and the covariances among iterations:
To see why, expand the variance of a sum: there are \(T\) variance terms, each equal to \(\sigma_S^2\), and for each lag \(k\) there are \(T-k\) pairs with covariance \(\sigma_S^2\rho_k\). Dividing the variance of the sum by \(T^2\) gives the expression above. If correlations are positive, the bracket is larger than one, so the variance exceeds the independent-draw benchmark \(\sigma_S^2/T\). This is a justification for treating chain length and information content as different things, not a guarantee that the stationarity and correlation assumptions hold in a particular MICE run.
Worked Example: A Long Run Can Still Have Limited Precision
Consider a hypothetical chain-level summary whose stationary variance is \(4\) summary-units squared. Suppose the chain has approximately AR(1) autocorrelation, with lag-one correlation \(\rho=0.80\), and is run for \(T=100\) iterations. Under this approximation, the variance of the chain average is approximately
Substitute the values:
The approximate Monte Carlo standard error is therefore \(\sqrt{0.36}=0.600\) summary units. If the 100 iterations were independent, the standard error would instead be \(\sqrt{4/100}=0.200\) units. Thus, in this simplified example, positive autocorrelation makes the chain average substantially less precise than an independent-draw calculation suggests.
This calculation is illustrative, not a universal formula for every chained imputation run. It assumes a stable process and a particular autocorrelation pattern. It also concerns simulation error in the chain average, not the uncertainty in a scientific estimate after proper multiple-imputation analysis.
Checking Whether Imputed Values Are Plausible
Observed-imputed comparisons are useful for finding conspicuous mismatches. For a continuous variable, inspect whether imputed values have an impossible range, implausible concentration, or an unexpected tail pattern. For a categorical variable, check whether imputed categories occur in reasonable proportions and whether structurally impossible combinations appear. These are checks on the output and model implications, not direct assessments of unobserved truth.
A marginal comparison can mislead when missingness is related to observed covariates. A better question is often whether imputed values are credible conditional on relevant observed information. The following example shows how marginal differences can arise even when the imputed values look consistent within covariate groups.
Worked Example: Observed-Imputed Comparisons Must Respect Covariates
Suppose a hypothetical health dataset records systolic blood pressure in millimetres of mercury (mmHg), with a risk group observed for everyone. Among records with observed blood pressure, 60 low-risk patients have a mean of 118 mmHg and 20 high-risk patients have a mean of 142 mmHg. For 20 missing blood pressures, the imputed values have means of 120 mmHg for 10 low-risk patients and 140 mmHg for 10 high-risk patients.
The mean among observed values is
The mean among imputed values is
The imputed mean is 6 mmHg higher than the observed mean. That difference alone is not evidence of a problem: the observed records are 75% low-risk, while the missing records are evenly split between risk groups. Within the low-risk group, the two means are 118 and 120 mmHg; within the high-risk group, they are 142 and 140 mmHg. Those conditional comparisons look much closer.
This comparison supports only a limited conclusion: the group-specific means are not conspicuously inconsistent in this hypothetical example. It does not establish that the imputed distributions, associations, or missing values are correct. A more complete check would also consider spread, tails, other predictors, and the substantive plausibility of individual values.
What the Diagnostics Establish—and What They Do Not
Each diagnostic has a target and a boundary. Keeping those boundaries clear prevents an attractive plot or a reassuring comparison from being given more weight than it can support.
| Diagnostic target | Useful evidence | It does not establish |
|---|---|---|
| Stability | Whether monitored summaries show obvious drift or persistent changes | That the chain explored the full relevant range or that the model is correct |
| Mixing | Whether chains move through their ranges and whether separately started chains behave similarly | That all chains have found every important region or that imputations are plausible |
| Monte Carlo precision | Whether finite simulation error may be substantial for a summary | That the statistical model or missingness assumptions are appropriate |
| Imputation plausibility | Whether values and patterns conflict with observed data or substantive constraints | That unobserved values are correct or that missingness is ignorable |
The main alternatives to a trace-based check answer different questions rather than replacing it. Observed-imputed distributional comparisons focus on output plausibility. Checks of substantive relationships focus on whether associations are preserved or distorted. Re-running chains from different initializations can add evidence about sensitivity to starting points. Reviewing model specifications and missingness assumptions addresses choices that no plot can verify. In practice, these checks complement one another; none is a universal pass/fail test.
A central limitation is that the missing values are unobserved. Their true distribution cannot be compared directly with imputations in the analyzed dataset. Observed values may provide useful reference information, but if the probability of being missing depends on the missing value itself after conditioning on observed information, those observed-data comparisons cannot identify that dependence. Missingness assumptions require substantive justification and sensitivity analysis, not visual confirmation alone.
Common Diagnostic Mistakes
- “The trace is flat, so the chains converged.” A flat summary can conceal slow movement, poor mixing, or a chain trapped in a narrow range. Treat it as limited evidence about that summary, not a proof.
- “The chains overlap, so the imputations are valid.” Overlap can be reassuring about between-chain behavior, but all chains can share a misspecified model or produce implausible values.
- “The observed and imputed marginal means differ, so the model failed.” Different covariate compositions can produce a marginal difference even when conditional patterns are reasonable. Compare relevant groups and distributions before diagnosing failure.
- “More iterations guarantee accuracy.” More iterations can reduce simulation error when the chain is behaving appropriately, but highly autocorrelated or unstable chains may remain uninformative. Longer runs also cannot repair a poor model.
- “Plausible values demonstrate the missingness assumption.” Output that looks reasonable cannot establish why values are missing. That question depends on assumptions and context beyond the observed-imputed comparison.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A trace of an imputed-variable summary is level, but the chain changes little between iterations. Which diagnostic targets are supported, and which remain uncertain?
- Why can two chains with overlapping traces still provide insufficient evidence of good mixing?
- In a run, the variance of a chain summary is 9, the length is 100, and iterations are independent. What is the Monte Carlo standard error of the chain average?
- Observed outcomes have a lower mean than imputed outcomes. Give one reason this difference might not indicate a poor imputation model.
- What can a plausible observed-imputed comparison not establish about the missingness mechanism?
Solutions. (1) A level trace supports the limited observation that the chosen summary shows no obvious drift. Little movement raises a possible mixing concern; Monte Carlo precision and the model’s validity remain uncertain. (2) The chains may all occupy the same restricted region, so overlap does not show that they have explored the full relevant range. (3) Under independence, \(SE=\sqrt{9/100}=0.300\) summary units. (4) The records with missing outcomes may have a different covariate composition; compare observed and imputed values within relevant covariate groups. (5) It cannot show whether missingness depends on unobserved values after conditioning on observed information, nor prove that the missingness assumptions hold.