Diagnosing Multiple-Imputation Analyses

Why Diagnose Multiple Imputation?

Multiple imputation (MI) can carry uncertainty about missing values into a causal analysis, but that benefit depends on the imputation procedure being implemented sensibly. A fitted model may fail to capture important relationships, an iterative algorithm may not have settled into stable behavior, or the chosen number of imputations may leave appreciable Monte Carlo noise in the pooled result. Diagnostics help identify such problems before an estimate is interpreted.

Diagnostics have limits. A set of plausible-looking imputed values does not show that the imputation model is correct, that all relevant information was included, or that the missingness mechanism is missing at random (MAR). MAR is an assumption about how missingness relates to unobserved values after conditioning on observed information. It is not established by comparing observed and imputed distributions. A diagnostic should therefore be read as evidence about the behavior and coherence of an analysis, not as proof of its identifying assumptions.

The checks below assume that the analyst has specified an imputation procedure and a causal analysis with a defined target estimand. The causal assumptions required by that analysis still apply; diagnostics do not replace them. For pooled estimates, a standard summary uses \(m\) completed data sets, an estimate \(Q_j\) and estimated variance \(U_j\) from completed data set \(j\), and the usual within- and between-imputation quantities:

Here \(\bar{Q}\) is the pooled point estimate, \(\bar{U}\) is the average estimated variance within completed data sets, and \(B\) measures variation between their estimates. These summaries, familiar from multiple imputation for causal effects, are useful for diagnosing Monte Carlo stability.

Model Compatibility and Causal Structure

An imputation model needs to represent the relationships that matter for the analysis. In chained-equations procedures, separate conditional models are fitted for variables with missing values. A set of conditional models is compatible when it can arise from a single joint probability distribution. In practice, separately specified conditional models need not correspond to any such joint distribution. Even when an algorithm produces values, that fact alone does not show that the models work together coherently.

A useful review asks whether the imputation models include the variables and functional forms needed to preserve relationships in the planned causal analysis. For example, if the outcome model includes a treatment-by-covariate interaction, an imputation model that omits the interaction or otherwise fails to represent it may attenuate or distort that relationship. Nonlinear terms, interactions, clustered structure, and deterministic relationships may also need attention. The exact specification depends on the data and the analysis; there is no single diagnostic plot that can establish adequate compatibility.

Check deterministic relationships and known constraints directly. If a total is defined as the sum of its components, imputing the total and components independently can create contradictory records. If a variable is restricted to a range or has categories with a meaningful order, check that imputations respect its support and coding. Such violations are concrete warning signs; their absence does not prove that the model captures all relevant associations.

Convergence and Mixing

Some MI algorithms update missing values iteratively, drawing from conditional models in sequence. For these methods, analysts commonly inspect traces of quantities such as imputed-variable means, standard deviations, or selected regression coefficients over iterations. Multiple chains started from different initial values can help reveal whether chains settle into similar behavior. A stable-looking trace fluctuates around a consistent range without sustained drift, and chains that overlap substantially provide more reassurance than chains that remain in distinct regions.

A flat-looking trace alone is not enough. Strong autocorrelation means successive draws contain limited new information, so many iterations may still yield few effectively independent draws. Inspecting autocorrelation or effective sample size can help assess mixing. If chains have not stabilized or do not mix, the analyst may need to revise the model or run the algorithm longer, then re-check. Convergence checks are specific to the computational procedure: not every imputation method is a Markov-chain algorithm, so a trace plot is not a universal requirement or a substitute for understanding the method used.

Observed and Imputed Distributions

Comparing observed and imputed values can reveal implausible imputations, but the comparison must reflect who has missing values. If missingness depends on observed predictors, the people with observed values can differ systematically from those whose values are missing. Thus, the marginal distribution of observed values need not match the marginal distribution of imputations. A more informative check often compares distributions within relevant strata of observed predictors, such as treatment group, age category, or other variables used in the imputation model.

Look for values outside the variable’s possible range, unexpected spikes, or patterns that conflict with known relationships. For continuous variables, inspect distributions and summaries; for categorical variables, inspect category frequencies. Where the causal analysis compares treatment groups, examine whether the imputation procedure retains plausible relationships with treatment and important covariates. A visually close match is not a requirement in every marginal comparison, nor does it prove that the model is correct.

How Many Imputations Are Enough?

The number \(m\) controls the Monte Carlo precision of the pooled estimate. Increasing \(m\) generally reduces simulation noise, but it does not repair a misspecified imputation model or a failed causal assumption. One useful approximation to the relative efficiency of using \(m\) imputations rather than infinitely many is

Here \(\lambda\) is the fraction of missing information for the estimate, approximately the proportion of its total uncertainty attributable to missingness. Larger \(\lambda\) means more imputations may be needed to achieve a chosen level of relative efficiency. This approximation is a planning aid, not a universal cutoff.

A second useful quantity is the Monte Carlo standard error of \(\bar{Q}\), the simulation error in the pooled point estimate due to a finite number of imputations. If the \(Q_j\) are approximately independent draws with between-imputation variance \(B\), then:

This approximation assumes the imputations are sufficiently independent; serial dependence in an iterative algorithm reduces the effective number of draws. Comparing the Monte Carlo standard error with the standard error of the pooled estimate, or with a scientifically meaningful scale for \(Q\), helps judge whether simulation noise is negligible. One can also rerun with a larger \(m\) and check whether the point estimate, standard error, and interval materially change.

A Practical Diagnostic Sequence

Diagnostics are most useful when they are tied to a specific concern and followed by a decision. A compact review can proceed as follows:

Common Mistakes and Interpretation

• “Observed and imputed distributions should match exactly.” Not necessarily. Missing and observed cases may differ on observed predictors. Compare conditionally where appropriate and investigate meaningful discrepancies.
• “The chains look stable, so the imputation model is correct.” Convergence is about computational behavior, not whether the model includes the right variables or represents the data-generating relationships.
• “Plausible imputed values prove MAR.” They do not. MAR concerns the relationship between missingness and unobserved values conditional on observed information, and these diagnostics cannot verify that condition.
• “A large number of imputations removes bias.” More imputations can reduce Monte Carlo error, but they do not correct model misspecification or unsupported causal assumptions.
• “A single pooled estimate is enough to assess stability.” The estimate can look reasonable while Monte Carlo error is material. Examine uncertainty and, where needed, compare results at larger \(m\).