Adding Uncertainty to Observed-Imputed Comparisons

Why Add Uncertainty to a Diagnostic Comparison?

A difference between an observed-data summary and a summary of imputed values is not self-interpreting. It may reflect a substantial discrepancy, ordinary sampling variability, or variation in the stochastic imputations. An uncertainty display helps show how much precision the plotted summaries have, but only if the display matches the source of uncertainty and the quantity being summarized.

For a variable \(Y\), let \(O\) denote the records where \(Y\) is observed and \(M\) the records where it is missing. With \(m\) completed datasets, write \(T^{(l)}\) for a selected summary of the imputed values in \(M\) in completed dataset \(l\), for \(l=1,\ldots,m\). A summary of observed values, such as their mean, is calculated once from \(O\). The \(m\) values \(T^{(1)},\ldots,T^{(m)}\) show how that imputed-data summary varies across the imputations.

An uncertainty bar must therefore be labelled by what it represents. A confidence interval for the observed mean is not an interval for the imputed mean. The spread of \(T^{(1)},\ldots,T^{(m)}\) is not, by itself, a confidence interval for a population quantity. A diagnostic plot can display both, but should not make them appear to have the same interpretation.

Across-Imputation Variation for a Selected Summary

For any chosen summary \(T\), calculate \(T^{(l)}\) separately in each completed dataset using the same definition and the same set of records. The average and sample standard deviation across the \(m\) values are

Here, \(\bar{T}_{\mathrm{imp}}\) is the average displayed summary across imputations, and \(s_{\mathrm{imp}}\) describes their spread. This spread can be shown as a standard-deviation bar, or the individual \(T^{(l)}\) values can be shown as points. A range or quantile band can also be shown, although extreme quantiles are unstable when \(m\) is small.

The formula uses the sample standard deviation because the \(m\) values are used to describe a distribution of possible imputed summaries. It does not make the completed datasets \(m\) independent samples of people. In particular, \(s_{\mathrm{imp}}/\sqrt{m}\) would describe precision of an average over independent draws only under appropriate independence assumptions. It is not a general-purpose standard error for the diagnostic summary. Stored imputations may also be dependent across iterations in a chain.

Choosing an Uncertainty Method

The method depends on the diagnostic target. If the aim is to show precision of a summary calculated from observed records, an analytic interval or a bootstrap of observed records may be appropriate under its assumptions. If the aim is to show how a selected summary of missing values changes over stochastic imputations, display the distribution of its \(m\) values. If the aim is to quantify sampling uncertainty in a comparison involving the imputation procedure, resample the original records and rerun the imputation procedure in each resample.

For a bootstrap of observed records alone, resample those records with replacement and recalculate the observed-data statistic in each resample. This estimates sampling variability for that observed-data statistic under the bootstrap's sampling assumptions. It does not account for uncertainty in imputed values or remove possible differences between the observed and missing records.

For a bootstrap comparison that includes imputations, resample the original records as sampling units, retain each record's observed/missing pattern, rerun MICE in each bootstrap sample, and recalculate the comparison. This repeats the analysis pipeline so that the resulting distribution can reflect sampling variation and stochastic imputation, subject to the validity of the bootstrap design and imputation procedure. Resampling individual cells independently would generally break the record structure and missingness pattern.

A Full-Pipeline Bootstrap for a Comparison

Sometimes the diagnostic target is a comparison statistic itself, such as the difference between an observed-data mean and an imputed-data mean. A bootstrap can be used to show how that comparison varies under repeated sampling, but it must reproduce the relevant sampling and imputation steps. The resampling unit, design, and statistic must match the data and target. If records are clustered or sampled with unequal probabilities, an ordinary record-level bootstrap may be inappropriate.

Why Completed Datasets Are Not Independent Samples

Each completed dataset contains the same observed records and uses imputations for the same missing entries. The datasets are alternative completed versions of one incomplete dataset, not new samples of people. The shared observed data alone make it wrong to combine all \(m\) completed datasets into one large dataset and calculate a conventional standard error as if its records were independent. Imputed values can also be dependent across stored iterations of a chain.

The quantity \(s_{\mathrm{imp}}\) describes the spread of a selected summary across imputations. It is not the standard error of a pooled analysis estimate and should not be presented as a confidence interval without a separate justification. Conversely, a confidence interval for an observed-data summary does not show how uncertain the imputed values are. These distinctions matter even when the uncertainty bars look similar in length.

For a stable display, use enough imputations to represent the across-imputation distribution adequately, and inspect whether stored values are sufficiently separated and stable for the intended summary. If imputations are highly autocorrelated, the apparent number of stored datasets overstates the amount of independent information about that distribution. When an interval is too unstable to estimate, showing individual imputation summaries and describing their limited number is more transparent than implying precision with narrow quantile bars.

Diagnostics, Limitations, and Common Pitfalls

Before interpreting a display, check that each statistic is calculated on the intended records and scale. The observed summary and each imputed summary may refer to different subsets, so make that distinction explicit in axis labels or a caption. Check whether an analytic interval's sampling and distributional assumptions are reasonable; with sparse outcomes or strongly skewed data, a simple normal interval may behave poorly. For a bootstrap, check that the resampling unit and design are appropriate and that MICE converges and produces usable imputations in the resamples.

• Pitfall: treating \(m\) completed datasets as \(m\) independent samples. This inflates the apparent sample size and gives invalid standard errors. Calculate the chosen summary within each imputation; do not stack all completed datasets as independent observations.
• Pitfall: calling the across-imputation standard deviation a 95% confidence interval. It is a spread measure, not a confidence interval. Label it as across-imputation variation or display the imputation-specific points.
• Pitfall: using overlap of bars as a significance test. Overlap or non-overlap of intervals for different targets does not, by itself, provide a calibrated test of their difference. If inference on a difference is needed, define that target and use an appropriate inferential procedure.
• Pitfall: assuming uncertainty bars repair a poor model. A precise-looking display cannot establish that the imputation model is adequate or that the missingness process is ignorable. Uncertainty describes variation under the chosen procedure; it does not cover every model failure.

Uncertainty displays are most useful as complements to distribution comparisons and standardized distribution gaps: they prevent a visual difference from being read without regard to precision. They have limits. Sampling intervals for observed records do not directly resolve differences between observed and missing cases, and variation across imputations is conditional on the data and imputation procedure. The display should therefore support diagnosis, not replace substantive checks of the model and its assumptions.