Why compare chains separately?
In MICE, each chain produces imputed values for the records missing a variable. A plot that combines imputations from all chains can show an overall observed–imputed contrast, but it hides whether that contrast appears in every chain or is driven by one chain. Those are different diagnostic situations. A difference repeated across chains merits attention as a possible common feature of the imputation process; a difference confined to one chain suggests chain-specific behavior that a pooled plot can conceal.
For this comparison, use the same observed values as the reference for every chain, and plot the imputations from each chain separately against that reference. Keep the variable, plotting scale, bins or smoothing choices, and other display settings consistent. The question is not whether the plotted distributions prove the imputation model correct. It is whether the visible pattern is similar across chains, and what that pattern can reasonably tell you.
This definition concerns distributions shown by the plots, not matched pairs of observed and imputed values: an observed value belongs to a record with an observed outcome, whereas an imputed value belongs to a record whose outcome was missing. The observed values provide a useful descriptive reference, but they do not reveal the missing values’ true distribution. In particular, do not assume that observed and imputed marginals must match exactly.
Read the panels as a comparison
A practical display has one observed–imputed comparison per chain, with the observed distribution repeated as the common reference. Repeating the reference makes it easier to see whether each chain departs from the same baseline in the same way. If the panels do not use the same horizontal scale or bin boundaries, apparent differences can be created or obscured by the display rather than by the data.
For a binned plot, let \(p_{\text{obs},k}\) be the observed proportion in bin \(k\), and let \(p_{m,k}\) be the proportion of chain \(m\)'s imputed values in that bin. The bin-level difference is \(d_{m,k}=p_{m,k}-p_{\text{obs},k}\). A negative value means that chain \(m\) has a smaller proportion than the observed reference in bin \(k\); a positive value means it has a larger proportion. These differences sum to zero across bins when the bins cover the full plotted range and both sets of proportions are normalized to sum to one. They describe the plotted samples; they are not formal tests.
A shared pattern is suggested when the signs and approximate sizes of \(d_{m,k}\) are similar across chains. If one chain has a distinctly different pattern, the comparison is chain-specific even if averaging across chains makes the overall difference look small. Distribution plots can reveal shape features that a single mean or standard deviation would miss; conversely, a plot should be interpreted with its sample sizes and resolution in mind.
Worked Example: A Similar Difference in All Chains
Suppose a hypothetical continuous variable is displayed in three fixed bins: low, middle, and high. There are 100 observed values and 100 imputed values in each of three chains. The observed bin counts are 20, 50, and 30. The imputed counts are: chain 1, 10, 60, and 30; chain 2, 12, 58, and 30; chain 3, 11, 59, and 30.
The observed proportions are \(20/100=0.20\), \(50/100=0.50\), and \(30/100=0.30\). Chain 1 has proportions \(0.10, 0.60, 0.30\), so its differences are \(0.10-0.20=-0.10\), \(0.60-0.50=0.10\), and \(0.30-0.30=0\). For chain 2, they are \(0.12-0.20=-0.08\), \(0.58-0.50=0.08\), and \(0.30-0.30=0\). For chain 3, they are \(0.11-0.20=-0.09\), \(0.59-0.50=0.09\), and \(0.30-0.30=0\).
All three chains show fewer imputed values in the low bin and more in the middle bin, with no difference in the high bin. The direction and size of the contrast are similar across chains, which is evidence of a shared plotted discrepancy rather than a pattern limited to one chain. It does not establish why the contrast occurs or whether the imputed values are wrong; it identifies a pattern for further assessment.
Separate the average contrast from chain variation
The distinction can be made explicit for any one bin. Across \(M\) chains, let \(\bar d_k\) be the average of the chain-specific differences in bin \(k\). The average squared difference has an exact decomposition:
To verify it, write \(d_{m,k}=\bar d_k+(d_{m,k}-\bar d_k)\), square, and sum over chains. The cross-product term vanishes because \(\sum_{m=1}^{M}(d_{m,k}-\bar d_k)=0\). The first term on the right represents the squared average contrast; the second represents variation around that average. This identity is descriptive: it does not turn the chains into independent observations or supply a significance test.
A small average contrast alone is not enough to conclude that all chains match the reference. Chain differences can point in opposite directions and cancel. Likewise, a nonzero average can be dominated by one chain. Inspect the individual panels as well as any across-chain summary.
Worked Example: Opposing Chain Differences Cancel
Suppose the observed proportions in three bins are \(0.25, 0.50, 0.25\). Three chains have imputed proportions \(0.15, 0.55, 0.30\); \(0.25, 0.50, 0.25\); and \(0.35, 0.45, 0.20\), respectively. Each row sums to 1.
Subtracting the observed proportions gives the chain-specific differences: \(d_1=(-0.10,0.05,0.05)\), \(d_2=(0,0,0)\), and \(d_3=(0.10,-0.05,-0.05)\). Their average is \((0,0,0)\), since, for example, \((-0.10+0+0.10)/3=0\) in the first bin.
The average difference would suggest no contrast, but the separate comparisons show that chains 1 and 3 depart in opposite directions. In the first bin, the average squared difference is \((0.01+0+0.01)/3=0.00667\); the squared average difference is \(0\). Thus the departure in this bin is entirely represented by variation among the chain-specific differences in this descriptive decomposition. A pooled plot can hide that variation by averaging the opposing patterns.
Worked Example: One Chain Drives a Pooled Contrast
Now suppose the observed proportions are \(0.20, 0.60, 0.20\) in three bins. Chain 1 has proportions \(0.40, 0.20, 0.40\); chain 2 has \(0.18, 0.64, 0.18\); and chain 3 has \(0.22, 0.56, 0.22\). Imagine that each chain has 100 imputed values.
The differences are chain 1: \((0.20,-0.40,0.20)\); chain 2: \((-0.02,0.04,-0.02)\); and chain 3: \((0.02,-0.04,0.02)\). Pooling the 300 imputed values gives bin proportions \((80/300,140/300,80/300)\), or approximately \((0.267,0.467,0.267)\). Relative to the observed proportions, this pooled contrast is approximately \((0.067,-0.133,0.067)\).
The pooled plot suggests a moderate excess in the outer bins and a deficit in the middle bin. The separate panels show that chain 1 has a much larger version of this pattern, while chains 2 and 3 have smaller, opposing differences. The pooled contrast is therefore not a pattern of comparable size in every chain. The individual plots identify the chain that warrants closer checking; the pooled plot alone does not.
Assumptions, diagnostics, and alternatives
These plots require few mathematical assumptions, but their interpretation depends on several practical conditions. The observed and imputed values must refer to the same variable on a comparable scale; the imputed set must contain the relevant missing records from that chain; and the panels must use consistent plotting choices. The sample sizes should be reported or readily visible, since a small number of values can produce a jagged distribution that looks different by chance. If weights, transformations, or restricted ranges are used, apply them consistently and state what is displayed.
Treat a recurring shape difference as a diagnostic signal, not a verdict. A shared pattern could reflect a common model feature, the fact that missingness selects a different subset of records, limited observed data, or ordinary sample variation. A single unusual chain could reflect chain-specific variation, a coding or data-handling problem, or a genuinely different set of simulated imputations. The plot by itself cannot distinguish all these explanations or establish that an algorithm has converged.
Useful checks include confirming that every chain contains the expected number of imputations, checking that the same observations define the reference in every panel, and verifying that bins and axes have not changed between panels. Look for abrupt changes in plotted sample counts, impossible values, inconsistent categories, or a single panel whose scale differs. A suspicious display should first prompt a check of the data and plotting procedure, before it is interpreted as a substantive distributional finding.
The main alternative is to pool imputations across chains before plotting. Pooling can give a compact picture of the overall imputed distribution, but it discards chain identity and may conceal opposing patterns or a chain-specific outlier. A table of per-chain means or proportions is compact, but can conceal shape differences. Other distribution plots, such as density plots, quantile plots, empirical cumulative distribution functions, boxplots, or violin plots, can make different features easier to see; the essential requirement here is to show the chains separately with a consistent reference and display setup. These plots complement, rather than replace, numerical summaries and checks of the imputation procedure.
Common mistake
Wrong reasoning: “The pooled imputed distribution is close to the observed distribution, so every chain must be close.” The cancellation example shows why this fails: large opposing chain-specific differences can produce an average difference of zero. Correction: inspect each chain against the same observed reference before drawing a conclusion from a pooled plot. Also avoid the reverse overclaim: a difference in one panel does not prove that chain is defective. Check the counts, display choices, and plausible causes, and treat the plot as a diagnostic clue rather than a formal test.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why should the observed reference and plotting bins remain the same across chain-specific panels?
- For an observed bin proportion of \(0.30\) and chain proportions \(0.20, 0.29,\) and \(0.31\), calculate each chain’s difference and describe the pattern.
- Three chains have differences \(-0.08, 0,\) and \(0.08\) in one bin. What is their average difference, and why could that average mislead?
- What does a pooled plot retain, and what important information can it lose?
- Does a similar observed–imputed difference in every chain prove the imputations are invalid? Explain.
Solutions. (1) A common reference and bins make panel differences comparable rather than artifacts of changing baselines or display choices. (2) The differences are \(0.20-0.30=-0.10\), \(0.29-0.30=-0.01\), and \(0.31-0.30=0.01\). The first chain has a substantial deficit in that bin; the other two are close to the observed proportion. (3) The average is \((-0.08+0+0.08)/3=0\). It can mislead because the first and third chains depart in opposite directions. (4) Pooling retains an overall summary of the imputed values but can lose which chain produced a value and conceal chain-specific or opposing patterns. (5) No. The pattern is a descriptive diagnostic; selection into observed versus missing groups, model features, limited data, or chance may contribute, and the plot alone does not establish validity or invalidity.