Why Quantify an Observed–Imputed Gap?
Distribution plots can reveal important features of observed and imputed values, but visual comparisons do not always make the size of a discrepancy easy to summarize. A numerical gap can help describe the comparison, track it across variables or iterations, and communicate which features differ. The useful measure depends on what matters: a difference in average level, a shift in particular parts of a distribution, or a change in category proportions.
In a MICE diagnostic, “observed” values are the available recorded values of a variable, while “imputed” values are the replacements generated for its missing entries. These are different subsets of cases, not two measurements on every person. A numerical gap therefore describes the two sets of values being compared; by itself, it does not say whether their distributions ought to be identical or whether the imputation model is adequate.
For a continuous variable, three useful summaries are a standardized mean difference, differences between selected quantiles, and (after categorizing the variable) total variation between category distributions. Each captures something different. No one number describes every feature of a distribution.
Standardized Mean Difference
Let \(n_o\) and \(n_m\) be the numbers of observed and imputed values included in a comparison. Let \(\bar{x}_o\) and \(\bar{x}_m\) be their sample means, and let \(s_o\) and \(s_m\) be their sample standard deviations. A standardized mean difference (SMD) expresses the difference between the means in units of a pooled within-group standard deviation:
The sign records direction: a positive SMD means the imputed values have the larger mean, and a negative SMD means the observed values have the larger mean. The magnitude is dimensionless, so it does not depend on whether the same variable is recorded in, for example, centimetres or metres. This pooled standard deviation is a descriptive scaling choice. Its calculation does not require the two groups to have equal variances, though a very different spread in the groups is useful information that the SMD does not show.
Worked Example: Standardizing a Difference in Means
Suppose the observed values, measured in points, are 10, 12, 14, 16, and 18. The imputed values in a specified completed data set are 12, 13, 15, and 17 points. Calculate the SMD using the pooled standard deviation.
The observed mean is \(\bar{x}_o=(10+12+14+16+18)/5=70/5=14\) points. The sum of squared deviations from 14 is \(16+4+0+4+16=40\), so \(s_o^2=40/(5-1)=10\) points squared.
The imputed mean is \(\bar{x}_m=(12+13+15+17)/4=57/4=14.25\) points. Its squared deviations from 14.25 sum to \(5.0625+1.5625+0.5625+7.5625=14.75\). Thus \(s_m^2=14.75/(4-1)=4.9167\) points squared.
The pooled variance is \(\{(5-1)(10)+(4-1)(4.9167)\}/(5+4-2)=54.75/7=7.8214\) points squared. Therefore \(s_{\mathrm{pool}}=\sqrt{7.8214}=2.797\) points, and
The imputed mean is about 0.089 pooled standard deviations above the observed mean in this comparison. That small standardized mean gap does not establish that the full distributions are similar: their shapes or tails could still differ.
The pooled scale must be positive for this SMD to be defined. If both groups have no variation, it is zero and the ratio cannot be calculated. A very small pooled standard deviation can also produce a large SMD from a modest raw difference, so report the means and spread alongside the standardized value when practical. A standardized gap is not a p-value, and a chosen numerical threshold is not a universal pass/fail rule for MICE.
Quantile Differences
Means are sensitive to extreme values and can conceal differences in skewness or tails. Quantile differences directly compare specified locations in the distributions. For a probability level \(p\), let \(Q_o(p)\) and \(Q_m(p)\) denote the empirical \(p\)-quantiles of the observed and imputed values. Use the same quantile definition for both groups and calculate
For clarity, one possible convention is the empirical inverse distribution function: sort the \(n\) values and take the value at rank \(\lceil np\rceil\), for \(0<p\leq1\). Other software may interpolate between order statistics, so record the convention when exact numerical reproducibility matters. Quantile differences retain the units of the variable. A positive difference at a chosen \(p\) means the imputed \(p\)-quantile is higher.
Worked Example: Comparing the Median and Upper Quartile
Consider observed values of a hypothetical measurement, in centimetres: 2, 4, 5, 7, 9, 12. The imputed values are 3, 4, 6, 8, 10, 14 cm. Use the empirical inverse distribution function with rank \(\lceil np\rceil\).
At \(p=0.50\), each sample has six values, so the rank is \(\lceil6(0.50)\rceil=3\). The observed median is 5 cm and the imputed median is 6 cm; hence \(\Delta_Q(0.50)=6-5=1\) cm.
At \(p=0.75\), the rank is \(\lceil6(0.75)\rceil=5\). The observed quantile is 9 cm and the imputed quantile is 10 cm; hence \(\Delta_Q(0.75)=10-9=1\) cm. These two comparisons describe a 1 cm upward difference at each selected location, but say nothing by themselves about the rest of the distributions.
A set of quantile differences can help identify where a discrepancy occurs: for example, the median might be similar while an upper-tail quantile differs. Quantile summaries are less driven by the numerical magnitude of a single extreme observation than means are, but they depend on the sample size and quantile convention. Sparse tails can make extreme quantiles especially unstable. State which probability levels were compared rather than reporting an unexplained “quantile gap.”
Categorical Total Variation
For a categorical variable with \(K\) categories, let \(p_{o,k}\) and \(p_{m,k}\) be the observed and imputed proportions in category \(k\), respectively. Each set of proportions is nonnegative and sums to one. The total variation distance (TV) is half the sum of the absolute category-proportion differences:
TV is between 0 and 1: zero means the category proportions match, and larger values indicate greater discrepancy. It is dimensionless. In a binary variable, if the proportion in the first category differs by \(d\), the proportion in the second category differs by \(-d\); the two absolute differences sum to \(2|d|\), so TV equals \(|d|\), not half of \(|d|\).
Worked Example: Total Variation for a Binary Variable
Suppose 25 of 40 observed records are in category A and 15 are in category B. Among 40 imputed records, 22 are in category A and 18 are in category B. The observed proportions are \(p_{o,A}=25/40=0.625\) and \(p_{o,B}=15/40=0.375\). The imputed proportions are \(p_{m,A}=22/40=0.550\) and \(p_{m,B}=18/40=0.450\).
The category differences are \(0.550-0.625=-0.075\) and \(0.450-0.375=0.075\). Therefore,
The total variation distance is 0.075, or 7.5 percentage points of probability mass. For this binary variable it also equals the absolute difference between the proportions in either one of the categories. It is not a test of whether the proportions are equal.
For any two categorical distributions, TV also equals the largest absolute difference in probability assigned to any event (any collection of categories). To see why, set \(d_k=p_{m,k}-p_{o,k}\). Since both distributions sum to one, \(\sum_k d_k=0\); the sum of positive \(d_k\) values therefore equals the absolute sum of negative \(d_k\) values. Each equals half of \(\sum_k|d_k|\). Taking the event consisting of categories with positive \(d_k\) achieves that difference, and no event can exceed the total positive mass. Thus the formula summarizes the largest event-level probability discrepancy.
Choosing and Checking a Gap Measure
| Measure | Captures | Units | Important limitation |
|---|---|---|---|
| Standardized mean difference | Difference in means relative to a pooled spread | None | Can conceal shape, tail, or spread differences |
| Quantile difference | Difference at a specified distribution location | Original variable units | Depends on the selected quantile and its estimation convention |
| Total variation | Largest discrepancy in category-event probabilities | None; between 0 and 1 | Requires categories and can depend on how a continuous variable is categorized |
A raw mean or quantile difference is a useful alternative when the original units are substantively meaningful and the scale itself matters. Standardizing a mean difference can aid comparisons across variables measured on different scales, but it sacrifices direct interpretation in those units. For categorical variables, reporting each category proportion difference alongside TV makes the direction and location of the discrepancy visible. These measures complement distribution plots; they do not replace them.
Calculate a gap for a clearly defined comparison, such as a specified chain and iteration or a specified completed data set. Keep the observed and imputed subsets, variable coding, and summary conventions fixed when comparing gaps over time. Check the underlying summaries and plots if a result seems surprising. A single number can obscure multimodality, rounding, unequal spreads, or subgroup differences. Earlier material on comparing distribution plots and stratifying observed–imputed comparisons addresses these features visually and by subgroup.
Most importantly, observed cases and cases with missing values may differ systematically. Their marginal distributions need not match even when an imputation procedure is working as intended. A nonzero gap is therefore a descriptive signal to interpret in context, not automatic evidence of failure; a small gap is not proof of validity either. These calculations do not account for uncertainty across imputations or provide hypothesis tests.
Common Mistakes and Limitations
- Using half the binary proportion difference as TV. The two category differences have equal absolute size, so the factor of one-half in the sum cancels the duplication. For proportions 0.625 versus 0.550 in category A, TV is 0.075.
- Calling an SMD a test statistic. It is a descriptive, standardized difference. It does not supply a p-value or determine whether the null hypothesis should be rejected.
- Reporting a quantile difference without its location or units. State both the probability level, such as the median, and the variable’s units.
- Assuming matching means prove matching distributions. Different shapes can share the same mean. Examine quantiles and distribution plots as well.
- Treating a gap as a model verdict. The observed and missing cases can differ, and a gap alone cannot separate that selection from model problems.
Check Your Understanding
Answer each question using the definitions and conventions above. Brief solutions follow the questions.
- Observed values have mean 8 and standard deviation 2; imputed values have mean 9 and standard deviation 2. If the pooled standard deviation is 2, calculate the SMD and state its direction.
- With the empirical inverse distribution function, what rank defines the \(p=0.75\) quantile for \(n=8\)?
- For a binary variable, observed proportions are 0.40 and 0.60, while imputed proportions are 0.46 and 0.54. Calculate TV.
- Can an SMD of zero establish that the observed and imputed distributions are identical? Explain.
- Why should TV not be interpreted as a hypothesis test?
Solutions. (1) \(\mathrm{SMD}=(9-8)/2=0.500\), positive, so the imputed mean is higher. (2) The rank is \(\lceil8(0.75)\rceil=\lceil6\rceil=6\). (3) \(\mathrm{TV}=\tfrac12(|0.46-0.40|+|0.54-0.60|)=\tfrac12(0.06+0.06)=0.060\). (4) No. Equal means can coexist with different spreads, shapes, or tails. (5) TV is a descriptive distance between proportions; it does not model sampling uncertainty or specify a null distribution, so it does not yield a p-value.