Why Check Both Levels?
In clustered data, a set of imputations can look reasonable in a pooled histogram and still distort the structure that matters. Imputed values might pull cluster means toward one another, or preserve those means while making individuals within each cluster too similar. Either pattern can affect analyses of cluster differences, individual-level associations, and uncertainty.
Diagnostics should therefore ask two distinct questions: do the imputations produce plausible values for individuals within their cluster, and do they preserve plausible differences between clusters? These are targeted checks, not tests that can reveal the unobserved values with certainty. The missing values are unknown; agreement between observed and imputed summaries is evidence to examine, not proof that an imputation model is correct.
A useful working model for a continuous outcome is the random-intercept model
This model assumes that cluster effects \(u_j\) are independent across clusters, individual deviations \(e_{ij}\) are independent across individuals conditional on the cluster effects, and the two components are independent with mean zero. It describes a setting in which clusters differ in their average level and individuals vary around that level. The between-cluster variance is \(\tau^2\), the within-cluster variance is \(\sigma^2\), and the intraclass correlation is
These quantities are useful diagnostic summaries when the model is suitable; they are not universal targets. A model with random slopes, additional levels, or covariates may need more specific checks. The assumptions should also be assessed in context: for example, strong skewness or unequal within-cluster spreads may make a normal random-intercept model an inadequate description.
Cluster Means: Compare Carefully
For cluster \(j\), calculate the observed mean using only its observed values, and calculate a completed-data mean separately in each imputed data set. The observed mean is based on \(n_{j,\mathrm{obs}}\) values, whereas a completed-data mean is based on all \(n_j\) individuals. If the observed cases are not representative of the missing cases within that cluster, the two means can differ even when the imputations are reasonable. Thus, do not treat a difference as an automatic failure.
Look for patterns across clusters and imputations. Do imputations systematically shrink cluster means toward the grand mean? Are large shifts concentrated in clusters with very few observed values? Do means change substantially from one imputed data set to another? Interpret these patterns with cluster size, observed counts, and relevant covariates in view. Where cluster membership and repeated observations matter, summaries should be formed at the level that matches the data structure.
Worked Example: Comparing Observed and Completed Cluster Means
Consider three hypothetical clusters, each with four individuals. The values below are in outcome units; “missing” denotes a value imputed once in a completed data set.
| Cluster | Observed values | Imputed value |
|---|---|---|
| A | 8, 10, 12 | 10 |
| B | 14, 16, 18 | 16 |
| C | 20, 22, 24 | 18 |
For cluster A, the observed mean is \((8+10+12)/3=10\). The completed mean is \((8+10+12+10)/4=40/4=10\). For cluster B, the observed mean is \((14+16+18)/3=16\), and the completed mean is \((14+16+18+16)/4=64/4=16\). For cluster C, the observed mean is \((20+22+24)/3=22\), while the completed mean is \((20+22+24+18)/4=84/4=21\).
The one-unit difference in cluster C is not, by itself, evidence of a defective imputation: the imputed value is below the observed mean, so it lowers the completed mean. It is a prompt to check whether a value of 18 is plausible for a person in cluster C given the predictors and the imputation model. Across multiple imputations, also examine how variable this mean is. The comparison is more concerning if many high-mean clusters are pulled downward in a consistent way without a substantive explanation.
Within-Cluster Spread and Predictive Checks
Means alone cannot show whether individual-level variability has been preserved. Compare within-cluster distributions using suitable summaries such as standard deviations, quantiles, ranges, and plots. Inspect these by cluster or by meaningful strata of cluster-level and individual-level predictors. A pooled comparison can hide a model that works in some clusters but fails in others.
Observed values are not a perfect reference for imputed values: the observed cases may differ systematically from the missing cases. A particularly direct diagnostic is to temporarily mask some observed values, impute them using the intended procedure, and compare the predictions with the values that were hidden. This validation check treats known values as a stand-in for missing ones, so its relevance depends on whether the masking and observed cases resemble the actual missingness process. Compare both the predictive center and the predictive spread; a procedure that predicts the conditional mean accurately but supplies almost no uncertainty may still be inadequate.
Worked Example: Detecting Collapsed Within-Cluster Variation
Suppose, for a hypothetical validation check within a particular cluster and covariate stratum, four observed values are temporarily hidden. Their known values are 8, 10, 12, and 14 outcome units. A deterministic prediction procedure returns 10 for each hidden value.
The mean of the known values is \((8+10+12+14)/4=44/4=11\). Their sample variance is
The four predictions have mean 10 and sample variance \(\{(10-10)^2+(10-10)^2+(10-10)^2+(10-10)^2\}/3=0\). Their mean is one outcome unit below the known-value mean, and their predictive spread is zero rather than 6.667 square outcome units. This is a clear sign that the procedure supplies no individual-level variation in this check. It does not establish that the true missing values have exactly the observed variance, but it demonstrates why matching a mean alone is insufficient. A stochastic imputation model should represent predictive uncertainty rather than assigning every case its fitted mean.
Variance Components as a Targeted Summary
For balanced clusters with \(n\) individuals each, the random-intercept model has a simple method-of-moments calculation. Let \(MS_W\) be the within-cluster mean square and \(MS_B\) the between-cluster mean square from the one-way analysis of variance decomposition. Under the model assumptions, the expected within mean square is \(\sigma^2\), and the expected between mean square is \(\sigma^2+n\tau^2\). Subtracting the first expectation from the second and dividing by \(n\) gives the usual moment estimate of the between-cluster variance:
The justification is that cluster means contain both the cluster effect and the average of \(n\) individual errors. The latter has variance \(\sigma^2/n\), so the variance of a cluster mean is \(\tau^2+\sigma^2/n\). Multiplying this by \(n\), as in the between mean square, yields \(\sigma^2+n\tau^2\). This derivation relies on equal cluster sizes and the independent, equal-variance random-intercept model. With unequal sizes or a more complex structure, use an appropriate mixed model rather than applying this balanced formula mechanically. A negative moment estimate can occur from sampling variability; it is not a negative population variance and is commonly constrained to zero in a variance-component fit.
Worked Example: Comparing Variance Components Across Completed Data Sets
Consider three hypothetical clusters with three completed values each. In completed data set 1, the values are \(8,10,12\), \(13,15,17\), and \(18,20,22\). The cluster means are 10, 15, and 20, and the grand mean is 15.
Within each cluster, the sum of squared deviations from its mean is \(4+0+4=8\). Across three clusters, the within sum of squares is \(24\), with \(9-3=6\) degrees of freedom, so \(MS_W=24/6=4\) square outcome units. The between sum of squares is \(3\{(10-15)^2+(15-15)^2+(20-15)^2\}=3(25+0+25)=150\). With \(3-1=2\) degrees of freedom, \(MS_B=150/2=75\).
Here \(n=3\), so \(\widehat{\sigma}^2=4\) and \(\widehat{\tau}^2=(75-4)/3=71/3\approx23.67\) square outcome units. The estimated ICC is \(23.67/(23.67+4)\approx0.855\).
Now consider completed data set 2 with values \(9,10,11\), \(14,15,16\), and \(19,20,21\). Its cluster means and grand mean are unchanged, so \(MS_B=75\). Each cluster has a within sum of squares of \(1+0+1=2\), giving \(MS_W=6/6=1\). Thus \(\widehat{\sigma}^2=1\), \(\widehat{\tau}^2=(75-1)/3=24.67\), and the estimated ICC is \(24.67/(24.67+1)\approx0.961\).
This comparison shows that nearly identical cluster means can coexist with a substantial change in within-cluster spread and the ICC. If the second data set arose from imputations, investigate why its values are more tightly concentrated. The estimates are diagnostic summaries, not proof that one data set is correct; in practice, calculate suitable model-based variance components for each imputed data set and inspect their distributions across imputations.
Practical Workflow, Alternatives, and Limitations
A useful diagnostic sequence separates levels and compares like with like. First, summarize observed counts and missingness by cluster. Second, compare observed and completed cluster means, noting how much observed information supports each mean. Third, inspect within-cluster spreads and conditional distributions, including any masked-value validation that is credible for the setting. Finally, compare variance components and their stability across imputations. Review plots and numerical summaries together; a single global statistic can conceal a local failure.
Record cluster sizes, observed counts, and observed summaries. Flag clusters with very sparse observed data, for which their observed means and spreads are imprecise.
Compare cluster-level summaries and within-cluster distributions across completed data sets, with attention to covariates and missingness patterns that affect comparability.
Look for systematic shrinkage, implausible within-cluster compression, extreme imputed values, and unstable variance components. Check the model specification and whether its assumptions are plausible.
Where possible, combine observed-data comparisons with masked-value checks and substantive review. No single diagnostic identifies the true missing values.
A pooled histogram is a useful alternative when the main concern is the overall marginal distribution, but it does not show whether values are assigned to the right clusters. An observed-only analysis avoids imputations but discards incomplete records and may not describe the full clustered population. Model-based predictive checks assess whether a specified model can reproduce relevant features, while sensitivity analysis examines how conclusions change under alternative assumptions about missingness. These approaches answer different questions; none makes the within- and between-cluster checks redundant.
There are important limitations. A cluster with few observed values provides little information about its own within-cluster distribution, and its completed mean can be strongly influenced by the model. Even when observed and imputed summaries align, a misspecified model can reproduce those summaries while distorting other features, such as tails or associations. Conversely, an observed-versus-imputed difference can arise because missingness depends on observed predictors or because the completed mean includes individuals excluded from the observed mean. Interpret diagnostic differences in context, and do not use an arbitrary cutoff for the ICC or for a difference in means as a universal pass/fail rule.
Common Mistakes
- Comparing means as if they used the same cases. The observed mean excludes missing cases; the completed mean includes them. A difference can be expected, so examine its direction, context, and consistency across imputations.
- Checking only the pooled distribution. A plausible pooled histogram can conceal compressed within-cluster variation or distorted differences between clusters. Stratify summaries by cluster or relevant predictors.
- Treating an ICC as a quality score. The ICC depends on both variance components and the data structure. A high or low value is not, by itself, evidence that the imputation model is good or bad.
- Calling observed-only summaries the truth. They describe observed cases. If observed and missing cases differ, those summaries may not represent the missing values or the full cluster.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A cluster has observed values 4, 6, and 8, and one missing value imputed as 10. Calculate its observed and completed means. Does the difference alone show a failed imputation?
- In a masked-value check, the known values are 2, 4, and 6, while the imputed predictions are 4, 4, and 4. Calculate the two sample variances and identify the diagnostic concern.
- For balanced clusters with \(n=2\), suppose \(MS_W=3\) and \(MS_B=11\). Calculate the moment estimates of \(\sigma^2\) and \(\tau^2\).
- Why can a plausible pooled histogram fail to detect an important problem in clustered imputations?
Solutions. 1. The observed mean is \((4+6+8)/3=6\); the completed mean is \((4+6+8+10)/4=28/4=7\). The difference alone does not establish failure because the completed mean includes an imputed value above the observed mean. 2. The known-value mean is 4, and its sample variance is \(\{(2-4)^2+(4-4)^2+(6-4)^2\}/2=8/2=4\). The predictions have variance 0, indicating collapsed predictive spread in this check. 3. \(\widehat{\sigma}^2=MS_W=3\), and \(\widehat{\tau}^2=(MS_B-MS_W)/n=(11-3)/2=4\), in squared outcome units. 4. Pooling can hide cluster-specific problems: the overall distribution may look plausible even if values are too similar within clusters or cluster means have been pulled toward one another.