Checking Random-Effects Variance Through Replication

Why Check Random-Effects Variance Through Replication?

A random-effects variance describes how much clusters differ in their latent effects, but those effects are not observed directly. Instead, each cluster supplies outcomes subject to within-cluster sampling variation. A cluster with a small sample can have an unusual observed rate even when its latent effect is ordinary; conversely, shrinkage can make estimated cluster effects look less dispersed than the underlying effects. A useful diagnostic therefore asks whether outcomes generated by the fitted model show cluster-to-cluster variation like the variation actually observed.

The check is a predictive comparison. Fit the GLMM, calculate a statistic describing cluster-level spread in the observed data, and calculate the same statistic for data replicated under the fitted model. If the observed statistic lies far into a tail of the replicated values, that is evidence that the fitted model does not reproduce this aspect of cluster heterogeneity. It is not, by itself, proof that the random-effects variance alone is wrong: fixed effects, the random-effects distribution, dependence assumptions, and the outcome sampling model can all affect the comparison.

A Cluster-Level Discrepancy

For a binary outcome, suppose cluster \(i\) has \(n_i\) observations and \(Y_i\) successes. Its observed proportion is \(\hat{p}_i=Y_i/n_i\). Let \(m_i\) be the fitted marginal mean for that cluster: the model-based expected proportion after averaging over the fitted random-effects distribution. For a random-intercept logistic model, for example, \(m_i\) is the expectation of the inverse-logit probability over a random intercept with the fitted variance. It can be calculated by numerical integration or simulation.

One simple discrepancy is the mean squared deviation of cluster proportions from their fitted marginal means:

where \(K\) is the number of clusters. The denominator \(K-1\) provides a convenient scale; it does not make this statistic an unbiased estimator of the random-effects variance. In a replicate, use its simulated cluster proportions and the corresponding fitted means, with the design and model parameters handled consistently. With covariates, \(m_i\) can differ across clusters, so comparing raw proportions to one common average would confuse fitted mean differences with unexplained heterogeneity.

For counts, an analogous statistic can use cluster-specific rates, with exposure handled in the fitted means and the same exposure values retained in each replicate. The precise discrepancy should match the outcome and design. Whatever the choice, the replicated distribution is essential: it reflects both the latent cluster-to-cluster variation and the within-cluster noise expected under the fitted model.

Why Replication Includes Sampling Variation

The distinction between latent heterogeneity and observed spread follows from the law of total variance. For a binomial cluster with common size \(n\), let \(p_i(b_i)\) be its success probability conditional on random effect \(b_i\), and let \(\hat{p}_i=Y_i/n\). Conditional on \(b_i\), the variance of the proportion is \(p_i(b_i)\{1-p_i(b_i)\}/n\), and its conditional mean is \(p_i(b_i)\). Therefore:

The first term is variation among the cluster probabilities induced by the random effects. The second is binomial sampling variation within clusters. A statistic formed from observed proportions contains both contributions. Replicating outcomes under the fitted model naturally includes both; comparing observed spread directly with the fitted random-effects variance would not.

This identity also explains why unequal cluster sizes matter. The sampling component is generally larger for smaller \(n_i\). A check that ignores the actual cluster sizes can make a reasonable fitted variance look too small or too large. Replicates should retain the observed cluster sizes, covariate patterns, and exposures relevant to the model.

Constructing and Interpreting the Replicated Distribution

For a random-intercept model, an unconditional parametric replication draws a new intercept for every cluster from the fitted random-effects distribution, then draws outcomes from the fitted conditional outcome distribution. For a binary GLMM, this means drawing the random effects and then drawing binomial outcomes using the resulting probabilities. The same cluster sizes and covariates are used as in the observed data. This procedure asks whether the fitted population distribution of random effects, together with the rest of the model, can reproduce the observed cluster spread.

For each replicate \(b\), calculate \(T^{(b)}\) exactly as \(T_{\mathrm{obs}}\) was calculated. An estimated upper-tail predictive proportion is:

where \(I(\cdot)\) equals 1 when its statement is true and 0 otherwise. A small upper-tail proportion indicates that replicated data rarely have at least as much spread as observed. A correspondingly small lower-tail proportion indicates that the observed spread is unusually low under the fitted model. The added 1 in numerator and denominator avoids reporting a simulated tail proportion of exactly zero; it does not turn the check into a calibrated hypothesis test.

When model parameters are treated as fixed at their estimates, this is a plug-in predictive check. Parameter uncertainty can be incorporated by drawing parameters from a fitted posterior distribution or by using a parametric bootstrap that refits the model in each simulated data set. These approaches answer related but not identical questions. The replication scheme and whether the model is refitted should be reported because both affect the reference distribution.

Worked Model-Based Check

Choosing Among Diagnostic Comparisons

The replication check is most informative when the statistic isolates the feature of interest and the simulation reproduces the fitted design. Several alternatives can be useful, but they answer different questions.

A variance of estimated cluster effects may appear to be a direct check of the random-effects variance, but the two quantities are not interchangeable. Estimated effects are informed by finite, noisy cluster data and are commonly shrunk toward the population mean. The replicated-outcome approach instead asks whether the full fitted model reproduces an observable implication of the variance component. It does not remove the need to examine the random-effects distribution shape, outcome fit, and influential clusters.

Diagnostics, Failure Modes, and Limits

A very large upper-tail discrepancy suggests that observed clusters are more separated than the fitted model typically produces. A very small lower-tail discrepancy suggests the model tends to produce more separation than observed. Either finding calls for investigation, not an automatic change to the variance estimate. Check whether the statistic was computed consistently, whether cluster sizes and covariates were retained, and whether a few clusters dominate the statistic. Cluster-level influence diagnostics can help determine whether the pattern is driven by one or two clusters.

A common mistake is to compare the observed variance of cluster proportions directly with \(\hat{\tau}^2\), then declare the random-effects variance too small if the former is larger. These quantities are on different scales: proportions include binomial sampling variation, while \(\hat{\tau}^2\) is variance on the latent random-effect scale. The correction is to compare an observed-data discrepancy with the same discrepancy from data simulated under the fitted model, which carries sampling variation and the link function through the calculation.

• Unequal cluster sizes: Small clusters have noisier proportions. Preserve their sizes in every replicate.
• Inadequate statistic: A single overall spread measure can miss specific patterns, such as a few extreme clusters. Use a complementary discrepancy when the scientific concern calls for it.
• Parameters estimated from the data: A plug-in reference distribution may not fully reflect parameter uncertainty. State whether parameters were fixed, drawn, or re-estimated.
• Small numbers of clusters: Replication cannot create information about the random-effects distribution that the data do not contain. Tail proportions may be imprecise, and the fitted variance itself may be weakly determined.
• Model misspecification beyond the variance: A tail result can reflect an incorrect mean structure, random-effects distribution, or conditional outcome model, not only a misestimated variance.

Predictive tail proportions are diagnostic summaries, not conventional p-values with a guaranteed reference distribution under a null hypothesis. A moderate tail proportion is not evidence that every model assumption holds, and a small one does not identify a unique correction. The result should be interpreted alongside the model specification and other checks of outcome fit.