Shrinkage And Cluster-Specific Predictions

Why Cluster-Specific Predictions Are Shrunk

In multilevel imputation, a cluster with only a few observed outcomes can have a highly variable estimated cluster effect. Treating that estimate as if it were known can make predictions for missing values too extreme. At the other extreme, ignoring all cluster information can erase genuine differences between clusters. A random-intercept model balances these sources of information: it lets each cluster have its own effect, while pulling uncertain cluster effects toward the population mean. This is partial pooling, and the pull is called shrinkage.

The amount of shrinkage is not a fixed correction applied equally to every cluster. It depends on the number of observed outcomes in that cluster and on the relative sizes of within-cluster and between-cluster variation. Consequently, predictions for a small cluster are typically pulled more strongly toward the overall mean than predictions for a large cluster.

Model and Assumptions

Consider a continuous outcome \(Y_{ij}\) for individual \(i\) in cluster \(j\). Conditional on included covariates, write the model as

Here \(\boldsymbol{x}_{ij}^{\mathsf T}\boldsymbol{\beta}\) is the fixed part of the conditional mean, \(u_j\) is the cluster-specific random intercept, and \(e_{ij}\) is individual-level variation. The standard calculation below assumes that the random effects are independent across clusters, the residuals are independent across individuals conditional on the random effects, and the random effects and residuals are independent. It also assumes that the specified normal distributions and variance structure are adequate for the conditional outcome model. For this derivation, treat \(\boldsymbol{\beta}\), \(\tau^2\), and \(\sigma^2\) as known; uncertainty in these quantities must also be represented in a proper multiple-imputation procedure.

For a given cluster, subtract the fixed part from each observed outcome. Let \(r_{ij}=Y_{ij}-\boldsymbol{x}_{ij}^{\mathsf T}\boldsymbol{\beta}\) be this residual from the fixed part, and let \(n_j\) be the number of observed outcomes in cluster \(j\). Under the model, \(r_{ij}=u_j+e_{ij}\). The shrinkage calculation uses the cluster’s residual mean, \(\bar r_j\), rather than its raw outcome mean when covariates are present.

Deriving the Shrinkage Estimate

The observed residuals in a cluster provide information about \(u_j\), while the random-effects distribution gives a normal prior centered at zero with variance \(\tau^2\). Combining these two sources gives a normal conditional distribution for the cluster effect. Its mean is the shrinkage estimate, and its variance measures remaining uncertainty about that effect.

The multiplier \(B_j=n_j\tau^2/(\sigma^2+n_j\tau^2)\) is the shrinkage factor. Because \(0\leq B_j<1\), the conditional mean \(B_j\bar r_j\) lies between zero and the unpooled cluster residual mean. It approaches zero when the cluster supplies little information and approaches \(\bar r_j\) as the cluster becomes large relative to the variance ratio.

The formula follows by multiplying the normal likelihood for the residuals by the normal distribution of \(u_j\). Ignoring factors that do not depend on \(u_j\), the log of that product is

The terms involving \(u_j\) have quadratic coefficient \(\tau^{-2}+n_j\sigma^{-2}\) and linear coefficient \(\sum_i r_{ij}\sigma^{-2}\). Completing the square therefore gives posterior variance \((\tau^{-2}+n_j\sigma^{-2})^{-1} =\tau^2\sigma^2/(\sigma^2+n_j\tau^2)\) and posterior mean equal to that variance times \(\sum_i r_{ij}/\sigma^2\). Since \(\sum_i r_{ij}=n_j\bar r_j\), the mean is \(B_j\bar r_j\), as stated. If \(n_j=0\), the data provide no cluster-specific information, so the conditional mean is zero and the conditional variance remains \(\tau^2\).

Cluster Size Changes the Amount of Shrinkage

The shrinkage factor increases with \(n_j\), holding the variance components fixed. For a large cluster, its observed residual mean is measured more precisely, so the data receive more weight relative to the common distribution of cluster effects. In a small cluster, a high or low residual mean could reflect substantial sampling variation, so the estimate is pulled more toward zero. The conditional variance of the cluster effect also decreases as \(n_j\) grows.

Using Shrinkage in Imputation

The conditional mean is useful for understanding the expected value of a missing outcome, but it is not, by itself, a multiple imputation. If every missing value is replaced by its conditional mean, the imputed values have no residual variation around those means. They also fail to reflect uncertainty about the cluster effects. This can make completed data sets artificially alike and can understate uncertainty in subsequent analyses.

With parameters treated as known, an imputation for one missing outcome can be generated by first drawing \(u_j\) from its conditional distribution given the cluster’s observed outcomes, then drawing a new residual from \(N(0,\sigma^2)\), and adding both to the fixed part. In proper MICE, the parameter values should also be drawn or otherwise their uncertainty incorporated, consistently with the multilevel imputation model. This tutorial’s formulas describe the cluster-effect part of that calculation, not the full parameter-updating procedure.

Alternatives, Diagnostics, and Limitations

Partial pooling is a middle course between two simpler approaches. Full pooling uses one common mean and assigns no cluster-specific effects. No pooling estimates each cluster mean separately, without borrowing information across clusters. A random-intercept model uses cluster-specific information but moderates unstable estimates, which is especially relevant when cluster sizes vary.

Shrinkage is not a remedy for a poorly specified model. Examine whether residual patterns differ systematically by cluster, whether the assumed within-cluster variation is plausible, and whether the estimated between-cluster variation is consistent with the observed data. Compare observed and imputed outcome distributions within clusters and across cluster sizes. If predictions for small clusters are all nearly identical, check that the imputation procedure is drawing appropriate uncertainty rather than inserting only conditional means. If large clusters remain excessively pulled toward the overall mean, check the model specification and variance components.

The Gaussian random-intercept formula is not automatically appropriate for binary, ordinal, count, or strongly skewed outcomes; those require conditional models suited to their outcome type. It also relies on the assumption that clusters are adequately represented by a common random-effects distribution. With very few clusters, the estimated variance components can be unstable; with covariates omitted from the imputation model, conditional predictions can be systematically wrong. Finally, the calculation assumes the fixed and variance parameters are known. In actual multiple imputation, their uncertainty and the missingness assumptions required for valid inference remain important.

Common Mistake

A common mistake is to say, “The small cluster’s mean residual is 10 units, so every missing outcome in that cluster should be imputed as 60 units.” That uses the unpooled cluster mean and ignores both shrinkage and predictive variation. Under the example’s model, the conditional mean for a new outcome in the small cluster is about 54.71 units, not 60. Even 54.71 is an expected value, not a suitable deterministic imputation: a proper imputation also draws uncertainty in the cluster effect and individual residual, and accounts for parameter uncertainty.