Raw Group Deviations Versus Shrunk Predictions

Why Raw Group Deviations and Predictions Differ

A group’s observed mean can differ from the overall mean for two reasons: the group may truly have a different latent effect, and its observations may contain sampling noise. A raw deviation records the observed difference without separating those sources. In a Gaussian random-effects model, an empirical Bayes prediction accounts for both: it retains more of the observed difference when the group provides substantial information, and shrinks it toward zero when the observed difference is relatively noisy.

The distinction matters when interpreting group estimates. A raw deviation is a direct summary of the group’s observed data relative to a reference mean. A shrunk prediction estimates the latent group effect under a model that combines the group’s data with the assumed distribution of effects across groups. Neither quantity is universally preferable: the choice depends on the question and the model’s adequacy.

Model and Definitions

Consider a Gaussian random-intercept model with \(J\) groups. For group \(j\), observation \(i\) follows

Assume the group effects are mutually independent; the errors are mutually independent; and all errors are independent of all group effects. Assume \(\tau^2>0\) and \(\sigma^2>0\). For the derivation and numerical examples below, take the common mean \(\beta_0\) and the variance components as known. This isolates the difference between the raw and shrunk quantities. In practical empirical Bayes analysis, these parameters are usually estimated and substituted into the prediction.

Under this model, \(d_j=b_j+\bar{\varepsilon}_j\), where \(\bar{\varepsilon}_j\) has variance \(\sigma^2/n_j\). Thus a raw deviation contains the latent effect plus sampling noise. The model-based prediction estimates \(b_j\), not the full observed deviation.

Deriving the Shrinkage Factor

The group mean supplies a noisy measurement of \(b_j\): conditional on \(b_j\), \(d_j\sim N(b_j,\sigma^2/n_j)\). The prior distribution across groups is \(b_j\sim N(0,\tau^2)\). Multiplying these normal densities gives a conditional normal distribution. To see its mean, collect the terms involving \(b_j\) in the log density:

Completing the square identifies the conditional variance as the reciprocal of the coefficient on \(b_j^2\), and the conditional mean as that variance multiplied by the coefficient on \(b_j\):

The conditional variance is \(\tau^2\sigma^2/(\sigma^2+n_j\tau^2)\). Since \(0<\lambda_j<1\), the prediction lies between zero and the raw deviation. The shrinkage is toward zero because the model assumes that group effects are centered at zero, corresponding to the common mean \(\beta_0\). It is not an arbitrary adjustment: it is the conditional mean under the stated Gaussian model.

How Sample Size and Variances Control the Difference

The shrinkage factor can be read as a comparison of between-group variation with the total variation in the observed group deviation:

A larger \(n_j\) reduces the sampling variance \(\sigma^2/n_j\), increasing \(\lambda_j\). A larger between-group variance \(\tau^2\) also increases \(\lambda_j\), because the model then allows substantial real differences between groups. Conversely, large within-group variance \(\sigma^2\), small \(n_j\), or small \(\tau^2\) makes the observed deviation less compelling as evidence of a distinct latent effect. For fixed variance components, groups with different sample sizes have different shrinkage factors even if their raw deviations are identical.

Comparison, Diagnostics, and Limitations

The raw deviation, empirical Bayes prediction, and complete-pooling estimate answer different questions. The raw deviation is unpooled: it uses the group mean alone. The empirical Bayes prediction is partially pooled: it combines the group’s data with the across-group distribution. Complete pooling sets every group effect to zero and uses the common mean for all groups. The table summarizes these distinctions.

A useful algebraic check is that, under positive finite variances, \(d_j\) and \(\widehat b_j\) have the same sign, and \(|\widehat b_j|\leq |d_j|\). If a calculation yields a prediction farther from zero than the raw deviation, check the formula, the variance inputs, and the sign. When \(\tau^2\) approaches zero, \(\lambda_j\) approaches zero and predictions approach complete pooling. When \(n_j\) becomes large, or \(\sigma^2/n_j\) becomes small relative to \(\tau^2\), \(\lambda_j\) approaches one and predictions approach raw deviations.

These checks concern the algebra, not whether the model is suitable. Look for evidence that the Gaussian assumptions are implausible, such as extreme outliers, strongly asymmetric group-level patterns, or within-group variability that changes markedly across groups when the model assumes a common \(\sigma^2\). Also consider whether groups are independent and whether the common mean and the variance components are adequately estimated. In empirical Bayes practice, replacing unknown parameters with estimates adds uncertainty that the known-parameter derivation does not include. Treating plug-in predictions as if all model parameters were known can therefore overstate their precision.

A common mistake is to say, “The raw deviation is 10 units, so the group effect must be 10 units,” or to view shrinkage as an unjustified bias that should always be removed. The raw deviation includes sampling noise; the prediction is a model-based estimate of the latent effect that balances the group’s evidence against the assumed distribution across groups. The balance is justified only to the extent that the model and its variance estimates are credible. If the goal is simply to describe the observed group mean, report that mean or raw deviation. If the goal is prediction of latent group effects under the model, the shrunk prediction is the relevant quantity.