Empirical Bayes Predictions for Sparse Groups

Why Sparse Groups Need Careful Prediction

A group with only a few observations can show an extreme outcome by chance. For example, if both of two responses are successes, the observed proportion is 1. A binary GLMM does not usually treat that raw proportion as a precise estimate of the group’s underlying response probability. It combines the group’s data with a model in which group effects are drawn from a population distribution. When the group contributes little information, its predicted effect is pulled toward the population level. This is shrinkage.

Sparse does not simply mean “few rows.” In a binomial model, the number of trials and the fitted probabilities matter. In a count model, both the count and its exposure matter. The useful question is how much the group’s likelihood informs its latent effect, relative to the assumed spread of effects across groups.

For the examples, consider a random-intercept model with \(b_j\sim N(0,\tau^2)\). Conditional on \(b_j\), binary responses have logit probability \(\eta_j=\mathbf{x}_j^\mathsf{T}\boldsymbol{\beta}+b_j\), while counts have log mean \(\eta_j=\mathbf{x}_j^\mathsf{T}\boldsymbol{\beta}+b_j\), with an exposure multiplier when needed. The random effects are assumed independent across groups, and the responses are conditionally independent given the random effects and covariates. These are model assumptions, not guarantees supplied by the small group’s data.

How Sparse Information Produces Shrinkage

For one group, write its conditional log likelihood as \(\ell_j(b)\), and suppose its random intercept has distribution \(N(0,\tau^2)\). Apart from terms that do not depend on \(b\), the log conditional density being maximized is

Thus the group data pull the prediction toward values favored by \(\ell_j(b)\), while the normal random-effects distribution penalizes large departures from zero. The conditional mode satisfies

A small amount of likelihood information means that a modest departure from zero can balance the prior penalty. This gives a useful local derivation of the shrinkage effect. If the group log likelihood is approximately quadratic around its own maximum \(b_{Lj}\), with curvature \(I_j>0\), then

Combining this approximation with the normal penalty and setting the derivative to zero gives

The multiplier is between 0 and 1. When \(I_j\) is small, the multiplier is small and the prediction is near zero; when \(I_j\) is large, the multiplier approaches 1. This is a local approximation, not a universal formula for nonlinear GLMM predictions. In particular, a group with all successes or all failures may have no finite unpenalized likelihood estimate \(b_{Lj}\), even though its penalized conditional mode is finite.

Count Data: Counts and Exposure Both Matter

For a Poisson random-intercept model with known exposure \(e_j\), let the conditional mean be \(\mu_j=e_j\exp(\mathbf{x}_j^\mathsf{T}\boldsymbol{\beta}+b_j)\). For a group with count \(y_j\), the derivative of its log likelihood with respect to \(b_j\) is \(y_j-\mu_j\). Adding the normal random-effects penalty yields the score equation

The equation makes clear that a count alone does not describe the group’s information. A zero count over a long exposure can be stronger evidence for a low rate than a zero count over a very short exposure. Likewise, a small count may be informative when the exposure and expected count are both small, but one must assess that information relative to the model’s fitted mean and random-effect variance.

Diagnostics, Alternatives, and Limitations

For sparse groups, report the prediction alongside the amount and type of group-specific information. Useful checks include the group’s number of trials or exposure, its observed outcome pattern, and whether the fitted probability or mean is near a boundary. Inspect the conditional-mode score at the reported prediction: it should be near zero. Also check that the prediction is stable to reasonable changes in the fitted fixed effects and \(\hat\tau^2\). If the random-effect variance is estimated close to zero, predictions may be close to zero for nearly all groups; that behavior reflects the fitted model and should not be mistaken for strong group-specific evidence.

A raw group proportion or rate is a simple alternative, but it ignores the distribution of group effects and can be highly unstable for small denominators or exposures. A separate, unpooled group model avoids borrowing information across groups, but with sparse data it can yield extreme or undefined estimates, such as an infinite logit estimate for all successes. The empirical Bayes prediction borrows strength through the shared model. It is not automatically superior: misspecified fixed effects, an unsuitable random-effects distribution, or a poor variance estimate can systematically distort the predictions.

In practice, conditional modes are often obtained by numerical optimization rather than hand-solving score equations. Numerical convergence alone is not a diagnostic of model adequacy. Check for unusual group patterns, implausible fitted values, and sensitivity to the assumed random-effects distribution. Sparse data cannot reliably establish that a particular group truly has an extreme effect; the amount of shrinkage depends on estimated population-level parameters as well as that group’s outcomes. The prediction is also distinct from an uncertainty interval for the latent effect.