Empirical Bayes Prediction in a Gaussian Random-Intercept Model

Why Predict a Group Intercept?

A Gaussian random-intercept model allows groups to differ in their underlying response level while representing those differences with a distribution. For a particular group, the random intercept is not directly observed: it must be inferred from that group’s measurements while accounting for the model’s assumption that group effects come from a common distribution. The resulting prediction uses both the group’s data and that distribution.

The focus here is the conditional distribution of one group intercept when the model parameters are treated as known. Its mean gives a point prediction. In practice, an empirical Bayes prediction uses estimated population parameters in place of known ones; the calculation below shows the conditional result first and then how to use supplied fitted values. This tutorial does not account for uncertainty in those fitted parameters.

Definition: In a Gaussian random-intercept model, the observation for unit $i$ in group $j$ is $$ Y_{ij} = \mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta} + b_j + \varepsilon_{ij}, $$ where $\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}$ is the fixed-effects part, $b_j$ is the group-specific random intercept, and $\varepsilon_{ij}$ is the observation-level error. The model assumes $b_j \sim N(0,\tau^2)$, and, conditional on the random effects, $\varepsilon_{ij} \sim N(0,\sigma^2)$ independently across observations and independently of the random effects. The fixed-effects parameters $\boldsymbol{\beta}$, the random-intercept variance $\tau^2$, and the residual variance $\sigma^2$ are treated as known when deriving the conditional distribution.

Here $n_j$ denotes the number of observed units in group $j$. The fixed-effects part can vary across observations; it need not be a single common mean. Define the residual after subtracting that part as $r_{ij}=y_{ij}-\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}$. Given $b_j$, each residual has mean $b_j$ and variance $\sigma^2$. The information about the intercept in group $j$ is therefore carried by the residuals, not by the raw outcomes alone.

Deriving the Conditional Distribution

For the derivation, take a group with at least one observation and positive variances $\tau^2$ and $\sigma^2$. Conditional on the parameters, the prior density for $b_j$ is normal. The group’s observation likelihood is the product of the normal densities of its residuals given $b_j$. Multiplying likelihood and prior gives the conditional density of $b_j$ given the observed data and parameters.

Write the terms involving the intercept.
After omitting factors that do not depend on $b_j$, the prior density times the likelihood is proportional to the following exponential expression.

\exp\left\{-\frac{1}{2}\left[ \frac{b_j^2}{\tau^2}+ \frac{1}{\sigma^2}\sum_{i=1}^{n_j}(r_{ij}-b_j)^2 \right]\right\}.

Collect the quadratic and linear terms.
Expanding the squared residuals gives a quadratic coefficient $A_j=\tau^{-2}+n_j\sigma^{-2}$ and a linear coefficient $C_j=\sigma^{-2}\sum_i r_{ij}$.

\frac{b_j^2}{\tau^2}+\frac{1}{\sigma^2}\sum_i(r_{ij}-b_j)^2 =A_jb_j^2-2C_jb_j+\frac{1}{\sigma^2}\sum_i r_{ij}^2.

Complete the square.
The final term in the expansion does not depend on $b_j$, so it only contributes to the normalizing constant. Completing the square in the other terms identifies the conditional normal distribution.

A_jb_j^2-2C_jb_j =A_j\left(b_j-\frac{C_j}{A_j}\right)^2-\frac{C_j^2}{A_j}.

A normal density with exponent proportional to $-\tfrac12 A_j(b_j-C_j/A_j)^2$ has variance $A_j^{-1}$ and mean $C_j/A_j$. Thus the conditional distribution is

Result: Under the stated Gaussian model and with parameters treated as known, $$ b_j \mid \mathbf{y}_j,\boldsymbol{\beta},\tau^2,\sigma^2 \sim N\left( \frac{\tau^2}{\sigma^2+n_j\tau^2}\sum_{i=1}^{n_j}r_{ij}, \frac{\tau^2\sigma^2}{\sigma^2+n_j\tau^2} \right). $$ The conditional mean is the posterior mean for $b_j$; the second argument of $N(\cdot,\cdot)$ is its variance.

The notation $\mathbf{y}_j$ represents the observed outcomes in group $j$. If a group has no observations, its conditional distribution is just its prior $N(0,\tau^2)$, because its data contribute no information about its intercept. If $\tau^2=0$, the model assigns all groups an intercept of zero; the positive-variance derivation above does not apply directly, but the limiting prediction is zero.

Calculating an Empirical Bayes Prediction

With known parameters, the conditional mean is a model-based prediction given the group’s observations. In applied work, $\boldsymbol{\beta}$, $\tau^2$, and $\sigma^2$ are usually estimated from data across groups. Substituting those estimates into the conditional-mean formula gives an empirical Bayes prediction. When using supplied fitted values, treat them as fixed for this calculation: the result does not include uncertainty from estimating them.

Worked Example: A Group With Four Measurements

Suppose a hypothetical Gaussian model has supplied fixed-effect predictions for a group of four units. The observed outcomes minus their respective fixed-effect predictions are $2, 1, 3,$ and $2$ response units. Use $\tau^2=4$ and $\sigma^2=9$ to find the conditional distribution and its mean.

Assume the outcomes are conditionally independent normal observations with common residual variance $9$, and the group intercept has the stated normal distribution with variance $4$. These are the assumptions required by the formula. The sum of the four residuals is $2+1+3+2=8$, and $n_j=4$.

The conditional variance is $$ \left(\frac{1}{4}+\frac{4}{9}\right)^{-1} =\left(\frac{9+16}{36}\right)^{-1} =\frac{36}{25}=1.44. $$

The conditional mean is $$ 1.44\left(\frac{8}{9}\right)=1.28. $$

Therefore $b_j\mid\mathbf{y}_j$ is $N(1.28,1.44)$, and the empirical Bayes point prediction using these supplied parameter values is $1.28$ response units. It is a prediction of the group intercept, not of an individual outcome; predicting an outcome would also require its fixed-effects part and, for a future observation, its residual variation.

Worked Example: Residuals That Sum to a Small Value

For a second hypothetical group, suppose the three observed outcomes minus their fixed-effect predictions are $-1, 0,$ and $2$ units. Let the supplied values be $\tau^2=1$ and $\sigma^2=4$. Assume the same conditional independence and normality conditions as in the model.

Here $n_j=3$, and the residual sum is $-1+0+2=1$. The conditional precision, which is the reciprocal of the conditional variance, is $$ \frac{1}{\tau^2}+\frac{n_j}{\sigma^2} =1+\frac{3}{4}=\frac{7}{4}. $$

Thus the conditional variance is $4/7$, and the conditional mean is $$ \frac{4}{7}\left(\frac{1}{4}\right)=\frac{1}{7}\approx 0.1429. $$

The conditional distribution is $N(1/7,4/7)$. In particular, the positive prediction is small because the group’s residual sum is small; it does not imply a positive effect of the covariates or a causal group difference.

Worked Example: Applying Supplied Fitted Values

Suppose a hypothetical analysis supplies $\hat{\boldsymbol{\beta}}$, $\hat{\tau}^2=2.25$, and $\hat{\sigma}^2=1$. For one group, the observed outcomes minus the fixed-effect predictions computed from $\hat{\boldsymbol{\beta}}$ are $1.2, 2.1, 1.8, 2.4,$ and $2.5$ units. Treat the supplied fitted values as fixed and calculate the empirical Bayes prediction.

The residual sum is $1.2+2.1+1.8+2.4+2.5=10$, with $n_j=5$. The conditional variance using the supplied values is $$ \left(\frac{1}{2.25}+\frac{5}{1}\right)^{-1} =\left(\frac{4}{9}+5\right)^{-1} =\left(\frac{49}{9}\right)^{-1} =\frac{9}{49}\approx 0.1837. $$

The conditional mean is $$ \frac{9}{49}\left(\frac{10}{1}\right) =\frac{90}{49}\approx 1.837. $$

The empirical Bayes prediction is therefore about $1.837$ response units. Its interpretation is conditional on the specified Gaussian model and the fitted parameter values treated as fixed; it is not a statement that those parameter values are known without error.

What the Prediction Does—and Does Not—Represent

The group intercept is a latent deviation from the fixed-effects prediction. A prediction of $b_j=1.837$, for example, means that the model predicts the group’s response level to be about $1.837$ units above its fixed-effects component, under the model. It does not mean every outcome in that group is elevated by exactly that amount: individual residuals remain.

For comparison, setting $b_j=0$ gives the population-level prediction, which does not use this group’s residuals to estimate its particular intercept. The unpooled average residual, $\sum_i r_{ij}/n_j$, uses the group’s data but ignores the model’s distribution for group effects. The conditional mean uses the data and the assumed distribution together. In the Gaussian model, its conditional mode is also its conditional mean because a normal distribution is symmetric and reaches its maximum at its mean.

These are different prediction targets. A population-level prediction is appropriate when the target is the mean response across the random-effects distribution. A group-specific prediction is appropriate when the target is a particular group’s latent intercept. The choice should follow the question, rather than a preference for one prediction formula.

Assumptions, Diagnostics, and Limitations

The derivation relies on more than a normal-looking formula. It assumes a normal distribution for group intercepts, normal observation-level errors with a common variance, conditional independence of observations, and independence between the random intercept and observation errors. It also treats the fixed-effects part and variance parameters as known. If fixed-effect predictors are estimated, their estimation error is not included in the conditional distribution calculated here.

Diagnostics should assess whether the model is a plausible description of the data. Useful checks include examining residual patterns for changing spread or dependence, checking whether residual distributions show severe departures from normality, and looking for groups whose observations are inconsistent with the fitted model. Groups with unusual residual patterns may indicate outliers, omitted predictors, or a need for a different variance structure. A poor model fit can make a mathematically correct conditional calculation a poor prediction.

The group effects are also assumed to follow a normal distribution centered at zero. This is a distributional assumption about groups, not a claim that every group effect is near zero or that the observed groups are a random sample in every scientific setting. The Gaussian conjugate calculation is especially convenient because it produces an exact normal conditional distribution. For binary or count outcomes, the likelihood is not Gaussian and this completing-the-square result does not apply in the same way.

Common pitfall: Treating the average residual in a group as its empirical Bayes prediction ignores the assumed distribution of group effects. The average residual is the group-only estimate under a different calculation; the conditional mean derived here uses both the likelihood and the normal distribution for $b_j$. Conversely, using $b_j=0$ for every group discards group-specific observations. First identify whether the target is a population-level response or a group-specific intercept.

A second limitation is that the displayed conditional distribution is conditional on parameter values. If those values were estimated, the result does not automatically account for their uncertainty. The formula remains useful for a point prediction with supplied fitted values, but its apparent precision should not be mistaken for complete uncertainty quantification.

Check Your Understanding

Answer each question using the Gaussian random-intercept model and the assumptions stated above. Solutions follow the questions.

For $n_j=2$, residuals $1$ and $3$, $\tau^2=1$, and $\sigma^2=2$, calculate the conditional mean and variance.
What is the conditional distribution for a group with no observations?
Why is a group’s raw mean outcome not generally the right quantity to substitute for $\sum_i r_{ij}$?
What additional uncertainty is not represented when fitted parameter estimates are treated as fixed?

Solutions. (1) The residual sum is $4$. The variance is $(1+2/2)^{-1}=1/2$, and the mean is $(1/2)(4/2)=1$. Thus the conditional distribution is $N(1,1/2)$. (2) With no observations, the data provide no update, so the distribution remains the prior $N(0,\tau^2)$. (3) The formula uses outcomes after subtracting each unit’s fixed-effects prediction; raw outcomes can include variation explained by covariates and therefore do not equal the group’s residual information. (4) It omits uncertainty from estimating the fixed effects and variance parameters, including uncertainty in $\hat{\tau}^2$ and $\hat{\sigma}^2$.