Deriving Eta Shrinkage in a Gaussian Random-Intercept Model

Why a Random Intercept Is Shrunk

In a mixed-effects model, repeated observations on one individual provide evidence about that individual's deviation from the population level. That evidence is not equally precise for every individual: a short profile or noisy measurements leave more uncertainty than a long, precise profile. A Gaussian random-intercept model makes the balance between individual data and the population distribution explicit. Its conditional mean is a weighted version of the individual's observed deviation, with the weight determined by the number of observations and the variance components.

The result below builds on the conditional-distribution ideas in Conditional Distributions Behind Empirical Bayes Estimates. Here the focus is the derivation and interpretation of the weight for the particularly transparent Gaussian random-intercept case.

Definition: For individual \(i\), let \(y_{ij}\) be the response at observation \(j\), for \(j=1,\ldots,n_i\). The Gaussian random-intercept model is

y_{ij}=\mu+\eta_i+\epsilon_{ij},

where \(\mu\) is the population mean response, \(\eta_i\sim N(0,\omega^2)\) is the individual random intercept, and \(\epsilon_{ij}\sim N(0,\sigma^2)\) is residual error. Assume the random intercepts are independent across individuals, the residual errors are independent across observations and individuals, and all residual errors are independent of all random intercepts. The variance components satisfy \(\omega^2\geq 0\) and \(\sigma^2>0\).

The model treats \(\mu\), \(\omega^2\), and \(\sigma^2\) as known for the derivation. In practice, they are usually estimated from the population data and their estimates are substituted into the conditional formula. The derivation also assumes the same population mean and residual variance for the observations being combined, and that the individual's random intercept adds the same amount to every response.

Deriving the Conditional Mean

Write \(\bar y_i=n_i^{-1}\sum_{j=1}^{n_i}y_{ij}\) for the individual's sample mean. Conditional on \(\eta_i\), the observations are independent normal variables with mean \(\mu+\eta_i\) and variance \(\sigma^2\). Therefore,

\bar y_i\mid\eta_i\sim N\left(\mu+\eta_i,\frac{\sigma^2}{n_i}\right).

The individual's data thus estimate \(\eta_i\) through the deviation \(\bar y_i-\mu\), whose sampling variance is \(\sigma^2/n_i\). The population model supplies a second piece of information: before observing this individual's data, \(\eta_i\sim N(0,\omega^2)\). Combining these two normal distributions gives a normal conditional distribution.

Result: If \(\omega^2>0\), the conditional distribution of the random intercept given the individual's observations is

\eta_i\mid y_{i1},\ldots,y_{in_i} \sim N\left( \lambda_i(\bar y_i-\mu), \frac{\omega^2\sigma^2}{\sigma^2+n_i\omega^2} \right), \qquad \lambda_i=\frac{n_i\omega^2}{\sigma^2+n_i\omega^2}.

The conditional mean \(\lambda_i(\bar y_i-\mu)\) is the Gaussian empirical Bayes estimate of the individual random intercept when the variance components and \(\mu\) are treated as known. The factor \(\lambda_i\) is the retained fraction of the raw deviation; the fraction by which that deviation is reduced toward zero is \(1-\lambda_i\).

To verify the result, the likelihood contribution involving \(\eta_i\) can be expressed using the sample mean. The decomposition

\sum_{j=1}^{n_i}(y_{ij}-\mu-\eta_i)^2 = \sum_{j=1}^{n_i}(y_{ij}-\bar y_i)^2 +n_i(\bar y_i-\mu-\eta_i)^2

shows that the first term does not depend on \(\eta_i\). Multiplying the remaining normal likelihood by the normal distribution of \(\eta_i\), the part of the log density that depends on \(\eta_i\) is proportional to

-\frac{1}{2}\left[ \frac{\eta_i^2}{\omega^2} +\frac{n_i(\bar y_i-\mu-\eta_i)^2}{\sigma^2} \right].

Collecting the quadratic and linear terms in \(\eta_i\), its precision (the reciprocal of its variance) is \(1/\omega^2+n_i/\sigma^2\), and its mean is

\frac{(n_i/\sigma^2)(\bar y_i-\mu)} {1/\omega^2+n_i/\sigma^2} = \frac{n_i\omega^2}{\sigma^2+n_i\omega^2}(\bar y_i-\mu).

The reciprocal precision is \(\omega^2\sigma^2/(\sigma^2+n_i\omega^2)\), establishing both the conditional mean and variance. The same decomposition explains why, under these assumptions, the observations affect the conditional distribution through their mean and count; the within-individual deviations from \(\bar y_i\) do not provide further information about a constant intercept.

Worked Examples

Worked Example: Estimating One Individual's Intercept

Suppose, hypothetically, that an individual's four responses are 12, 11, 13, and 10 response units. Let \(\mu=10\) response units, \(\omega^2=4\) response-units squared, and \(\sigma^2=9\) response-units squared. Assume the model's normality and independence conditions hold, and treat these parameters as known.

The sample mean is \(\bar y_i=(12+11+13+10)/4=11.5\) response units, so the raw deviation is \(11.5-10=1.5\) response units. The weight is

\lambda_i=\frac{4(4)}{9+4(4)}=\frac{16}{25}=0.6400.

Thus the conditional mean estimate is \(0.6400(1.5)=0.9600\) response units. The unshrunk individual-level mean would imply an intercept deviation of 1.5 response units; the conditional estimate is closer to the population value of zero because the individual mean is noisy relative to the assumed between-individual variability. The conditional variance is \(4(9)/25=1.44\) response-units squared, and the conditional standard deviation is \(\sqrt{1.44}=1.200\) response units.

Worked Example: Increasing the Number of Observations

Consider hypothetical data with \(\omega^2=1.44\) response-units squared and \(\sigma^2=4\) response-units squared. To isolate the effect of observation count, compare individuals with the same sample-mean deviation, \(\bar y_i-\mu=2\) response units, but different \(n_i\). This is a comparison of the formula at a fixed deviation, not a claim that adding observations to a fixed dataset leaves its sample mean unchanged.

For \(n_i=1\), the weight is \(1.44/(4+1.44)=0.2647\), giving a conditional mean of \(0.2647(2)=0.5294\) response units. For \(n_i=4\),

\lambda_i=\frac{4(1.44)}{4+4(1.44)} =\frac{5.76}{9.76}=0.5902, \qquad E(\eta_i\mid y_i)=0.5902(2)=1.180

response units. For \(n_i=16\), the weight is \(23.04/(4+23.04)=0.8521\), so the conditional mean is \(0.8521(2)=1.704\) response units. The same observed deviation receives greater weight when it is supported by more observations, because the variance of the sample mean, \(\sigma^2/n_i\), is smaller.

Worked Example: Increasing the Residual Variance

Suppose hypothetically that \(n_i=3\), \(\omega^2=2.25\) response-units squared, and \(\bar y_i-\mu=1.2\) response units. Compare residual variances of 1 and 9 response-units squared. Assume the model conditions hold in each comparison.

When \(\sigma^2=1\), the weight is

\lambda_i=\frac{3(2.25)}{1+3(2.25)} =\frac{6.75}{7.75}=0.8710, \qquad E(\eta_i\mid y_i)=0.8710(1.2)=1.045

response units. When \(\sigma^2=9\), the weight is \(6.75/(9+6.75)=0.4286\), and the conditional mean is \(0.4286(1.2)=0.5143\) response units. The second estimate is closer to zero because the same sample-mean deviation is less precise when residual variance is higher. For reference, the conditional variances are \(2.25/7.75=0.2903\) and \(2.25(9)/15.75=1.286\) response-units squared, respectively.

What Controls the Amount of Shrinkage?

The formula can be written in terms of the relative precision of the individual data and population distribution:

\lambda_i = \frac{n_i/\sigma^2}{n_i/\sigma^2+1/\omega^2}.

The numerator is the precision of the individual's sample mean, while \(1/\omega^2\) is the precision of the population distribution for the intercept. When the individual data are more precise relative to the population distribution, \(\lambda_i\) is larger and the conditional mean remains closer to \(\bar y_i-\mu\). Equivalently, the observations' precision rises with \(n_i\) and falls with \(\sigma^2\); the prior precision rises as \(\omega^2\) decreases.

For \(\omega^2>0\), \(\sigma^2>0\), and finite \(n_i>0\), the weight satisfies \(0<\lambda_i<1\). Holding the other quantities fixed, it increases as \(n_i\) increases or \(\omega^2\) increases, and decreases as \(\sigma^2\) increases. If \(n_i\) tends to infinity, \(\lambda_i\) tends to 1; if \(\sigma^2\) tends to zero, it also tends to 1. If \(\omega^2=0\), the population model assigns all individuals an intercept of zero, and the conditional estimate is zero. These limiting cases match the interpretation of the model.

Assumptions, Diagnostics, and Alternatives

The algebra is exact for the specified model, but its interpretation relies on its assumptions. In particular, the same additive intercept must apply to each observation for an individual, residual errors must be independent with a common variance, and the random intercept and residual errors must follow the stated normal distributions. If residual variability changes across time or concentration, or errors are correlated, \(\sigma^2/n_i\) need not be the variance of the sample mean. The displayed weight would then not generally be the correct conditional weight.

Useful checks include plotting residuals against time, fitted response, and other predictors; looking for systematic within-individual patterns; and checking whether the observed spread across individuals is plausibly represented by a normal random-intercept distribution. Trends within an individual's profile may indicate that a constant intercept is inadequate. Unequal residual spread or serial correlation challenges the assumed residual model. An estimated \(\omega^2\) close to zero can also make the resulting conditional estimates close to zero for nearly everyone; that behavior follows the fitted model, but it may signal weak evidence for between-individual variability or a limitation of the model.

The conditional estimate sits between two simple alternatives. An unpooled estimate uses \(\bar y_i-\mu\) without shrinkage and can be highly variable when \(n_i\) is small or residual error is large. Complete pooling sets every \(\eta_i\) to zero and ignores the individual's observed deviation. The Gaussian conditional mean uses both sources of information, with the relative weight derived above. Unlike either simple alternative, it is also accompanied by a conditional variance that describes remaining uncertainty under the model.

The result has important limits. It treats the population parameters as fixed and known, so it does not include uncertainty from estimating \(\mu\), \(\omega^2\), or \(\sigma^2\). It is a result for an additive Gaussian random intercept, not a general formula for nonlinear individual parameters. Also, \(\lambda_i\) is the model-based retained fraction for the conditional mean; it should not be confused with a data-based summary of shrinkage calculated across many individuals. The formula clarifies the mechanism, but does not by itself establish that the assumptions or variance estimates are adequate.

Key takeaway: In a Gaussian random-intercept model, the conditional mean is \(\lambda_i(\bar y_i-\mu)\), where \(\lambda_i=n_i\omega^2/(\sigma^2+n_i\omega^2)\). More observations and greater between-individual variance increase the weight on an individual's data; greater residual variance increases shrinkage toward zero.

Check Your Understanding

Answer each question using the Gaussian random-intercept model and its assumptions stated above.

If \(n_i=2\), \(\omega^2=1\) response-units squared, \(\sigma^2=3\) response-units squared, and \(\bar y_i-\mu=1\) response unit, what is the conditional mean estimate?
With other quantities fixed, what happens to the shrinkage weight as the number of observations increases, and why?
What does the model imply for the conditional mean estimate if \(\omega^2=0\)?
Why might the derived weight be inappropriate when residual errors are serially correlated?

Solutions. (1) \(\lambda_i=2(1)/(3+2)=0.4000\), so the conditional mean is \(0.4000(1)=0.4000\) response units. (2) The weight increases toward 1 because the sample-mean variance \(\sigma^2/n_i\) decreases, so the individual's data become more precise relative to the population distribution. (3) The intercept is constrained to zero by the zero-variance population distribution, so its conditional mean is zero. (4) Serial correlation changes the variance of the sample mean from the assumed \(\sigma^2/n_i\); consequently the precision balance used to derive the weight no longer applies in general.