How Variance-Component Uncertainty Changes Shrinkage

Why Variance Estimates Matter for Shrinkage

A Gaussian empirical Bayes prediction combines a group’s observed deviation from the overall mean with the model’s estimate of how much groups genuinely differ. If the estimated random-effect variance changes, the balance between those two ingredients changes too. This tutorial quantifies that change for a Gaussian random-intercept model. The focus is not only whether shrinkage increases or decreases, but how strongly the group prediction responds to a change in the variance estimate.

The calculations below isolate the effect of the random-effect variance: the observed group data, fixed intercept, and residual variance are held constant while the random-effect variance varies. A complete model refit can change all of these quantities, so the derivative describes one component of the sensitivity, not every consequence of refitting.

Model and Shrinkage Weight

For group $j$, suppose the observations follow the Gaussian random-intercept model

Y_{ij}=\beta_0+b_j+\varepsilon_{ij},\qquad b_j\sim N(0,\tau^2),\qquad \varepsilon_{ij}\sim N(0,\sigma^2).

Assume the random effects and residual errors are mutually independent, observations are conditionally independent given $b_j$, and all groups share the same residual variance $\sigma^2$. Write $\bar{Y}_j$ for the observed mean in group $j$, based on $n_j$ observations. The group-mean residual variance is $q_j=\sigma^2/n_j$. For this derivation, treat $\beta_0$, $\sigma^2$, and the observed data as fixed.

Definition: Let $v=\tau^2$ denote the random-intercept variance and let $r_j=\bar{Y}_j-\beta_0$ denote the observed group-mean deviation from the fixed intercept. The Gaussian shrinkage weight and the empirical Bayes prediction of the group effect are $$ B_j(v)=\frac{v}{v+q_j},\qquad \widehat{b}_j(v)=B_j(v)r_j. $$ The weight lies between 0 and 1 for $v>0$ and $q_j>0$. It determines the fraction of the observed deviation retained in the prediction.

This formula is the conditional mean of $b_j$ given the group data under the stated Gaussian model. In fact, $\bar{Y}_j-\beta_0=b_j+\bar{\varepsilon}_j$, where the independent terms have variances $v$ and $q_j$. The conditional mean of one jointly Gaussian variable given their sum is its covariance with the sum, divided by the variance of the sum, times the observed sum. Here that ratio is $v/(v+q_j)$, giving the displayed weight.

Differentiating the Weight

To measure local sensitivity, differentiate with respect to the variance $v=\tau^2$, not with respect to the standard deviation $\tau$. Since $q_j$ is held fixed,

\frac{\partial B_j}{\partial v} =\frac{(v+q_j)-v}{(v+q_j)^2} =\frac{q_j}{(v+q_j)^2}>0.

Thus a larger random-effect variance gives more weight to the observed group deviation. The derivative has units of inverse response variance: $B_j$ is dimensionless, while $v$ is measured in squared response units. For a small change $\Delta v$, the corresponding first-order changes are

\Delta B_j\approx \frac{q_j}{(v+q_j)^2}\Delta v,\qquad \Delta\widehat{b}_j\approx r_j\frac{q_j}{(v+q_j)^2}\Delta v.

The sign of the prediction change also depends on $r_j$. If the group mean is above the fixed intercept, increasing $v$ moves its prediction upward; if it is below, the prediction moves downward, away from zero. If $r_j=0$, the prediction remains zero regardless of the weight.

A useful scale-free measure is the elasticity of the weight: the approximate percentage change in $B_j$ for a one-percent change in $v$. Direct substitution gives

\frac{\partial\log B_j}{\partial\log v} =\frac{v}{B_j}\frac{\partial B_j}{\partial v} =\frac{q_j}{v+q_j} =1-B_j.

The weight is therefore relatively more responsive to the variance when it is smaller, and less responsive in percentage terms when it is already close to one. This relative statement does not by itself determine the absolute change in a prediction: that also depends on the group residual $r_j$ and the units and size of the variance change.

Worked Example: Local and Exact Change in One Prediction

Suppose a hypothetical group has $n_j=10$, residual variance $\sigma^2=4$, observed group mean $\bar{Y}_j=12.5$, and fixed intercept $\beta_0=11.0$. The fitted random-effect variance is $v=0.64$. Calculate the weight and prediction, then estimate what happens if $v$ increases by $0.10$.

First, $q_j=\sigma^2/n_j=4/10=0.40$, and $r_j=12.5-11.0=1.5$. The fitted weight and group-effect prediction are

B_j(0.64)=\frac{0.64}{0.64+0.40} =\frac{0.64}{1.04}=0.6154, \qquad \widehat{b}_j=0.6154(1.5)=0.9231.

The derivative at $v=0.64$ is

\frac{\partial B_j}{\partial v} =\frac{0.40}{(1.04)^2} =\frac{0.40}{1.0816}=0.3698.

For $\Delta v=0.10$, the first-order change in the weight is $0.3698(0.10)=0.03698$, so the first-order change in the predicted effect is $1.5(0.03698)=0.05547$. The approximate updated prediction is $0.9231+0.0555=0.9786$.

For comparison, recomputing the formula gives $B_j(0.74)=0.74/(0.74+0.40)=0.6491$, and the exact updated prediction is $0.6491(1.5)=0.9737$. The exact change is $0.0506$. The first-order calculation is close, but not identical, because the derivative is a local approximation.

How Group Information Affects Sensitivity

The derivative depends on $q_j=\sigma^2/n_j$, so group size affects sensitivity. It is tempting to conclude that larger groups must always produce more sensitivity because they estimate a group mean more precisely. That conclusion is not generally correct for the absolute derivative with respect to $v$. For fixed $v>0$, consider the derivative as a function of $q_j$:

g(q_j)=\frac{q_j}{(v+q_j)^2},\qquad g'(q_j)=\frac{v-q_j}{(v+q_j)^3}.

It increases with $q_j$ when $q_j<v$ and decreases when $q_j>v$; its maximum occurs at $q_j=v$. Because $q_j$ decreases as $n_j$ increases, sensitivity can first rise and then fall as group size changes. This concerns the absolute derivative per unit of variance; the weight itself increases monotonically with $v$ and, for fixed $v$, increases as $n_j$ increases.

Worked Example: Comparing Group Sizes

Suppose $\sigma^2=1$ and the random-effect variance is $v=0.25$. Compare the weight and its derivative for groups of size $n_j=4$ and $n_j=16$. The group sizes are the only quantities changing.

For $n_j=4$, $q_j=1/4=0.25$. Therefore

B_j=\frac{0.25}{0.25+0.25}=0.5000,\qquad \frac{\partial B_j}{\partial v} =\frac{0.25}{(0.50)^2}=1.000.

For $n_j=16$, $q_j=1/16=0.0625$, so

B_j=\frac{0.25}{0.25+0.0625}=0.8000,\qquad \frac{\partial B_j}{\partial v} =\frac{0.0625}{(0.3125)^2}=0.6400.

The larger group has a higher weight, so it retains more of its observed deviation. Yet its absolute weight derivative is smaller at this fitted variance: 0.6400 rather than 1.000 per unit of variance. The comparison illustrates why the weight and its sensitivity should not be treated as interchangeable summaries.

Using Variance-Estimate Uncertainty

A plug-in empirical Bayes prediction substitutes an estimate $\hat{v}$ for $v$. If the estimate has standard error $SE(\hat{v})$, a first-order delta-method calculation can approximate the variation in the prediction attributable to estimating $v$, while treating the data, $\beta_0$, and $\sigma^2$ as fixed. The derivative of the prediction is $r_jq_j/(\hat{v}+q_j)^2$, so

\operatorname{Var}_{\hat v}\{\widehat b_j(\hat v)\} \approx \left[r_j\frac{q_j}{(\hat v+q_j)^2}\right]^2 \operatorname{Var}(\hat v).

This is a local approximation: it is most useful when the estimate’s uncertainty is not so large that the prediction function changes substantially across plausible values of $v$. It describes only the contribution from estimating this variance component. It does not include uncertainty about the latent group effect, uncertainty in the fixed intercept or residual variance, or other sources of sampling variation.

Worked Example: Approximate Sensitivity to an Uncertain Variance Estimate

Consider hypothetical data with $\hat{v}=0.36$, $SE(\hat{v})=0.12$, $\sigma^2=2.25$, $n_j=9$, and $r_j=-0.80$. Treat the other quantities as fixed and calculate the local change in the group prediction associated with one standard error of variance-estimate change.

Here $q_j=2.25/9=0.25$, so the fitted weight and prediction are

B_j(0.36)=\frac{0.36}{0.36+0.25} =\frac{0.36}{0.61}=0.5902, \qquad \widehat b_j=0.5902(-0.80)=-0.4721.

The derivative of the weight is $0.25/(0.61)^2=0.6719$ per unit of variance. Thus the derivative of the prediction is $(-0.80)(0.6719)=-0.5375$ response units per unit of variance. Multiplying its magnitude by $SE(\hat{v})$ gives an approximate one-standard-error change of

|r_j|\frac{q_j}{(\hat v+q_j)^2}SE(\hat v) =0.80(0.6719)(0.12)=0.06450.

A small increase of $0.12$ in the variance estimate would locally move the prediction downward by about $0.0645$, because the observed residual is negative; a decrease would locally move it upward by about the same amount. This is a sensitivity calculation, not a complete uncertainty interval for the group effect.

Diagnostics, Alternatives, and Limitations

Sensitivity calculations are useful when a reported prediction depends on an estimated variance component. They indicate whether modest changes in that estimate would materially change the prediction, and they help distinguish a stable prediction from one that is driven by a narrow plug-in value. They do not establish that the Gaussian model is appropriate; model fit and assumptions still need to be assessed.

Check what is being held fixed. The derivative above is a partial derivative. If a refit also changes $\hat{\beta}_0$, $\hat{\sigma}^2$, or the group residual, the total change can differ.
Check the scale of the change. A derivative is per unit of $v$. Multiply it by a variance change in matching squared-response units; do not interpret it as a change per unit of $\tau$.
Check whether the local approximation is adequate. Compare the first-order result with direct recalculation at nearby variance values. Large discrepancies signal that a local derivative is not summarizing the relevant range well.
Check the model assumptions. Strong departures from Gaussian random effects, a common residual variance, or conditional independence can make this Gaussian weight an unsuitable description of prediction sensitivity.
Do not confuse sources of uncertainty. This calculation isolates uncertainty in the estimated random-effect variance. It is not, by itself, uncertainty about the realized group effect or a full prediction interval.

The simplest alternative is to report only the plug-in prediction. That is concise, but it hides how much the result might move when the variance estimate changes. A derivative or delta-method calculation gives a quick local summary; direct recalculation over a range of variance values is a useful alternative when changes are not small. A full refit-based analysis can reflect additional parameter changes, but it answers a broader question than the partial derivative derived here.

A common mistake is to say, “The derivative is positive, so every group prediction increases when the random-effect variance increases.” The derivative of the weight is positive, but the derivative of the prediction is $r_jq_j/(v+q_j)^2$. A group with $r_j<0$ has a negative prediction derivative, so its prediction becomes more negative as the variance increases. The correction is to separate the direction of the weight change from the direction of each group’s prediction change.

Key takeaway: In the Gaussian random-intercept model, $\partial B_j/\partial\tau^2=q_j/(\tau^2+q_j)^2$. The induced local change in a group prediction is this derivative multiplied by the group’s observed residual. Sensitivity therefore depends on the variance estimate, the group-mean precision, and the direction and size of the group residual.

Check Your Understanding

Use the Gaussian random-intercept formulas above. Unless stated otherwise, treat all quantities other than the random-effect variance as fixed.

If $q_j=0.5$ and $v=0.5$, calculate $B_j$ and $\partial B_j/\partial v$.
For a group with $r_j=-2$, $q_j=0.4$, and $v=0.6$, what is the sign of the prediction’s derivative with respect to $v$? Explain in context.
Why is the derivative of the weight not a percentage change in the weight?
With $v=0.25$ fixed, how does the absolute derivative $q_j/(v+q_j)^2$ vary as $q_j$ changes, and where is it largest?

Solutions. (1) $B_j=0.5/(0.5+0.5)=0.5000$, and $\partial B_j/\partial v=0.5/(1.0)^2=0.5000$ per unit of variance. (2) The prediction derivative is $(-2)(0.4)/(1.0)^2=-0.8000$ response units per unit of variance; increasing $v$ makes this group’s predicted effect more negative. (3) The derivative is an absolute rate of change per unit of variance. A relative percentage response is the elasticity $\partial\log B_j/\partial\log v=1-B_j$. (4) The derivative increases with $q_j$ up to $q_j=v=0.25$, then decreases; its maximum occurs at $q_j=0.25$, where it equals $1/(4v)=1.000$ per unit of variance.