Hyperpriors And Variance Components In Multilevel Models

Why Hyperpriors Matter

In a multilevel model, group-specific effects are related but not assumed identical. Partial pooling lets estimates for groups with limited data borrow information from the overall distribution of effects. The amount of pooling depends on the between-group standard deviation: when it is small, group effects are pulled strongly toward the common mean; when it is large, the data for each group have more influence on its estimate.

A Bayesian model must therefore specify not only a distribution for group effects, but also a prior for the unknown scale of their variation. That prior is a hyperprior. Its influence can be substantial when there are few groups or little information within groups. Scale choices also determine how much prior probability is placed near complete pooling and in the tail corresponding to large heterogeneity.

Definition: A variance component is a variance parameter associated with a level of a hierarchical model. In a random-intercept model, \(\sigma^2\) is the within-group variance and \(\tau^2\) is the between-group variance. A hyperprior is a prior distribution on a parameter of a prior distribution; for example, \(p(\tau)\) specifies prior uncertainty about the group-level standard deviation \(\tau\).

A Basic Multilevel Model

Suppose there are \(J\) groups, with observations \(y_{ij}\) for individual \(i\) in group \(j\). A simple random-intercept model is

y_{ij}\mid \alpha_j,\sigma^2 \sim N(\alpha_j,\sigma^2), \qquad \alpha_j\mid\mu,\tau^2 \sim N(\mu,\tau^2), \qquad j=1,\ldots,J.

Here \(\alpha_j\) is the mean outcome in group \(j\), \(\mu\) is the overall mean of the group effects, \(\sigma^2\) describes individual variation around a group mean, and \(\tau^2\) describes variation among the group means. The standard deviation \(\tau\), not \(\tau^2\), is often assigned a hyperprior because its scale is directly interpretable in the units of the outcome. A complete Bayesian specification also gives priors to \(\mu\) and, when unknown, \(\sigma^2\); the examples below hold those quantities fixed where needed to isolate the role of \(\tau\).

For intuition, let \(\bar y_j\) be the sample mean in group \(j\), and suppose the within-group variance \(\sigma^2\) is known. Its sampling variance is \(v_j=\sigma^2/n_j\). Integrating out the group effect gives

\bar y_j\mid\mu,\tau^2,\sigma^2 \sim N\!\left(\mu,\tau^2+v_j\right).

Derivation: Conditional on \(\alpha_j\), the group mean has distribution \(\bar y_j\mid\alpha_j\sim N(\alpha_j,v_j)\), while \(\alpha_j\mid\mu,\tau^2\sim N(\mu,\tau^2)\). Write \(\bar y_j=\alpha_j+e_j\), where \(e_j\) is independent of \(\alpha_j\), has mean zero, and has variance \(v_j\). Then \(\bar y_j\) has mean \(\mu\) and variance \(\tau^2+v_j\), giving the stated normal distribution.

Consequently, conditional on \(\mu\), the likelihood for \(\tau\) based on the group means is proportional to

L(\tau\mid\bar y,\mu) \propto \prod_{j=1}^{J} (\tau^2+v_j)^{-1/2} \exp\!\left\{ -\frac{(\bar y_j-\mu)^2}{2(\tau^2+v_j)} \right\}.

The posterior for \(\tau\) combines this likelihood with its hyperprior and the prior information about other unknown parameters. The expression shows why the number and precision of groups matter: observed differences among \(\bar y_j\) must be judged against both the within-group sampling variances \(v_j\) and the proposed between-group variance \(\tau^2\). Large differences can support a larger \(\tau\), but noisy group means make those differences less informative.

Choosing a Hyperprior and Its Scale

Common choices put a distribution on the nonnegative standard deviation \(\tau\). A half-normal prior is the distribution of the absolute value of a zero-centered normal variable. A half-\(t\) prior is the absolute value of a centered Student \(t\) variable. Both have support \(\tau\geq0\); the half-\(t\) has heavier tails, allowing more prior probability for very large heterogeneity. The scale parameter must be stated in outcome units.

An alternative is an inverse-gamma prior directly on the variance \(\tau^2\). It can be convenient for algebra, but its behavior near zero and in the upper tail depends on its shape and scale parameters. It is not a neutral or automatically noninformative choice. Moreover, a prior on \(\tau\) and a prior on \(\tau^2\) are not interchangeable without a change of variables.

Worked Example: Interpreting a Half-Normal Scale

Suppose the group effects are measured in percentage points, and the prior is \(\tau\sim\text{Half-Normal}(0,4)\). The density is proportional to \(\exp[-\tau^2/(2\cdot4^2)]\) for \(\tau\geq0\). What does the scale of 4 imply about prior heterogeneity?

For a half-normal distribution with scale \(A\), \(P(\tau\leq t)=2\Phi(t/A)-1\) for \(t\geq0\), where \(\Phi\) is the standard normal cumulative distribution function. Thus

P(\tau>4)=2\{1-\Phi(4/4)\} =2\{1-\Phi(1)\} \approx 0.3173.

The prior assigns about 31.7% probability to a between-group standard deviation above 4 percentage points. Its prior mean is \(4\sqrt{2/\pi}\approx3.192\) percentage points. These quantities give the scale a concrete interpretation before seeing the data. They do not say that the posterior standard deviation will equal 4; the data update the prior.

When choosing a scale, first use the outcome's units and a plausible range of group differences. For example, if a 10-percentage-point standard deviation across groups would already be unusually large, a prior that routinely places substantial mass far beyond that range may be too diffuse for the scientific scale. Conversely, a prior concentrated near zero can force extensive pooling. Standardizing the outcome can simplify scale choices, but the prior must still be interpreted on that standardized scale.

Posterior Uncertainty and Shrinkage

If \(\mu\) and \(\tau\) are fixed, the group effect has a normal posterior. For known \(v_j\), its posterior mean is a weighted average of the observed group mean and the common mean:

E(\alpha_j\mid\bar y_j,\mu,\tau^2) = \lambda_j\bar y_j+(1-\lambda_j)\mu, \qquad \lambda_j=\frac{\tau^2}{\tau^2+v_j},

and its posterior variance is \((1/\tau^2+1/v_j)^{-1}\). The weight \(\lambda_j\) increases with \(\tau^2\) and decreases with \(v_j\). Thus, more precise group data receive more weight, while a smaller between-group variance draws estimates more strongly toward \(\mu\). When \(\tau\) is unknown, the posterior uncertainty about it must also be reflected in uncertainty about \(\alpha_j\).

Worked Example: Comparing Shrinkage at Two Variance Scales

Suppose a hypothetical group has sample mean \(\bar y_j=12\) outcome units and sampling variance \(v_j=4\). For illustration, take the common mean as known, \(\mu=10\). Compare the conditional group-effect posterior for \(\tau=3\) with that for \(\tau=1\).

When \(\tau=3\), the between-group variance is \(\tau^2=9\), so

\lambda_j=\frac{9}{9+4}=\frac{9}{13}\approx0.6923, \qquad E(\alpha_j\mid\bar y_j,\mu,\tau^2) =\frac{9}{13}(12)+\frac{4}{13}(10) \approx11.385.

The conditional posterior variance is

\left(\frac{1}{9}+\frac{1}{4}\right)^{-1} =\frac{36}{13} \approx2.769 \quad\text{outcome-units squared}.

When \(\tau=1\), the variance is 1 and

\lambda_j=\frac{1}{1+4}=0.2, \qquad E(\alpha_j\mid\bar y_j,\mu,\tau^2) =0.2(12)+0.8(10)=10.4.

The conditional posterior variance is \((1+1/4)^{-1}=0.8\) outcome-units squared. The smaller \(\tau\) produces stronger shrinkage toward 10 and a more concentrated conditional posterior. These are conditional calculations: if \(\tau\) is uncertain, the full posterior averages over its possible values and is generally less summarized by either calculation alone.

The propagation of hyperparameter uncertainty follows the law of total variance. Conditional on \(\mu\), for example,

\operatorname{Var}(\alpha_j\mid y) = E\!\left\{\operatorname{Var}(\alpha_j\mid y,\tau,\mu)\mid y\right\} + \operatorname{Var}\!\left\{E(\alpha_j\mid y,\tau,\mu)\mid y\right\}.

The first term averages the remaining uncertainty in the group effect when the hyperparameters are known. The second term captures changes in the group-effect posterior mean as \(\tau\) and \(\mu\) vary over their posterior distributions. Ignoring the second term understates posterior uncertainty whenever uncertainty in the hyperparameters matters. This decomposition is one reason that estimating a variance component and then treating its estimate as known does not, in general, reproduce full Bayesian uncertainty.

Worked Example: A Conditional Variance-Component Update

Suppose a hypothetical model uses \(\tau^2\sim\text{Inverse-Gamma}(a,b)\), with density proportional to \((\tau^2)^{-a-1}\exp(-b/\tau^2)\) for \(\tau^2>0\). For illustration, condition on a known \(\mu\) and known group effects with deviations \(-2,1,3,-2\) outcome units. Let \(a=2\) and \(b=4\) in squared outcome units. Find the conditional posterior of \(\tau^2\).

The group-effect likelihood as a function of \(\tau^2\) contributes \((\tau^2)^{-J/2}\exp[-\sum_j(\alpha_j-\mu)^2/(2\tau^2)]\). The squared deviations sum to

$$ (-2)^2+1^2+3^2+(-2)^2=4+1+9+4=18. $$

Multiplying the likelihood by the prior adds \(J/2=4/2=2\) to the inverse-gamma shape and adds \(18/2=9\) to its scale. Therefore,

\tau^2\mid\alpha,\mu \sim \text{Inverse-Gamma}\left(2+\frac{4}{2},\,4+\frac{18}{2}\right) =\text{Inverse-Gamma}(4,13).

Under this parameterization, an inverse-gamma distribution with shape greater than 1 has mean equal to its scale divided by its shape minus 1. The conditional posterior mean is \(13/(4-1)=4.333\) squared outcome units. This is a conditional update with the group effects and \(\mu\) held fixed, not the marginal posterior for \(\tau^2\) given only the observations. In the full model, uncertainty about the unobserved group effects and other parameters must also be integrated over.

The Boundary at Zero

The value \(\tau=0\) is a boundary of the parameter space. At that value, the group-level distribution collapses to a point: \(\alpha_j=\mu\) for every group, so the model has complete pooling. For \(\tau>0\), groups can differ. A continuous prior such as a half-normal or half-\(t\) assigns probability zero to the single point \(\tau=0\), even if its density is positive near zero. It can support posterior values very close to zero, but it does not give positive posterior probability to exact complete pooling.

If the scientific question requires a model with positive probability of exact equality across groups, a continuous prior alone does not provide it. One possibility is a mixture prior with a point mass at zero and a continuous distribution for positive \(\tau\). Such a model makes the distinction between exact pooling and small-but-nonzero heterogeneity explicit.

The parameterization matters near this boundary. If a prior is specified on \(\tau\), the induced density on \(v=\tau^2\) is

p(v)=p(\sqrt v)\left|\frac{d\sqrt v}{dv}\right| =\frac{p(\sqrt v)}{2\sqrt v},\qquad v>0.

The factor \(1/(2\sqrt v)\) means that a prior described on the standard deviation cannot simply be relabeled as a prior on the variance. Their density shapes, including their behavior near zero, differ.

Common Mistakes and Practical Checks

Treating a hyperprior as a harmless default. A variance prior can meaningfully affect pooling, especially with few groups. State its scale and explain what range of between-group differences it considers plausible.
Calling an inverse-gamma prior “noninformative.” Its shape and scale determine both tail and near-zero behavior. Inspect those implications rather than relying on the label.
Equating a small variance with exact pooling. A continuous prior can put posterior mass near zero but has zero probability at exactly zero. Exact pooling requires a point mass if it is to be a distinct posterior possibility.
Plugging in one estimate of \(\tau\) and ignoring its uncertainty. This omits the between-hyperparameter component of posterior variance and can make group-level uncertainty appear too small.
Confusing \(\tau\) with \(\tau^2\). The former is measured in outcome units and the latter in squared outcome units. A prior on one induces a transformed, not identical, prior on the other.

Key takeaway: A group-level variance component controls how much groups can differ and therefore how strongly their estimates are pooled. Its hyperprior influences both shrinkage and uncertainty; assess its units, scale, tails, and behavior near zero as part of the model specification.

Check Your Understanding

Try these questions without looking back.

In the random-intercept model, distinguish the roles and units of \(\sigma^2\) and \(\tau^2\).
Why does integrating out \(\alpha_j\) give a group-mean variance of \(\tau^2+v_j\)?
If \(\tau\) decreases while \(v_j\) and \(\mu\) stay fixed, what happens to the shrinkage weight and the conditional posterior mean?
For a continuous prior on \(\tau\), why can the posterior concentrate near zero without assigning positive probability to exact complete pooling?
In the conditional inverse-gamma example, why is the posterior update not itself the marginal posterior given only the observed individual outcomes?