Conjugate Variance Updates in Gaussian Hierarchical Models

Why a Gaussian Variance Can Have a Closed-Form Update

In a Gaussian hierarchical model, group-level effects are modeled as draws from a distribution whose spread is an unknown variance component. When the effects are treated as part of the unknown model state, their Gaussian density supplies a quadratic expression in that variance. An inverse-gamma prior has a matching form, so multiplying the prior density by the effect likelihood gives another inverse-gamma density.

The closed form depends on specific conditions: the random effects must have a Gaussian distribution with a variance proportional to the component being updated, and the prior must use the inverse-gamma parameterization stated below. This is a conditional result: it describes the distribution of the variance given the current values of the random effects and the other model quantities. It is not, by itself, a method for integrating out the random effects or a complete analysis of the marginal data likelihood.

Model and Assumptions for the Update

Consider \(J\) groups with scalar random intercepts \(u_1,\ldots,u_J\). For example, a Gaussian mixed model for a continuous outcome may be written \(y_{ij}=x_{ij}^{\mathsf T}\beta+u_j+\epsilon_{ij}\), where \(i\) indexes observations within group \(j\). For the variance update, suppose the outcome model and its parameters imply that the data likelihood depends on \(\tau^2\) only through the random effects \(u_j\). Conditional on those effects and the other parameters, the likelihood for \(\tau^2\) is therefore determined by their hierarchical distribution.

The assumptions for the basic update are:

• Conditional on \(\tau^2\), the group effects are independent and identically distributed as \(u_j\sim N(0,\tau^2)\).
• The inverse-gamma prior \(\tau^2\sim\operatorname{IG}(a,b)\) is independent of the other parameters before observing the data, with \(a,b>0\).
• The update conditions on the realized effects \(u_1,\ldots,u_J\) and on the remaining parameters. The likelihood for the observed outcomes has no additional direct dependence on \(\tau^2\) once the effects are given.

These assumptions specify what is being updated. In particular, \(J\) is the number of independent group effects in this variance component—not the number of individual observations. If effects have unequal known variances, or if the component multiplies a known covariance matrix, the quadratic term must reflect that structure.

Derivation of the Full Conditional

The joint conditional density of the effects given \(v=\tau^2\) is the product of \(J\) normal densities. Discarding factors that do not depend on \(v\), it is proportional to \(v^{-J/2}\exp[-\sum_j u_j^2/(2v)]\). Multiplying by the inverse-gamma prior kernel gives

This is the inverse-gamma kernel with shape \(a+J/2\) and scale \(b+\frac12\sum_j u_j^2\). Both updated parameters are positive because \(a,b>0\) and the sum of squares is nonnegative, so the kernel is normalizable. The conditional distribution is consequently proper under the stated proper prior.

Known Variance Multipliers and Vector Effects

The same calculation applies when a variance component is multiplied by a known positive quantity. Suppose independent scalar effects satisfy \(u_j\mid\tau^2\sim N(0,c_j\tau^2)\), where each \(c_j>0\) is known. Their densities contribute \((\tau^2)^{-J/2}\exp[-\sum_j u_j^2/(2c_j\tau^2)]\), apart from terms not depending on \(\tau^2\). Therefore the updated scale uses a weighted sum of squares, \(\frac12\sum_j u_j^2/c_j\), while the shape still increases by \(J/2\).

For a vector effect \(u_j\in\mathbb{R}^K\), suppose \(u_j\mid\tau^2\sim N_K(0,\tau^2 I_K)\), independently across groups. The density for each group contributes a factor \((\tau^2)^{-K/2}\) and an exponential term involving \(\lVert u_j\rVert^2/(2\tau^2)\). There are \(JK\) scalar Gaussian coordinates, so the shape increases by \(JK/2\), while the scale adds half the total squared Euclidean norm. This is a restrictive isotropic model: all coordinates share the same variance and have zero conditional correlations. It is not a general update for an unrestricted random-effect covariance matrix.

Interpretation, Diagnostics, and Alternatives

The updated shape and scale summarize how the prior and the currently conditioned-on effects combine. Larger squared effects increase the posterior scale, shifting the conditional distribution toward larger variances. More independent scalar effects increase the shape, which changes the distribution's concentration as well. These statements concern the conditional distribution; the variance's marginal posterior also accounts for uncertainty in the effects and other unknown quantities.

A useful diagnostic is to verify the model-to-formula match before calculating:

• Count independent scalar Gaussian coordinates. For \(J\) scalar intercepts use \(J/2\); for \(J\) independent \(K\)-vectors with covariance \(\tau^2I_K\), use \(JK/2\).
• Use the quadratic form implied by the covariance. With variance \(c_j\tau^2\), divide \(u_j^2\) by \(c_j\); with a more general known covariance matrix, use its corresponding inverse-covariance quadratic form.
• Check the inverse-gamma convention. Under the density used here, the scale adds to the numerator of the exponent as \(\exp(-B/v)\).
• Check units. The variance and inverse-gamma scale have squared outcome units, whereas a random effect has outcome units.

A frequent mistake is to substitute estimated group effects, such as empirical best predictions, into the formula and call the result the exact full conditional. The derivation conditions on the actual latent effects in the joint model. An estimate is not the same thing as conditioning on a latent effect draw; substituting it generally ignores uncertainty and does not produce the stated conditional distribution. Similarly, counting individual observations instead of random-effect coordinates overstates the shape increment.

This conjugate update is convenient when the Gaussian hierarchy and prior match its assumptions. Alternatives include using a different prior on the standard deviation, for which this inverse-gamma conditional form need not apply, or working with a marginal likelihood after integrating out effects, which yields a different calculation. Conjugacy also does not establish that the prior is substantively appropriate: prior sensitivity, especially when few groups inform a variance, remains important. Nor does a closed-form conditional alone resolve whether the hierarchical Gaussian model is adequate for the data.