Metropolis-Hastings Updates for Variance Components

Why a Variance-Component Update Needs Care

A variance component must be positive, while a computational proposal may be easier to define on an unconstrained scale. Metropolis–Hastings (MH) provides a general way to move between candidate values while preserving a specified posterior distribution. For variance components, the essential care is to target the density on the scale on which the proposal is made. Transforming from a variance to its logarithm changes the density through a Jacobian; omitting it changes the posterior being sampled.

Consider a GLMM with group-specific random intercepts $b_1,\ldots,b_J$, where $b_j\mid v\sim N(0,v)$ and $v=\tau^2>0$. A variance-component update typically conditions on the observed data, fixed effects, random effects, and any other parameters currently held fixed. The resulting conditional posterior for $v$ is the target for that update. It need not have a convenient standard form, even when a normal prior is used for the random effects.

Definition: Let $\pi(x)$ be a target density, known up to a positive constant, and let $q(x^*\mid x)$ be a proposal density for a candidate $x^*$ given the current value $x$. The Metropolis–Hastings acceptance probability is $$ \alpha(x,x^*)=\min\left\{1,\frac{\pi(x^*)q(x\mid x^*)}{\pi(x)q(x^*\mid x)}\right\}. $$ The proposed value is accepted with probability $\alpha(x,x^*)$; otherwise the chain remains at $x$. The proposal ratio is essential unless it cancels by symmetry.

Deriving the Acceptance Rule on the Log-Variance Scale

Write $\pi_v(v\mid \cdots)$ for the conditional posterior density with respect to $v$, where the ellipsis denotes the data and other parameters held fixed. Define the log-variance $\eta=\log v$, so $v=\exp(\eta)$. A density must be transformed when its variable changes. Since $dv/d\eta=\exp(\eta)=v$, the target density with respect to $\eta$ is

\pi_\eta(\eta\mid\cdots)=\pi_v(\exp(\eta)\mid\cdots)\exp(\eta).

Suppose a random-walk proposal is made on the log scale: $\eta^*=\eta+\epsilon$, with $\epsilon$ drawn from a distribution symmetric about zero. Then $q_\eta(\eta^*\mid\eta)=q_\eta(\eta\mid\eta^*)$. The proposal densities cancel in the MH ratio, leaving

\alpha(\eta,\eta^*)=\min\left\{1,\frac{\pi_v(v^*\mid\cdots)v^*}{\pi_v(v\mid\cdots)v}\right\},\qquad v=\exp(\eta),\quad v^*=\exp(\eta^*).

The factor $v^*/v$ is the Jacobian contribution expressed as a ratio. Equivalently, compute the ratio using the transformed target $\pi_\eta$. Both routes give the same answer. A symmetric proposal on the log scale is generally not symmetric as a proposal density on the variance scale, so it is not valid to use the variance-scale target ratio alone.

Worked Example: Forming a Conditional Target for a Random-Intercept Variance

Suppose a hypothetical logistic random-intercept GLMM has two groups, with current random effects $b_1=0.5$ and $b_2=-0.3$. Consider updating $v$ while holding the data, fixed effects, and these random effects fixed. Give $v$ an exponential prior with rate $\lambda=2$, so $p(v)=2\exp(-2v)$ for $v>0$.

The conditional data likelihood given the random effects does not depend directly on $v$, so it cancels when comparing candidate values. The normal random-effects density contributes $v^{-J/2}\exp[-\sum_j b_j^2/(2v)]$. Here $J=2$ and

$\sum_j b_j^2=0.5^2+(-0.3)^2=0.25+0.09=0.34$.

Thus, up to a constant independent of $v$, the conditional target on the variance scale is

\pi_v(v\mid b_1,b_2,y)\propto v^{-1}\exp\left(-\frac{0.34}{2v}\right)\exp(-2v),\qquad v>0.

On the log-variance scale, multiply this density by $v$. The $v^{-1}$ term cancels, giving $\pi_\eta(\eta\mid b_1,b_2,y)\propto \exp[-0.34/(2v)-2v]$. Compare current $v=0.25$ with proposed $v^*=0.50$, using a symmetric proposal in $\eta$. The log target ratio is

\log\frac{\pi_\eta(\eta^*)}{\pi_\eta(\eta)} =-\frac{0.34}{2}\left(\frac{1}{0.50}-\frac{1}{0.25}\right)-2(0.50-0.25) =-0.17(2-4)-0.50=0.34-0.50=-0.16.

Therefore the acceptance probability is $\min\{1,\exp(-0.16)\}=0.8521$. The candidate is accepted with probability about 0.852; otherwise the chain retains $v=0.25$. This calculation uses the conditional posterior, not just the prior or the random-effects density.

A Numerical Update with a Log-Scale Random Walk

For independent normal random effects $b_j\mid v\sim N(0,v)$, let $S=\sum_{j=1}^J b_j^2$. If the data likelihood has no additional direct dependence on $v$ after conditioning on the random effects, and the prior density for $v$ is $p(v)$, then

\pi_v(v\mid b,y,\ldots)\propto v^{-J/2}\exp\left(-\frac{S}{2v}\right)p(v).

This expression assumes the stated normal random-effects distribution and a prior supported on $v>0$. Any other factors in the conditional posterior that depend on $v$ must also be included. The data likelihood may cancel from this particular conditional update, but it does not disappear from the full posterior target.

Worked Example: Accepting or Rejecting a Log-Variance Proposal

Take hypothetical current random effects $b_1=0.4$, $b_2=-0.2$, and $b_3=0.6$, so $J=3$ and $S=0.4^2+(-0.2)^2+0.6^2=0.16+0.04+0.36=0.56$. Use the exponential prior $p(v)=\exp(-v)$, with rate $1$. The log-scale target, including the Jacobian, is proportional to

\pi_\eta(\eta\mid b,y)\propto v^{1-J/2}\exp\left(-\frac{S}{2v}-v\right) =v^{-1/2}\exp\left(-\frac{0.56}{2v}-v\right).

Suppose the current value is $v=0.25$, and the proposal is a symmetric log-scale random walk with $\eta^*=\eta+0.8$. Then $v^*=0.25\exp(0.8)=0.5564$. The log target ratio is

\begin{aligned} \log\frac{\pi_\eta(\eta^*\mid b,y)}{\pi_\eta(\eta\mid b,y)} &=-\frac{1}{2}\log\left(\frac{0.5564}{0.25}\right) -\frac{0.56}{2}\left(\frac{1}{0.5564}-\frac{1}{0.25}\right) -(0.5564-0.25)\\ &\approx -0.4000+0.6168-0.3064=-0.0896. \end{aligned}

The acceptance probability is $\min\{1,\exp(-0.0896)\}=0.9143$. For example, if the algorithm’s acceptance draw is $0.70$, the proposal is accepted because $0.70<0.9143$. If the draw is $0.95$, it is rejected and the next state is the old value $v=0.25$. A rejection is part of the Markov chain; it is not a missing draw.

Proposal Asymmetry and Equivalent Calculations

The proposal correction matters whenever the proposal is asymmetric. For example, an additive proposal on $\eta$ with a nonzero mean increment is not symmetric. If $\eta^*=\eta+\epsilon$, where $\epsilon\sim N(m,s^2)$, and $\delta=\eta^*-\eta$, then

\log\frac{q_\eta(\eta\mid\eta^*)}{q_\eta(\eta^*\mid\eta)} =-\frac{2\delta m}{s^2}.

Consequently the log MH ratio is the log transformed-target ratio plus this proposal log ratio. Alternatively, if a candidate is proposed directly on the variance scale, use the variance-scale target and the corresponding variance-scale proposal densities. Do not mix a target density from one scale with a proposal ratio from another without transforming the densities.

Worked Example: Correcting for a Drifted Log-Scale Proposal

Use the target and current state from the preceding example, with $v=0.25$, $v^*=0.5564$, and log-target ratio approximately $-0.0896$. Now suppose the proposal increment has distribution $N(m,s^2)$ with $m=0.10$ and $s=0.50$. The increment is $\delta=\log(v^*/v)=0.8$, so the proposal log ratio is

-\frac{2\delta m}{s^2}=-\frac{2(0.8)(0.10)}{0.50^2}=-0.64.

The total log MH ratio is $-0.0896-0.64=-0.7296$, and the acceptance probability is $\exp(-0.7296)=0.4820$. The proposal is less likely to be accepted than in the symmetric-proposal example because the proposal mechanism favors positive increments, so the reverse move receives a correction. Ignoring the proposal ratio would incorrectly give an acceptance probability of 0.9143.

Checks, Alternatives, and Limitations

Before interpreting an update, check that the target includes every factor depending on the variance component, that its support is $v>0$, and that the proposal ratio is calculated on the proposal’s scale. A useful algebra check is to derive the acceptance ratio in two ways when feasible: transform the variance-scale target with its Jacobian, and then form the ratio on the log scale. The two results should agree.

A chain that repeatedly proposes invalid values, contains non-finite log densities, or rarely moves may indicate an implementation problem or a poorly explored target. Verify that $v=\exp(\eta)$ is used consistently, that the normal random-effects contribution is $v^{-J/2}$ rather than $v^{-J}$, and that rejected proposals leave the state unchanged. Acceptance alone does not establish that posterior summaries are reliable; the chain must explore the relevant posterior adequately.

When a full conditional distribution can be sampled directly, a Gibbs update can replace the MH step and does not require a proposal acceptance decision. For nonconjugate variance-component conditionals, MH offers flexibility, but its performance depends on the proposal and it produces correlated draws. It also relies on a correctly specified, suitably explored posterior target. It does not correct a poor prior, model misspecification, or weak information about a variance component.

Common pitfall: A symmetric random walk in $\eta=\log v$ does not justify using only $\pi_v(v^*)/\pi_v(v)$. The target must first be expressed as a density in $\eta$, which contributes the Jacobian $v$. The correction is to use $\pi_\eta(\eta)=\pi_v(\exp(\eta))\exp(\eta)$, or equivalently to include the factor $v^*/v$ in the ratio.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

If $v=\exp(\eta)$, what Jacobian factor converts a density with respect to $v$ to a density with respect to $\eta$?
A symmetric proposal is made on $\eta$. If the log transformed-target ratio is $0.12$, what is the acceptance probability?
For a random-intercept model with $J=4$ and random effects $0.2,-0.1,0.3,-0.2$, calculate $S$.
Why must an asymmetric proposal include a proposal-density ratio?

Solutions. (1) Since $dv/d\eta=\exp(\eta)=v$, the transformed density is $\pi_\eta(\eta)=\pi_v(v)v$. (2) The ratio is $\exp(0.12)>1$, so the acceptance probability is $\min\{1,\exp(0.12)\}=1$. (3) $S=0.2^2+(-0.1)^2+0.3^2+(-0.2)^2=0.04+0.01+0.09+0.04=0.18$. (4) The proposal probability of moving from the candidate back to the current value may differ from the probability of proposing the candidate, so the ratio corrects the transition probabilities to preserve the target posterior.