Metropolis-Within-Gibbs Updates

Combining Metropolis-Hastings and Gibbs Updates

In a multivariate model, one parameter may have a convenient full conditional distribution while another does not. Metropolis-within-Gibbs handles this by using an exact Gibbs draw for the convenient parameter and a Metropolis-Hastings (MH) update for the difficult one. The resulting sweep is still a Markov transition on the full parameter vector. To justify it, we will show that each update preserves the specified joint target; then their ordered composition preserves it as well.

Consider a finite product state space \(S=\mathcal X\times\mathcal Y\times\mathcal Z\). A state is \((x,y,z)\). Suppose the target probability mass function has the factorization \(\pi(x,y,z)=Z^{-1}\phi(x,y)\psi(y,z)\), where all factor values are strictly positive and \(Z\) is the sum of the unnormalized weights over all states. Strict positivity ensures that each conditional distribution used below is defined. We use a fixed systematic sweep: update \(x\) by MH conditional on the current \(y,z\), draw \(y\) exactly conditional on the updated \(x\) and current \(z\), and then draw \(z\) exactly conditional on the updated \(x,y\).

Let \(q(x'\mid x)\) be the proposal probability for the \(x\)-update. For fixed \(y\), the target conditional mass of \(x\) is proportional to \(\phi(x,y)\): the factor \(\psi(y,z)\) does not depend on \(x\), so it cancels in the conditional distribution. The MH acceptance probability for a proposal from \(x\) to \(x'\) is therefore \(\alpha_y(x,x')=\min\{1,\phi(x',y)q(x\mid x')/[\phi(x,y)q(x'\mid x)]\}\) whenever the proposal probabilities in the ratio are positive. A proposal with \(q(x'\mid x)=0\) is never made; if \(q(x\mid x')=0\) while \(q(x'\mid x)>0\), the acceptance probability is zero. Rejection leaves \(x\) unchanged. The \(y\)- and \(z\)-updates draw, respectively, from masses proportional to \(\phi(x,y)\psi(y,z)\) and \(\psi(y,z)\).

Exact Conditional Draws in the Same Sweep

For this factorization, the full conditional distributions needed for the Gibbs updates follow directly by holding the other coordinates fixed and normalizing over the coordinate being updated: \[ \mathbb P(Y=y\mid X=x,Z=z) =\frac{\phi(x,y)\psi(y,z)}{\sum_{\tilde y\in\mathcal Y}\phi(x,\tilde y)\psi(\tilde y,z)}, \qquad \mathbb P(Z=z\mid X=x,Y=y) =\frac{\psi(y,z)}{\sum_{\tilde z\in\mathcal Z}\psi(y,\tilde z)}. \]

Why the Ordered Sweep Preserves the Target

Proof. First fix \(y\), and let \(A_y=\sum_x\phi(x,y)\). The conditional target for \(x\) is \(m_y(x)=\phi(x,y)/A_y\). For distinct \(x,x'\) with both proposal probabilities positive, the probability flow from \(x\) to \(x'\) under the MH transition is \[ m_y(x)q(x'\mid x)\alpha_y(x,x') =\frac{1}{A_y}\min\{\phi(x,y)q(x'\mid x),\phi(x',y)q(x\mid x')\}. \] The expression is unchanged when \(x\) and \(x'\) are interchanged. If either proposal probability is zero, both directed accepted flows are zero, so the equality still holds. Thus the accepted flows balance in both directions. Rejected proposals stay at their starting state and contribute only to diagonal flow. Summing the balanced flows over the starting state shows \(m_yR_y=m_y\), where \(R_y\) is the conditional MH transition matrix. For each fixed \(y,z\), the target conditional of \(x\) is \(m_y\), since \(\psi(y,z)\) cancels from its normalization. Therefore \(K_x\) preserves the full joint target.

Next, write \(\pi(x,z)=\sum_y\pi(x,y,z)\). Under \(K_y\), the probability of arriving at \((x,y',z)\) is \[ \sum_y\pi(x,y,z)\mathbb P(Y=y'\mid X=x,Z=z) =\pi(x,z)\mathbb P(Y=y'\mid X=x,Z=z) =\pi(x,y',z). \] The first equality uses that the conditional draw is the same for every old \(y\) when \(x,z\) are fixed. The second is the definition of the conditional probability. The same calculation for the \(z\)-update gives \(\sum_z\pi(x,y,z)\mathbb P(Z=z'\mid X=x,Y=y)=\pi(x,y,z')\). Thus \(K_y\) and \(K_z\) each preserve \(\pi\). Using row-vector convention, \(\pi K=\pi K_xK_yK_z=\pi\), because each factor in turn leaves \(\pi\) unchanged. \(\square\)

What Invariance Does—and Does Not—Say

Invariance guarantees that the target is unchanged by one sweep when the chain starts with the target distribution. It does not say that one sweep transforms every starting state into a target draw, nor does it by itself establish that a chain will converge to the target. Questions about convergence require additional properties of the transition, considered separately.

Common mistake: omitting the reverse proposal probability. In the first example, using only the target ratio for \(0\to1\) would give the incorrect acceptance probability \(\tfrac12\). The correct value is \(\tfrac16\), because the reverse proposal probability is \(q(0\mid1)=\tfrac13\). With the incorrect value, the flow from \(0\) to \(1\) would be \(\tfrac47\cdot\tfrac12=\tfrac27\), while the reverse flow under the correct \(1\to0\) transition would be \(\tfrac27\cdot\tfrac13=\tfrac{2}{21}\). The flows do not balance. Including the reverse proposal gives the correct forward flow \(\tfrac47\cdot\tfrac16=\tfrac{2}{21}\), matching the reverse flow.

A second practical safeguard is to use the conditional target for the coordinate being updated, not the whole joint weight without simplification. In this example, the \(x\)-update conditions on \(y,z\), but \(\psi(y,z)\) cancels because it is constant as \(x\) varies. The \(y\)-update, in contrast, depends on both factors. Finally, update order is part of the definition of a sweep: the \(y\)-draw here uses the newly updated \(x\), and the \(z\)-draw uses the newly updated \(y\).