Gibbs Sampling from Full Conditional Distributions

Updating One Component at a Time

In Markov chain Monte Carlo, a target distribution may be difficult to sample from jointly even when sampling one variable at a time is straightforward. Gibbs sampling uses this structure: retain the current value of one component, and draw the other from its conditional distribution given the retained value. The resulting transition is designed around the target itself, so its invariance can be checked directly from the definition of conditional probability.

Let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space, and let the state space be a product \(E_1\times E_2\). Suppose the target distribution has a density \(\pi(x,y)\) with respect to a product reference measure \(\nu_1(dx)\nu_2(dy)\). Assume \(\pi\geq 0\) and \(\int_{E_1}\int_{E_2}\pi(x,y)\nu_2(dy)\nu_1(dx)=1\). Define the marginal densities \(\pi_1(x)=\int_{E_2}\pi(x,y)\nu_2(dy)\) and \(\pi_2(y)=\int_{E_1}\pi(x,y)\nu_1(dx)\).

A Gibbs update of the first component starts from \((X_n,Y_n)=(x,y)\), draws \(X_{n+1}\) from \(\pi(\,\cdot\mid y)\), and sets \(Y_{n+1}=y\). A Gibbs update of the second component instead draws \(Y_{n+1}\) from \(\pi(\,\cdot\mid x)\) and keeps \(X_{n+1}=x\). Thus each update changes one coordinate, and the conditional used for that draw depends on the coordinate that is held fixed.

For clarity, the first-coordinate transition kernel is, for measurable sets \(A\subseteq E_1\) and \(B\subseteq E_2\),

The current value \(x\) does not appear on the right-hand side: after the update, the new first coordinate depends on the old state through \(y\), while the second coordinate remains exactly \(y\). The analogous second-coordinate kernel is \(K_Y((x,y),A\times B)=\mathbf{1}_A(x)\int_B\pi(v\mid x)\nu_2(dv)\).

Why a Full Conditional Update Preserves the Target

Proof. Consider first the update of the first coordinate. The current second coordinate has marginal density \(\pi_2(y)\), and, given \(Y_n=y\), the new first coordinate has density \(\pi(x'\mid y)\). Therefore the joint density of \((X_{n+1},Y_{n+1})\), at \((x',y)\), is \(\pi_2(y)\pi(x'\mid y)\). For every \(y\) with \(\pi_2(y)>0\), the definition of the conditional gives \(\pi_2(y)\pi(x'\mid y)=\pi(x',y)\). Values with \(\pi_2(y)=0\) contribute zero to the joint distribution. Hence the updated density equals \(\pi(x',y)\) almost everywhere. For the second-coordinate update, the same calculation gives \(\pi_1(x)\pi(y'\mid x)=\pi(x,y')\) wherever \(\pi_1(x)>0\), and the zero-marginal values again contribute zero. Thus either update preserves the joint target. \(\square\)

This calculation expresses the key idea: the update keeps the distribution of the fixed coordinate equal to its target marginal and then draws the changed coordinate from the target conditional given that value. Multiplying the marginal and conditional reconstructs the joint distribution. Invariance is a statement about what happens when the chain is already distributed according to \(\pi\); by itself it does not say that a chain started elsewhere will approach \(\pi\).

Proof. The theorem shows that applying either update to a state with distribution \(\pi\) produces a state with distribution \(\pi\). After the first update the distribution is therefore still \(\pi\), so the theorem applies again to the next update. Repeating this argument for each update in the finite sequence proves the assertion. \(\square\)

Worked Conditional Calculations

What Invariance Does—and Does Not—Tell Us

Gibbs updates are useful when the full conditionals are easier to sample from than the joint target. In the two-variable setting, the basic construction requires only the joint density, its marginals, and the ability to draw from the resulting conditional distributions. The invariance proof does not require a special form for the density, such as independence or a Gaussian structure. It does require correct conditionals, including the correct support and normalizing constants.

A common mistake is to use a conditional distribution that ignores the support imposed by the joint target. For the triangular target above, one might incorrectly draw \(X\) uniformly on \((0,1)\) while holding \(Y=y\). At \(y=\tfrac14\), this incorrect rule assigns probability \(\tfrac14\) to \(0<X<\tfrac14\), since the interval has length \(\tfrac14\). But the target assigns probability zero to that region: its support requires \(X>Y\). The correct conditional is uniform on \((\tfrac14,1)\), which assigns zero probability below \(\tfrac14\). The error is not a harmless change in sampling efficiency; it uses the wrong conditional and fails to preserve the stated target.

There is also a separate limitation to keep in mind. A one-coordinate update leaves the other coordinate unchanged, so that single update cannot explore the whole product state space from every starting point. In the continuous case, for instance, a first-coordinate update retains the initial \(y\) exactly. Invariance is therefore not a convergence theorem. Whether a sequence of updates can explore the relevant support, and whether its distributions converge to the target, are separate questions that require additional conditions.