Gibbs Sampling From Conjugate Full Conditionals

Why Cycle Through Conditional Distributions?

A trial model may have a simple likelihood but parameters that are dependent in the posterior. Gibbs sampling addresses this by repeatedly drawing each parameter from its distribution conditional on the current values of the others. The key requirement is that these full conditional distributions can be sampled from. Each conditional update leaves the joint posterior unchanged, so a complete cycle also preserves it.

We develop the method for a continuous outcome in one trial arm. This normal model is deliberately conjugate: its joint posterior and full conditionals can both be derived exactly. That makes it possible to check every step of the sampler. In this example direct draws from the joint posterior are also available, so Gibbs sampling is a teaching device rather than a computational necessity. In more complicated models, conditionals may remain tractable even when a direct joint draw is inconvenient.

We use the inverse-gamma shape-and-scale convention \(p(v)=b^a v^{-a-1}\exp(-b/v)/\Gamma(a)\) for \(v>0\), when \(v\sim\operatorname{InvGamma}(a,b)\). Thus \(b\) appears in the numerator of the exponent as \(-b/v\). The prior is proper under the stated positive hyperparameters. The normal likelihood and this prior give a proper posterior.

Deriving the Two Full Conditionals

Write \(\bar{x}\) for the sample mean and \(\sum_i(x_i-\bar{x})^2\) for the within-sample sum of squares. The joint posterior density is proportional to likelihood times joint prior. Keeping only terms that depend on the parameters gives

For the first full conditional, hold \(\sigma^2\) fixed and collect the quadratic terms in \(\theta\). Completing the square gives a normal distribution. For the second, hold \(\theta\) fixed; the power of \(\sigma^2\) and the exponent have the inverse-gamma form. These are the distributions used for the Gibbs updates.

The \(n+1\) in the second shape parameter matters. The likelihood contributes \(n/2\) to the power of \((\sigma^2)^{-1}\), and the conditional normal prior \(p(\theta\mid\sigma^2)\) contributes another \(1/2\). Omitting the latter produces an incorrect full conditional.

A Gibbs Cycle and Why It Preserves the Posterior

Choose starting values \(\theta^{(0)}\) and \((\sigma^2)^{(0)}>0\). In each iteration, draw \(\theta\) conditional on the current variance, then draw the variance conditional on the newly updated mean. Repeat the cycle. Each draw must be from the stated full conditional, not from a marginal distribution or from a conditional based on stale values.

The preservation result follows directly from conditional probability. Suppose a pair \((\theta,\sigma^2)\) currently has the joint posterior distribution. Updating only \(\theta\) by drawing from \(p(\theta\mid\sigma^2,x)\) gives the joint density of the updated pair as

The variance marginal is unchanged, and conditional on that variance the new mean has exactly the posterior conditional distribution. The updated pair therefore still has the joint posterior distribution. The same argument applies when updating \(\sigma^2\) conditional on \(\theta\). Since composing two updates that each preserve a distribution also preserves it, a full Gibbs cycle preserves the joint posterior. This establishes invariance; it does not by itself establish that a chain from every starting point has converged after a chosen finite number of iterations. Also, a systematic mean-then-variance cycle need not itself satisfy detailed balance, even though it preserves the target distribution.

Dependence, Checks, and Alternatives

Several implementation checks follow from the derivation:

• Check the inverse-gamma shape. It is \(a_0+(n+1)/2\) for this conditional variance update; the extra half comes from \(p(\theta\mid\sigma^2)\).
• Use the updated mean. In the stated scan order, calculate the variance conditional on \(\theta^{(t+1)}\), not \(\theta^{(t)}\).
• Check units and support. The mean and its standard deviation are in outcome units; the variance and inverse-gamma scale are in squared outcome units. The variance draws must be positive.
• Check that the model matches the data. Conditional independence and a common normal distribution are substantive assumptions. Strong skewness, outliers, or important differences in variability may make this model inadequate. A correctly implemented sampler does not repair a misspecified likelihood.

A common mistake is to say that both parameters are drawn independently because each is updated separately. They are not: each draw is conditional on the current value of the other parameter, and the prior itself is specified by \(p(\theta\mid\sigma^2)p(\sigma^2)\), not by a product of independent marginal priors. Another mistake is to interpret invariance as a guarantee that a short run is representative. Invariance identifies the target distribution of the transition; assessing whether a finite simulated chain adequately explores that target requires careful computation and diagnostics.

Compared with a direct draw from the Normal–Inverse-Gamma posterior, Gibbs sampling is less efficient for this simple example: the conjugate joint posterior can be sampled directly, avoiding serial dependence between successive states. Gibbs becomes useful when direct joint simulation is difficult but the full conditionals are available. Compared with Metropolis–Hastings, an exact Gibbs update needs no proposal distribution or acceptance decision. If a full conditional cannot be sampled directly, a different update strategy is needed; the conditional step is then no longer an exact Gibbs draw. The method’s limitations are therefore practical as well as modeling-related: conditionals must be derived correctly, dependence can make movement through the joint posterior slow, and a preserved target does not guarantee rapid exploration.