Coherence and speed are different properties
Compatibility answers a question about the model: do the specified conditional distributions arise from one joint distribution? Convergence speed answers a different question: how quickly do repeated updates move through the regions of that joint distribution? A coherent target does not by itself ensure that an algorithm explores it quickly.
This distinction matters in fully conditional specification (FCS). If every conditional update is compatible with a proper joint distribution and the chain satisfies suitable regularity conditions, that joint distribution can be stationary for the update scheme. But stationarity describes what distribution is preserved, not how many iterations are needed to explore it adequately from a particular starting point. Strong conditional dependence can make individual updates small, so the chain may take a long time to move across the joint distribution.
A compatible Gaussian example
Consider standardized variables \(X\) and \(Y\) with a bivariate normal joint distribution, mean zero, unit marginal variances, and correlation \(\rho\), where \(-1<\rho<1\). Its density is
Completing the square in \(x\), while holding \(y\) fixed, gives a normal conditional distribution for \(X\). Completing the square in \(y\) gives the other conditional. Thus the conditionals are
The displayed conditionals are compatible by construction: they come from the joint density above. Suppose an FCS sweep first draws \(X\) given the current \(Y\), then draws \(Y\) given the updated \(X\). For this illustration, updates are exact random draws, and the correlation parameter is fixed. Those assumptions make the procedure a two-variable systematic-scan Gibbs chain. They allow us to isolate the relationship between dependence and movement without introducing parameter estimation or other sources of uncertainty.
Worked Example: Checking the Conditional Scale
Let \(\rho=0.8\). The conditional mean of \(X\) when \(Y=1.5\) is
The conditional variance and standard deviation are
Therefore \(X\mid Y=1.5\sim N(1.20,0.36)\), where the second argument is the variance. The conditional model is relatively concentrated compared with the marginal \(X\sim N(0,1)\). This narrower conditional distribution is a first indication that a single update may make only limited progress through the wider joint distribution.
Why strong dependence slows a full sweep
The effect becomes explicit by writing each random draw as its conditional mean plus a normal innovation. Let \(\varepsilon_t\) and \(\eta_t\) be independent \(N(0,1)\) random variables, independent across sweeps. One sweep is
Substituting the first update into the second gives a one-dimensional transition for \(Y\):
Conditional on \(Y_t\), the two innovation terms have mean zero and are independent. Their combined variance is \(\rho^2(1-\rho^2)+(1-\rho^2)=1-\rho^4\). Hence
After \(k\) full sweeps, repeated conditioning yields \(E(Y_{t+k}\mid Y_t)=\rho^{2k}Y_t\). The conditional variance after \(k\) sweeps, given the fixed starting value \(Y_t\), is \(1-\rho^{4k}\), which approaches 1 as \(k\) grows; if the chain starts in its stationary distribution, its marginal variance is 1. These expressions show how the chain's distribution relaxes from a fixed starting value. When \(|\rho|\) is close to 1, \(\rho^2\) is close to 1: the conditional mean retains much of its previous value at each sweep, and the conditional innovations are small relative to the marginal scale.
This calculation is a justification for the central distinction. The joint distribution is well-defined and preserved by the compatible Gibbs updates, but preservation does not remove the slow movement implied by the transition. Compatibility establishes a coherent stationary target; the transition equation reveals how the chosen update scheme approaches it.
Worked Example: Comparing Moderate and Strong Dependence
Suppose the chain starts at \(Y_0=2\), and compare two compatible bivariate normal models after 10 full sweeps. The mean after \(k\) sweeps is \(E(Y_k\mid Y_0)=\rho^{2k}Y_0\).
For \(\rho=0.5\), the per-sweep factor is \(\rho^2=0.25\). Thus
For \(\rho=0.95\), the per-sweep factor is \(\rho^2=0.9025\). Therefore
The two models are both compatible, but their chains behave very differently over these 10 sweeps. With \(\rho=0.5\), the expected value has moved close to zero, the marginal mean. With \(\rho=0.95\), the expected value remains about 0.717 units above that mean. This compares the distributional relaxation from the stated starting value; it does not mean that every realized chain follows its expected path.
What a narrow update looks like
When \(|\rho|\) is large, both conditional distributions have variance \(1-\rho^2\), which is small. For positive \(\rho\), the joint normal density is elongated roughly along the line \(x=y\); for negative \(\rho\), it is elongated roughly along \(x=-y\). A scalar FCS update changes one coordinate conditional on the other, so it tends to move within a narrow cross-section of that elongated distribution. Repeated updates can be needed to travel a substantial distance along its long direction.
Worked Example: One Sweep With Correlation 0.98
Take \(\rho=0.98\), begin with \(Y_0=2\), and suppose the standard-normal innovations for one sweep happen to be \(\varepsilon_0=0.3\) and \(\eta_0=-0.4\). The conditional standard deviation is \(\sqrt{1-0.98^2}=\sqrt{0.0396}\approx0.1990\). The first update is
The second update is
In this particular sweep, \(Y\) changes from 2 to about 1.900. The conditional expectation after the sweep, before observing innovations, is \(0.98^2(2)=1.9208\). These values illustrate the scale of movement, not a claim that every sweep changes \(Y\) by the same amount. Both conditionals are narrow relative to the unit marginal standard deviation, so a sequence of scalar updates may explore the joint distribution slowly.
Reading the distinction in practice
A chain can have the correct stationary distribution and still be a poor finite-run representation of it. In an imputation setting, that distinction matters because a chain that has not explored its target adequately can leave summaries of the imputed values sensitive to where the chain began or to a limited section of its path. This is a computational limitation, not proof that the conditional models are incompatible.
Some warning signs include long runs of small changes in strongly related variables, different apparent behavior from different starting regions, or summaries that shift substantially when more iterations are run. These observations do not diagnose the cause on their own: implementation errors, an incorrect target, and other chain problems can also affect behavior. In this example, the known transition factor \(\rho^2\) explains why strong dependence slows movement. In practical FCS, the analogous dependence may involve several variables and need not have such a simple expression.
The main alternatives differ in what they change. A joint or blocked update can move several strongly related variables together, if a suitable joint draw is available and targets the intended distribution. Scalar chained updates are simpler and can be used when the conditionals are specified separately, but can be inefficient for strongly dependent variables. Replacing one conditional model with a different one merely to obtain larger changes is not a guaranteed remedy: it can change the implied target or break compatibility. The appropriate comparison is therefore not “small updates versus large updates” alone, but whether an alternative preserves the intended model while improving exploration.
| Situation or approach | What it establishes | What it does not establish |
|---|---|---|
| Compatible scalar conditionals | A coherent joint target exists; under appropriate chain conditions, it is stationary. | That the chain reaches or explores the target quickly. |
| Incompatible conditional models | May still produce a stable iterative process in some settings. | A single joint distribution whose conditionals are all the specified models. |
| Joint or blocked updates | Can update dependent variables together when a valid joint update is available. | Automatic efficiency or validity if the update does not preserve the intended target. |
There is no universal number of iterations that makes a compatible chain “fast enough.” The rate depends on the dependence structure, the update scheme, the starting state, and the summaries of interest. Also, the Gaussian calculation is an illustrative case: it does not provide a general rate formula for arbitrary FCS models. In the Gaussian case, however, it makes the lesson precise: as \(|\rho|\) approaches 1, the full-sweep mean-retention factor \(\rho^2\) approaches 1, even though the conditional models remain compatible for every \(-1<\rho<1\).
Common mistake
- Wrong reasoning: “The conditionals are compatible, so their FCS chain must converge quickly.”
- Correction: Compatibility identifies a joint distribution consistent with the conditionals. It does not bound the time needed to explore that distribution. In the Gaussian example, compatibility holds for \(|\rho|<1\), while the full-sweep retention factor \(\rho^2\) can be arbitrarily close to 1.
- Another mistake: Treating a stable chain or a plausible-looking summary as evidence that the chain has explored all relevant parts of its target. Stability and exploration are separate issues; neither a coherent model nor a stable-looking finite run alone guarantees adequate exploration.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- For the standardized bivariate normal conditionals, what is the conditional variance when \(\rho=0.6\), and how does it compare with the marginal variance?
- In a systematic sweep that updates \(X\) and then \(Y\), what is the conditional expectation \(E(Y_{t+1}\mid Y_t)\)?
- If \(\rho=0.9\) and \(Y_0=1\), calculate \(E(Y_3\mid Y_0)\) after three full sweeps.
- Why does a highly correlated compatible model potentially produce slow exploration under scalar updates?
- Does finding that a chain is stable establish that its conditional models are compatible? Explain.
Solutions. (1) The conditional variance is \(1-0.6^2=0.64\), while the marginal variance is 1, so the conditional distribution is narrower than the marginal. (2) Substitution of the \(X\) update into the \(Y\) update gives \(E(Y_{t+1}\mid Y_t)=\rho^2Y_t\). (3) The per-sweep factor is \(0.9^2=0.81\), so \(E(Y_3\mid Y_0=1)=1(0.81)^3=0.531441\), approximately 0.5314. (4) Strong dependence makes the conditional variance \(1-\rho^2\) small; each scalar update changes a variable within a narrow conditional distribution, so substantial movement through the joint distribution can take many sweeps. (5) No. Stability concerns behavior of the iterative process, whereas compatibility asks whether a joint distribution has all the specified conditionals. A stable chain can be based on incompatible conditionals.