FCS and the Gibbs-sampling analogy
Fully conditional specification (FCS) updates incomplete variables one at a time using models conditional on the other variables. That sequential structure resembles Gibbs sampling, which also updates one variable at a time. But the resemblance does not by itself make an FCS chain a Gibbs sampler for a joint probability model. The key question is whether the conditional distributions used in the updates can all be obtained from one common joint distribution.
This distinction matters for interpretation. If the updates are the full conditional distributions of a specified joint model, the chain can be understood as a Gibbs sampler targeting that model. If the conditionals do not belong to any common joint distribution, the updates may still define an iterative imputation procedure, but the usual Gibbs interpretation is unavailable. The algorithm's sequential form alone cannot establish that a joint target exists.
In missing-data applications, the joint distribution of interest may be understood conditional on observed data. For example, with two variables \(X\) and \(Y\), an update might draw a missing \(X\) from a model for \(X\mid Y\), then draw a missing \(Y\) from a model for \(Y\mid X\). For these updates to be Gibbs updates, both models must be the corresponding full conditionals of the same joint distribution, given the observed information. FCS does not require that this condition hold merely to specify and run its conditional updates.
What Gibbs sampling requires
Suppose a joint distribution for variables \(X_1,\ldots,X_p\) is specified. Its full conditional for \(X_j\) is the distribution of \(X_j\) given all the other variables, \(X_{-j}\). A systematic Gibbs sweep updates \(X_1\), then \(X_2\), and so on, drawing each variable from that joint distribution's full conditional given the current values of the others. The defining requirement is not just that each step is stochastic: each update must use the appropriate conditional from the same joint target.
In FCS, the analyst instead specifies a conditional model for each variable with missing values. These models may use different distributional forms to suit different variable types—for instance, a linear regression for a continuous variable and a logistic regression for a binary variable. A sequence of draws from these models resembles a Gibbs sweep. However, the collection of fitted conditional models need not correspond to the full conditionals of a single joint distribution. Different model forms are not automatically incompatible, but neither do they guarantee compatibility.
The distinction is about the probability models, not whether the updates are deterministic or stochastic. A deterministic cycling procedure is not Gibbs sampling because it does not draw from full conditional distributions. Conversely, stochastic draws are not sufficient for a Gibbs interpretation if their conditional distributions cannot arise from one joint model. Parameter draws and residual variation can make FCS updates stochastic without resolving this issue.
A consistency condition from a joint distribution
A short derivation shows why conditional distributions cannot be chosen arbitrarily. Let \(X\) and \(Y\) each be binary, and suppose a proposed joint distribution assigns positive probability \(q_{xy}=P(X=x,Y=y)\) to each of the four possible pairs. The odds of \(X=1\) rather than \(X=0\), conditional on \(Y=y\), must be
Likewise, the odds of \(Y=1\) rather than \(Y=0\), conditional on \(X=x\), are \(q_{x1}/q_{x0}\). Multiplying the two conditional-odds ratios across the opposite values gives the same expression in either direction:
Therefore, if the four cells have positive probability, the change in conditional odds for \(X\) as \(Y\) changes must match the change in conditional odds for \(Y\) as \(X\) changes. This is a necessary consistency condition: it follows from the existence of the same four joint probabilities in both sets of conditionals. It is not a general recipe for checking every complicated FCS model, but it can expose incompatibility in a simple case.
Worked Example: Conditional Models That Are Compatible
Consider a hypothetical joint distribution for two binary variables, \(X\) and \(Y\), with probabilities \(P(X=0,Y=0)=0.4\), \(P(X=1,Y=0)=0.1\), \(P(X=0,Y=1)=0.1\), and \(P(X=1,Y=1)=0.4\). The probabilities are positive and sum to \(1\). Derive the two conditional models.
For \(X\mid Y=0\), the total probability at \(Y=0\) is \(0.4+0.1=0.5\). Thus \(P(X=1\mid Y=0)=0.1/0.5=0.2\). For \(X\mid Y=1\), the total is also \(0.5\), so \(P(X=1\mid Y=1)=0.4/0.5=0.8\).
For \(Y\mid X=0\), the total probability at \(X=0\) is \(0.4+0.1=0.5\), giving \(P(Y=1\mid X=0)=0.1/0.5=0.2\). For \(Y\mid X=1\), the total is \(0.1+0.4=0.5\), giving \(P(Y=1\mid X=1)=0.4/0.5=0.8\).
Both conditional models were obtained from the displayed joint distribution, so they are compatible by construction. Using these full conditionals for successive draws is Gibbs sampling for that joint distribution. For instance, when the current value is \(Y=0\), a Gibbs update draws \(X=1\) with probability \(0.2\), not with a probability chosen independently of the joint model.
Worked Example: Detecting an Incompatible Pair
Now suppose the proposed conditional models are \(P(X=1\mid Y=0)=0.2\), \(P(X=1\mid Y=1)=0.8\), \(P(Y=1\mid X=0)=0.2\), and \(P(Y=1\mid X=1)=0.6\). Assume a candidate common joint distribution would give positive probability to all four pairs. Check whether these conditionals can come from that joint distribution.
The conditional odds for \(X=1\) are \(0.2/0.8=0.25\) when \(Y=0\), and \(0.8/0.2=4\) when \(Y=1\). Their ratio is \(4/0.25=16\).
For \(Y=1\), the conditional odds are \(0.2/0.8=0.25\) when \(X=0\), and \(0.6/0.4=1.5\) when \(X=1\). Their ratio is \(1.5/0.25=6\). The necessary consistency condition requires these two ratios to be equal, but \(16\ne6\). Therefore no joint distribution with positive probability on all four pairs can have all four of these conditionals.
The conclusion is specific to the stated positive-support assumption. If some joint cells have probability zero, odds ratios involving those cells may not be defined, and a support-based analysis is needed instead. Under the stated assumption, however, the proposed updates cannot be full conditionals of a common joint model. Cycling through them may still define an FCS procedure, but it is not Gibbs sampling for such a joint distribution.
Why FCS may still be useful
The absence of a common joint distribution does not, by itself, show that an FCS procedure is useless or that its imputations are necessarily poor. FCS lets the analyst specify a conditional model suited to each variable's scale and type. This flexibility can be practical when a single joint model is difficult to formulate. The key is to avoid claiming that the updates are Gibbs sampling from a joint model unless that claim is justified.
Worked Example: Interpreting a Mixed-Variable FCS Specification
Consider a hypothetical data set with a continuous outcome \(Y\), a binary exposure \(B\), and a count covariate \(C\), all of which have missing values. An analyst specifies a normal linear regression for \(Y\mid B,C\), a logistic regression for \(B\mid Y,C\), and a Poisson regression for \(C\mid Y,B\), then cycles through draws from those models.
The specification is clearly an FCS procedure: each variable has a conditional model using the other variables as predictors, and the updates are stochastic. But these facts do not establish that a single joint distribution has precisely these three conditionals. In particular, the models were specified separately, so their mutual consistency has not been demonstrated.
A careful report can state that the missing values were generated by chained conditional models, describe the model used for each variable, and explain the diagnostics used to assess the procedure. It should not call the procedure a Gibbs sampler for a joint model merely because it cycles through conditional draws. Whether a particular set of models is compatible requires a separate argument.
Comparison, diagnostics, and limitations
| Question | Gibbs sampling | FCS specification |
|---|---|---|
| What is specified? | A joint target distribution, with its full conditionals used for updates | A collection of conditional models, often chosen variable by variable |
| What justifies the updates? | Each update is derived from the same joint target | Each update follows its specified conditional model; a common joint target is not guaranteed |
| Can the methods coincide? | Yes, when the FCS conditionals are the full conditionals of a joint model and the update procedure uses them accordingly | Yes; compatibility makes a Gibbs interpretation possible, but is not implied by cycling alone |
In practice, the main warning sign is a mismatch between the strength or form of relationships implied by different conditional models. The binary example showed one precise way to diagnose such a mismatch. For larger or more complex models, establishing compatibility can be substantially harder; visual stability of the imputations or completion of many sweeps is not a proof that a common joint distribution exists. Likewise, a chain that appears to settle down does not by itself establish that its conditionals are compatible.
There are limits to what the distinction tells us. Compatibility asks whether the conditional models can be recovered from one joint distribution; it does not establish that the models describe the data well. Conversely, incompatibility means there is no joint distribution having exactly those specified conditionals, but this fact alone does not quantify bias in a particular analysis or determine how useful the resulting imputations are. Assessing an FCS procedure also requires attention to its conditional models, the data, and the intended estimand.
A common mistake is to reason, “Each variable has a probability model, and the algorithm draws from each model in turn, so the procedure must be Gibbs sampling.” The correction is that Gibbs sampling requires more than separate conditional distributions and random draws: the conditionals must all be full conditionals of one common joint target. A second mistake is to infer compatibility from stable-looking updates. Stability is not a derivation of a joint distribution, and it cannot replace a compatibility argument.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- What additional fact is required before a sequence of stochastic conditional updates can be called Gibbs sampling?
- In the incompatible binary example, what were the two conditional-odds ratios, and why did their disagreement matter?
- Does incompatibility alone prove that an FCS procedure produces biased estimates? Explain.
- An analyst uses a logistic model for one binary variable and a linear model for a continuous variable, then cycles through draws. What can be concluded from this description, and what cannot?
- Why is a chain that appears stable not enough to establish that its updates are Gibbs updates?
Solutions. (1) The conditional distributions must be the full conditionals of one common joint target distribution. (2) The ratios were \(16\) and \(6\); a positive joint distribution would require them to agree, so the proposed conditionals cannot all come from such a joint distribution. (3) No. Incompatibility rules out that particular common-joint interpretation, but alone it does not quantify bias or determine performance for a specific estimand. (4) The procedure is an FCS specification with stochastic conditional updates. The description does not establish that the conditionals are compatible or that the procedure is Gibbs sampling. (5) Apparent stability is not proof that a joint distribution exists with the specified conditionals; a compatibility argument is a separate requirement.