An Incompatible Pair of Gaussian Conditionals

Why Two Valid Conditional Models May Not Fit Together

In fully conditional specification, each incomplete variable can be updated from a model conditional on the current values of the other variables. A chosen conditional model may be a perfectly valid probability distribution on its own, yet the collection of conditional models need not come from one joint distribution. This tutorial examines that issue for a pair of linear Gaussian conditionals: it identifies the constraints required for a shared bivariate normal distribution and shows what goes wrong when those constraints fail.

The distinction is important for interpretation. If both conditionals are compatible with a bivariate normal distribution, they can be understood as updates from that common model. If they are incompatible, the two conditional specifications do not define that shared Gaussian model, even though each update can still be calculated. This is a mathematical check on the models’ relationship, not a test of whether either regression is scientifically appropriate.

Set-Up and Compatibility Conditions

Let \(X\) and \(Y\) be continuous variables. Consider the conditional models \(X\mid Y=y\sim N(a+by,\tau_X^2)\) and \(Y\mid X=x\sim N(c+dx,\tau_Y^2)\), where \(a,c\) are intercepts, \(b,d\) are slopes, and \(\tau_X^2,\tau_Y^2\) are positive, constant conditional variances. Here \(N(m,v)\) denotes a normal distribution with mean \(m\) and variance \(v\). Each model is assumed to describe the full conditional distribution, not merely a regression for its conditional mean.

For a shared bivariate normal distribution, the slope in each conditional is a covariance divided by the variance of the conditioning variable. The conditional variances are the corresponding residual variances. Those facts imply a constraint linking each slope to its conditional variance. They also imply that the joint distribution must have a positive-definite covariance matrix, equivalently a positive-definite precision matrix (the inverse covariance matrix).

Why the Constraints Hold

The first constraint can be seen directly from the conditional densities. If a shared, positive joint density \(f(x,y)\) exists, then the mixed derivatives of its log density, first differentiating with respect to \(x\) and then \(y\), must agree regardless of which conditional factorization is used. From the \(X\mid Y\) model, the terms involving \(x\) and \(y\) in the log density are \(-(x-a-by)^2/(2\tau_X^2)\); their mixed derivative is \(b/\tau_X^2\). From the \(Y\mid X\) model, the corresponding mixed derivative is \(d/\tau_Y^2\). Therefore, they must be equal.

The second condition concerns whether the implied joint density can be normalized. Under the equality of ratios, the quadratic terms in the log joint density have precision matrix \(\begin{pmatrix}1/\tau_X^2&-b/\tau_X^2\\-d/\tau_Y^2&1/\tau_Y^2\end{pmatrix}\). Its diagonal entries are positive. Its determinant is \((1-bd)/(\tau_X^2\tau_Y^2)\), so the matrix is positive definite exactly when \(bd<1\). If the determinant is zero, the quadratic form does not confine the density in every direction; if it is negative, the quadratic form is not positive definite. Neither case yields a proper, nondegenerate bivariate normal distribution.

Finally, the intercepts determine the means by substituting \(\mu_Y\) and \(\mu_X\) into the two conditional-mean equations: \(\mu_X=a+b\mu_Y\) and \(\mu_Y=c+d\mu_X\). Solving these two linear equations gives the stated formulas because \(1-bd>0\). This also shows why intercepts cannot repair a mismatch in the slope-to-variance ratios: they change the location of the distribution, not its cross-derivative constraint.

A Second Failure: The Implied Joint Model Is Not Proper

Passing the slope-to-variance equality is necessary but not sufficient. The product of the slopes must also be less than 1. This condition is sometimes overlooked because both conditional models can still be ordinary, well-defined normal distributions when the product is too large.

What a Compatible Pair Looks Like

A worked compatible case helps confirm that the constraints are sufficient, not just necessary. When both checks pass, the intercepts determine the means, and the precision matrix determines a proper joint normal distribution. The implied covariance matrix can then be found by inverting that precision matrix.

Using the Check in Fully Conditional Specification

For a pair of continuous variables modeled with linear Gaussian conditionals, a practical check is to write down each conditional residual variance as well as each slope. Compare the two slope-to-variance ratios, then check the slope product. Checking slopes alone is not enough: different residual variances change the required relationship. Checking the ratio equality alone is also not enough: the product condition rules out an improper implied joint distribution.

These checks diagnose compatibility under the specified model form. In practice, estimated slopes and variances have sampling uncertainty, so plugging estimates into an exact equality is not a formal statistical test of compatibility. A visible discrepancy may motivate examining the model choices, scales, or estimates, but compatibility of the mathematical specifications and uncertainty in estimated parameters are distinct questions.

A joint-model approach specifies one multivariate distribution and derives its conditionals, so compatibility is built into the construction. Fully conditional specification instead specifies each conditional model directly, which can offer modeling flexibility but leaves compatibility to be checked. If the two Gaussian conditionals fail the checks above, they cannot both be conditional distributions of the same bivariate normal model; in particular, the pair cannot be treated as Gibbs updates from that common Gaussian joint model.

The check has limits. It applies to the stated pair of linear conditional means and constant positive Gaussian conditional variances; it does not establish compatibility for other model forms or settle global compatibility in settings with more variables. It also does not show that either conditional is correctly specified for the data. Finally, passing the checks establishes the existence of a compatible bivariate normal distribution, not that the distribution is substantively plausible or that a particular MICE analysis is otherwise adequate.

Common Mistake

A common mistake is to reason, “Both regression models have normal errors, so together they define a bivariate normal distribution.” Normal errors make each conditional model a valid distribution, but they do not force the two models to agree about the joint dependence. For example, the first worked example has individually valid Gaussian conditionals but different slope-to-variance ratios. The correction is to check both constraints: equality of the ratios and \(bd<1\). Neither condition can be omitted.