Conditional Independence in Gaussian Block Systems

Why Conditional Independence Needs Its Own Covariance Check

For jointly Gaussian random vectors, zero cross-covariance is equivalent to independence. Conditional independence asks a different question: after the value of a third vector \(Z\) is known, do the remaining blocks \(X\) and \(Y\) separate? The relevant covariance is then not the marginal cross-covariance \(\operatorname{Cov}(X,Y)\), but the cross-covariance left after accounting for \(Z\).

This distinction matters in both directions. Two blocks that are dependent before conditioning can become conditionally independent once their shared Gaussian variation is accounted for. Conversely, blocks that are marginally independent can become dependent after conditioning. For a jointly Gaussian system, both situations can be assessed with a block covariance calculation.

Definition: Random vectors \(X\) and \(Y\) are conditionally independent given \(Z\), written \(X\perp Y\mid Z\), if their conditional joint distribution factors into the product of their conditional marginal distributions for almost every value of \(Z\). Equivalently, for all Borel sets \(A\) and \(B\), \(P(X\in A,Y\in B\mid Z)=P(X\in A\mid Z)P(Y\in B\mid Z)\) almost surely.

Let \(X\in\mathbb{R}^p\), \(Y\in\mathbb{R}^q\), and \(Z\in\mathbb{R}^r\) be jointly Gaussian. Write their covariance matrix in the block order \((X,Y,Z)\). First suppose that \(\Sigma_{ZZ}=\operatorname{Var}(Z)\) is invertible. The conditional cross-covariance of \(X\) and \(Y\) given \(Z\) is

\Sigma_{XY\cdot Z} = \Sigma_{XY} - \Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}.

The subtracted term is the cross-covariance accounted for by the linear prediction from \(Z\). The remaining matrix compares the parts of \(X\) and \(Y\) that are not explained by \(Z\). In general distributions, a zero conditional cross-covariance need not imply conditional independence. Joint Gaussianity is what makes the covariance criterion decisive.

The Gaussian Conditional Independence Criterion

Theorem: Suppose \((X,Y,Z)\) is jointly Gaussian and \(\Sigma_{ZZ}\) is invertible. Then \(X\perp Y\mid Z\) if and only if \(\Sigma_{XY\cdot Z}=0\), where \(\Sigma_{XY\cdot Z}=\Sigma_{XY}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}\).

The conditional normal formulas give conditional means linear in \(Z\) and conditional covariance blocks that do not depend on the observed value of \(Z\). In particular, the conditional cross-covariance is \(\Sigma_{XY\cdot Z}\). The proof below also explains why the criterion is an independence result rather than merely a covariance calculation.

Remove the linear effects of \(Z\).
Define residual vectors \(U=X-E[X]-\Sigma_{XZ}\Sigma_{ZZ}^{-1}(Z-E[Z])\) and \(V=Y-E[Y]-\Sigma_{YZ}\Sigma_{ZZ}^{-1}(Z-E[Z])\).

Check their covariances.
Both residuals have zero covariance with \(Z\), and direct expansion gives \(\operatorname{Cov}(U,V)=\Sigma_{XY\cdot Z}\).

Use joint Gaussianity.
The residuals and \(Z\) are jointly Gaussian. Thus \(U\) and \(V\) are independent exactly when their cross-covariance is zero; the joint residual vector \((U,V)\) is independent of \(Z\).

Translate back to conditional distributions.
Given \(Z\), the conditional laws of \(X\) and \(Y\) are translated versions of the laws of \(U\) and \(V\). Their conditional joint law factors exactly when \(U\) and \(V\) are independent.

Here are the covariance details behind those steps. From the definitions, \(\operatorname{Cov}(U,Z)=\Sigma_{XZ}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZZ}=0\), and similarly \(\operatorname{Cov}(V,Z)=0\). Expanding the residual cross-covariance gives

\begin{aligned} \operatorname{Cov}(U,V) &=\Sigma_{XY} -\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY} -\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}\\ &\quad+\Sigma_{XZ}\Sigma_{ZZ}^{-1} \Sigma_{ZZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}\\ &=\Sigma_{XY}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY} =\Sigma_{XY\cdot Z}. \end{aligned}

Since \((U,V,Z)\) is jointly Gaussian, zero covariance between \((U,V)\) and \(Z\) implies their independence. Also, for jointly Gaussian \(U,V\), zero cross-covariance implies independence. If \(\Sigma_{XY\cdot Z}=0\), then \(U\) and \(V\) are independent and their joint residual vector is independent of \(Z\); consequently, the conditional joint law of \(X,Y\) factors given \(Z\). In the other direction, conditional independence implies zero conditional cross-covariance. The conditional covariance is the constant matrix \(\Sigma_{XY\cdot Z}\), so that matrix must be zero. This proves both directions. \(\square\)

Worked Covariance Calculations

Worked Example: Shared Variation Produces Conditional Independence

Let \(Z\sim N(0,1)\), \(\varepsilon_X\sim N(0,1)\), and \(\varepsilon_Y\sim N(0,4)\) be mutually independent. Set \(X=Z+\varepsilon_X\) and \(Y=2Z+\varepsilon_Y\). Determine whether \(X\) and \(Y\) are conditionally independent given \(Z\).

The variances and covariances are \(\operatorname{Var}(X)=1+1=2\), \(\operatorname{Var}(Y)=4+4=8\), \(\operatorname{Cov}(X,Y)=2\), \(\operatorname{Cov}(X,Z)=1\), and \(\operatorname{Cov}(Z,Y)=2\). Thus, in the order \((X,Y,Z)\), the covariance matrix is

\begin{pmatrix} 2&2&1\\ 2&8&2\\ 1&2&1 \end{pmatrix}.

The conditional cross-covariance is \(\Sigma_{XY\cdot Z}=2-(1)(1)^{-1}(2)=0\). Therefore \(X\perp Y\mid Z\). Indeed, given \(Z=z\), \(X=z+\varepsilon_X\) and \(Y=2z+\varepsilon_Y\), and the independent errors give conditional distributions \(N(z,1)\) and \(N(2z,4)\). Marginally, the covariance is \(2\), so the blocks are not independent before conditioning.

Worked Example: Conditioning Can Create Dependence

Let \(X\) and \(Y\) be independent \(N(0,1)\) variables and set \(Z=X+Y\). Are \(X\) and \(Y\) conditionally independent given \(Z\)?

Here \(\operatorname{Cov}(X,Y)=0\), \(\operatorname{Var}(Z)=2\), and \(\operatorname{Cov}(X,Z)=\operatorname{Cov}(Y,Z)=1\). Applying the conditional cross-covariance formula gives

\Sigma_{XY\cdot Z} =0-(1)(2^{-1})(1)=-\frac12.

The conditional cross-covariance is not zero, so \(X\) and \(Y\) are not conditionally independent given \(Z\). The full conditional covariance matrix is \(\begin{pmatrix}1/2&-1/2\\-1/2&1/2\end{pmatrix}\), and each conditional mean is \(Z/2\). The negative covariance reflects the constraint \(X+Y=Z\): for a fixed sum, a larger value of one variable requires a smaller value of the other. The conditional distribution is degenerate along that fixed-sum constraint, but the covariance criterion still applies.

Worked Example: Checking a Vector Cross-Covariance

Let \(Z\sim N(0,1)\). Let \(E=(E_1,E_2)^\mathsf{T}\) and \(F=(F_1,F_2)^\mathsf{T}\) be jointly Gaussian and independent of \(Z\), with \(\operatorname{Var}(E)=\operatorname{Var}(F)=I_2\) and \(\operatorname{Cov}(E,F)=\begin{pmatrix}0.4&0\\0&-0.2\end{pmatrix}\). This residual covariance is valid: its two coordinate pairs have correlations \(0.4\) and \(-0.2\), both strictly between \(-1\) and \(1\). Define \(X=(Z+E_1,\,2Z+E_2)^\mathsf{T}\) and \(Y=(Z+F_1,\,-Z+F_2)^\mathsf{T}\). Test conditional independence given \(Z\).

The loading vectors on \(Z\) are \(a=(1,2)^\mathsf{T}\) and \(b=(1,-1)^\mathsf{T}\). Hence \(\Sigma_{XZ}=a\), \(\Sigma_{ZY}=b^\mathsf{T}\), and the shared-\(Z\) contribution to \(\Sigma_{XY}\) is

\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY} = \begin{pmatrix}1\\2\end{pmatrix} (1)^{-1} \begin{pmatrix}1&-1\end{pmatrix} = \begin{pmatrix}1&-1\\2&-2\end{pmatrix}.

The residual cross-covariance adds \(\begin{pmatrix}0.4&0\\0&-0.2\end{pmatrix}\), so \(\Sigma_{XY}=\begin{pmatrix}1.4&-1\\2&-2.2\end{pmatrix}\). Subtracting the shared-\(Z\) contribution yields \(\Sigma_{XY\cdot Z}=\begin{pmatrix}0.4&0\\0&-0.2\end{pmatrix}\), which is not the zero matrix. Thus the vector blocks are not conditionally independent. For blocks, every entry of the conditional cross-covariance matrix must be zero; a zero in only some positions is insufficient.

When the Conditioning Block Is Singular

The criterion also works when \(\Sigma_{ZZ}\) is singular, with the inverse replaced by the Moore–Penrose pseudoinverse \(\Sigma_{ZZ}^{+}\). The conditioning vector then lies almost surely in a lower-dimensional affine subspace, so conditional formulas are understood on that support.

Theorem: For jointly Gaussian \(X,Y,Z\), whether or not \(\Sigma_{ZZ}\) is invertible, \(X\perp Y\mid Z\) if and only if \(\Sigma_{XY}-\Sigma_{XZ}\Sigma_{ZZ}^{+}\Sigma_{ZY}=0\).

To verify the formula, write \(K=\Sigma_{ZZ}\) and define \(U=X-E[X]-\Sigma_{XZ}K^{+}(Z-E[Z])\), with \(V\) defined in the same way using \(Y\). Positive semidefiniteness of the joint covariance implies that \(\Sigma_{XZ}v=0\) whenever \(v\) is in the null space of \(K\): indeed, \(v^\mathsf{T}(Z-E[Z])\) has variance zero and is therefore zero almost surely. Thus \(\Sigma_{XZ}K^{+}K=\Sigma_{XZ}\), and \(\operatorname{Cov}(U,Z)=0\); likewise, \(\operatorname{Cov}(V,Z)=0\). Joint Gaussianity makes \((U,V)\) independent of \(Z\). Expanding their cross-covariance gives

\operatorname{Cov}(U,V) = \Sigma_{XY}-\Sigma_{XZ}K^{+}\Sigma_{ZY}.

The residuals are Gaussian, so they are independent exactly when this cross-covariance is zero. Because the conditional means are the corresponding affine functions of \(Z\), that residual independence is equivalent to conditional independence. This establishes the singular case as well.

Common Mistake and Practical Use

A common mistake is to check only whether \(\operatorname{Cov}(X,Y)=0\) and then carry that conclusion over to conditioning. That is the wrong calculation: conditional independence concerns the covariance remaining after the contribution of \(Z\) has been removed. In the sum example, the marginal cross-covariance is zero, but the conditional cross-covariance is \(-1/2\). The correction is to compute the full Schur-complement expression, using a pseudoinverse if the conditioning covariance is singular.

This criterion is useful because it turns a conditional-distribution question into a finite matrix calculation. It also makes clear what the Gaussian assumption contributes: it guarantees that zero residual cross-covariance is enough for independence. Outside the jointly Gaussian setting, the same calculation can identify zero conditional covariance, but that alone does not establish conditional independence.

Key takeaway: For jointly Gaussian blocks, conditional independence given \(Z\) is equivalent to a zero conditional cross-covariance. Compute \(\Sigma_{XY}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}\) when \(\Sigma_{ZZ}\) is invertible, and use \(\Sigma_{ZZ}^{+}\) when it is singular.

Check Your Understanding

Use the conditional cross-covariance criterion to answer these questions.

For jointly Gaussian \(X,Y,Z\) with invertible \(\Sigma_{ZZ}\), write the matrix whose being zero is equivalent to \(X\perp Y\mid Z\).
In the model \(X=Z+\varepsilon_X\), \(Y=2Z+\varepsilon_Y\), what feature of the errors makes \(X\) and \(Y\) conditionally independent given \(Z\)?
If \(\operatorname{Cov}(X,Y)=0\), does that alone imply \(X\perp Y\mid Z\) for jointly Gaussian variables? Explain.
What changes in the criterion when \(\Sigma_{ZZ}\) is singular?
For vector blocks, must every entry of the conditional cross-covariance matrix be zero, or is it enough for its diagonal entries to be zero?