Conditional Independence in Gaussian Vectors

Conditional Independence in a Gaussian Vector

For a Gaussian vector, dependence between two groups of coordinates can change when we condition on a third group. The covariance between the first two groups before conditioning is therefore not, by itself, the right test for conditional independence. The relevant quantity is their conditional cross-covariance. When the full covariance matrix is invertible, the same test has a useful expression in terms of the precision matrix, the inverse covariance matrix.

Work on a probability space $(\Omega,\mathcal{F},\mathbb{P})$. Let $X=(X_A^T,X_B^T,X_C^T)^T$ be a jointly Gaussian vector, where $A,B,C$ are disjoint coordinate sets. For the conditional covariance calculation, assume that $\Sigma_{CC}=\operatorname{Cov}(X_C,X_C)$ is invertible. When $C$ is empty, the formulas below mean ordinary, unconditional covariance and independence. Write $\mu_D=\mathbb{E}[X_D]$ and $\Sigma_{DE}=\operatorname{Cov}(X_D,X_E)$ for coordinate sets $D,E$.

Definition: The subvectors $X_A$ and $X_B$ are conditionally independent given $X_C$, written $X_A\perp X_B\mid X_C$, if their conditional joint distribution given $X_C$ is the product of their conditional marginal distributions, almost surely. Equivalently, for almost every value of $X_C$, the conditional law of $(X_A,X_B)$ factors into the conditional laws of $X_A$ and $X_B$.

For jointly Gaussian vectors, the conditional distribution given $X_C$ is Gaussian. Its mean and covariance can be written as

\mathbb{E}[X_D\mid X_C] =\mu_D+\Sigma_{DC}\Sigma_{CC}^{-1}(X_C-\mu_C), \qquad \operatorname{Cov}(X_D\mid X_C) =\Sigma_{DD}-\Sigma_{DC}\Sigma_{CC}^{-1}\Sigma_{CD}.

Here the conditional covariance is a fixed matrix, not a random function of the observed value of $X_C$. For the combined vector $(X_A,X_B)$, the off-diagonal block of this matrix is

\Sigma_{AB\cdot C} =\Sigma_{AB}-\Sigma_{AC}\Sigma_{CC}^{-1}\Sigma_{CB}.

This is the conditional cross-covariance. The formula also covers the case $C$ is empty by taking $\Sigma_{AB\cdot C}=\Sigma_{AB}$.

Why Zero Conditional Cross-Covariance Is the Test

Theorem: Suppose $X=(X_A^T,X_B^T,X_C^T)^T$ is jointly Gaussian and $\Sigma_{CC}$ is invertible when $C$ is nonempty. Then $$ X_A\perp X_B\mid X_C \quad\Longleftrightarrow\quad \Sigma_{AB\cdot C}=0. $$ The result allows the conditional Gaussian distribution to be singular; the required condition is that the stated conditional covariance exists.

Proof. Let $D=A\cup B$ and form the residual vector $R_D=X_D-\mu_D-\Sigma_{DC}\Sigma_{CC}^{-1}(X_C-\mu_C)$. Its covariance with $X_C$ is $\Sigma_{DC}-\Sigma_{DC}\Sigma_{CC}^{-1}\Sigma_{CC}=0$. The pair $(R_D,X_C)$ is jointly Gaussian, since it is an affine transformation of a Gaussian vector. For jointly Gaussian vectors, zero cross-covariance implies independence: their joint characteristic function has exponent given by the covariance quadratic form, and when the cross-covariance is zero that exponent separates into the sum of the two marginal exponents. Thus $R_D$ is independent of $X_C$, and the conditional law of $X_D$ given $X_C$ is Gaussian with the mean and covariance displayed above.

In that conditional Gaussian law, write the covariance in blocks for $A$ and $B$. If $\Sigma_{AB\cdot C}=0$, the covariance is block diagonal. The same characteristic-function factorization shows that the conditional joint law is the product of its conditional marginals, for almost every $X_C$; hence $X_A\perp X_B\mid X_C$. Conversely, conditional independence gives a product conditional law. Its conditional cross-covariance is therefore zero almost surely, because the conditional centered coordinates from the two independent factors have product expectation zero. The conditional covariance is the fixed matrix $\Sigma_{AB\cdot C}$, so that matrix must be zero. This proves both directions. $\square$

The characteristic-function fact used here also applies when a Gaussian covariance matrix is singular. A Gaussian vector with block-diagonal covariance has a characteristic function that factors, so its two blocks are independent even if either block is supported on a lower-dimensional set.

Worked Example: Independent After Conditioning, Dependent Before

Let $(X,Y,Z)$ be a centered Gaussian vector with covariance matrix $\Sigma=\begin{bmatrix}2&\tfrac12&1\\ \tfrac12&2&1\\1&1&2\end{bmatrix}$. Condition on $Z$ and decide whether $X$ and $Y$ are conditionally independent.

The conditional means are $\mathbb{E}[X\mid Z]=\tfrac12 Z$ and $\mathbb{E}[Y\mid Z]=\tfrac12 Z$, since $\operatorname{Cov}(X,Z)=1$, $\operatorname{Cov}(Y,Z)=1$, and $\operatorname{Var}(Z)=2$. The conditional variances are $\operatorname{Var}(X\mid Z)=2-1\cdot\tfrac12\cdot1=\tfrac32$ and $\operatorname{Var}(Y\mid Z)=2-1\cdot\tfrac12\cdot1=\tfrac32$. The conditional cross-covariance is $\operatorname{Cov}(X,Y\mid Z)=\tfrac12-1\cdot\tfrac12\cdot1=0$.

Therefore $X\perp Y\mid Z$. This covariance matrix is valid and nonsingular: the residual vector $(X-\tfrac12Z,Y-\tfrac12Z)$ has covariance $\begin{bmatrix}\tfrac32&0\\0&\tfrac32\end{bmatrix}$, is uncorrelated with $Z$, and $\operatorname{Var}(Z)=2>0$. These facts give a positive-definite covariance for $(X,Y,Z)$.

Check: The marginal covariance is $\operatorname{Cov}(X,Y)=\tfrac12$, which is not zero, while the conditional covariance computed from the formula is $\tfrac12-\tfrac12=0$. The conclusion concerns independence given $Z$, not marginal independence.

Zeros in the Precision Matrix

Suppose now that the entire covariance matrix $\Sigma$ is positive definite, and define the precision matrix $K=\Sigma^{-1}$. Positive definiteness ensures that $K$ exists and is positive definite. For a coordinate set $D$, $K_{DD}$ denotes the principal submatrix with rows and columns indexed by $D$.

Theorem: Let $X$ be a jointly Gaussian vector with positive-definite covariance matrix $\Sigma$, and let $A,B,C$ partition all its coordinates into disjoint sets. Then $$ X_A\perp X_B\mid X_C \quad\Longleftrightarrow\quad K_{AB}=0,\qquad K=\Sigma^{-1}. $$ Here $K_{AB}=0$ means every entry in that block is zero. If $C$ is empty, this says that $X_A$ and $X_B$ are independent exactly when their precision cross-block is zero.

Proof. Put $D=A\cup B$. Partition $K$ into its $D$ and $C$ blocks. The Gaussian density, as a function of $x_D$ with $x_C$ fixed, has a quadratic term $x_D^T K_{DD}x_D$. Completing the square shows that its conditional covariance is $K_{DD}^{-1}$: the linear terms affect the conditional mean, while the quadratic coefficient is $K_{DD}$. This inverse exists because a principal submatrix of a positive-definite matrix is positive definite. Equivalently, the block inverse identity gives $K_{DD}^{-1}=\Sigma_{DD}-\Sigma_{DC}\Sigma_{CC}^{-1}\Sigma_{CD}$, the conditional covariance. Thus the conditional cross-covariance block is the $A,B$ block of $K_{DD}^{-1}$.

Write $M=K_{DD}$, partitioned into $A,B$ blocks. If $M_{AB}=0$, symmetry gives $M_{BA}=0$, so $M$ is block diagonal and its inverse is block diagonal; its $A,B$ block is zero. Conversely, if the $A,B$ block of $M^{-1}$ is zero, symmetry makes $M^{-1}$ block diagonal. The inverse of an invertible block-diagonal matrix is block diagonal, so $M$ is block diagonal and $M_{AB}=0$. The preceding theorem identifies zero conditional cross-covariance with conditional independence, proving the equivalence. $\square$

This statement uses the precision block between $A$ and $B$ when $C$ consists of all remaining coordinates. If only some variables are conditioned on, while other coordinates are left unconditioned, use the conditional covariance formula directly; a zero in the corresponding full precision block need not characterize that different conditioning question.

Worked Example: Reading Conditional Independence from a Precision Matrix

Consider a centered Gaussian vector $X=(X_1,X_2,X_3)^T$ with precision matrix $K=\begin{bmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{bmatrix}$. Verify that $X_1$ and $X_3$ are conditionally independent given $X_2$.

The leading principal minors of $K$ are $2$, $2\cdot2-(-1)(-1)=3$, and $2(4-1)-(-1)(-2)=4$; all are positive, so $K$ is positive definite. Its inverse is $\Sigma=\tfrac14\begin{bmatrix}3&2&1\\2&4&2\\1&2&3\end{bmatrix}$. To check the inverse, the first row of $K\Sigma$ is $\tfrac14(6-2,\,4-4,\,2-2)=(1,0,0)$; the second is $\tfrac14(-3+4-1,\,-2+8-2,\,-1+4-3)=(0,1,0)$; and the third is $\tfrac14(-2+2,\,-4+4,\,-2+6)=(0,0,1)$.

From $\Sigma$, the needed entries are $\operatorname{Cov}(X_1,X_3)=\tfrac14$, $\operatorname{Cov}(X_1,X_2)=\tfrac12$, $\operatorname{Cov}(X_2,X_3)=\tfrac12$, and $\operatorname{Var}(X_2)=1$. Thus $\operatorname{Cov}(X_1,X_3\mid X_2)=\tfrac14-\tfrac12\cdot1^{-1}\cdot\tfrac12=0$. The conditional independence theorem gives $X_1\perp X_3\mid X_2$. The precision check gives the same conclusion immediately: $K_{13}=0$.

Check: The conditional variances are $\operatorname{Var}(X_1\mid X_2)=\tfrac34-(\tfrac12)^2=\tfrac12$ and $\operatorname{Var}(X_3\mid X_2)=\tfrac34-(\tfrac12)^2=\tfrac12$. The conditional covariance matrix is therefore diagonal, with entries $\tfrac12,\tfrac12$, consistent with the zero cross-covariance.

What the Criterion Does—and Does Not—Say

Conditional independence is stronger than a zero covariance statement for general, non-Gaussian variables. The Gaussian assumption matters: it is what allows zero cross-covariance to imply that the conditional joint distribution factors. For Gaussian vectors, the test is exact, including when a conditional covariance is singular.

A common mistake is to infer that $X$ and $Y$ are not conditionally independent given $Z$ just because their marginal covariance is nonzero. In the first example, that incorrect calculation would be “$\operatorname{Cov}(X,Y)=\tfrac12\ne0$, so $X$ and $Y$ are not independent given $Z$.” It uses the wrong covariance. The correction is to calculate $\operatorname{Cov}(X,Y\mid Z)=\operatorname{Cov}(X,Y)-\operatorname{Cov}(X,Z)\operatorname{Var}(Z)^{-1}\operatorname{Cov}(Z,Y)=\tfrac12-\tfrac12=0$. For this Gaussian vector, that zero establishes conditional independence.

For scalar coordinates in a positive-definite Gaussian vector, $K_{ij}=0$ means that $X_i$ and $X_j$ are conditionally independent given every other coordinate. The condition is about the inverse covariance, not the covariance itself. In a multivariate block, the entire block $K_{AB}$ must be zero; a single zero entry does not establish independence of the two subvectors.

Worked Example: Nonzero Conditional Covariance Means Dependence

Let $(X,Y,Z)$ be centered Gaussian with covariance matrix $\Sigma=\begin{bmatrix}2&1&1\\1&2&1\\1&1&2\end{bmatrix}$. Find the conditional cross-covariance of $X,Y$ given $Z$, and determine whether they are conditionally independent.

The relevant entries are $\operatorname{Cov}(X,Y)=1$, $\operatorname{Cov}(X,Z)=1$, $\operatorname{Cov}(Y,Z)=1$, and $\operatorname{Var}(Z)=2$. Hence $\operatorname{Cov}(X,Y\mid Z)=1-1\cdot\tfrac12\cdot1=\tfrac12$, which is nonzero. The conditional independence criterion therefore says $X\not\perp Y\mid Z$. The covariance matrix is positive definite: it equals $I+J$, where $I$ is the $3$ by $3$ identity and $J$ is the all-ones matrix, and for every nonzero vector $v$, $v^T(I+J)v=v^Tv+(\sum_i v_i)^2>0$.

Check: Its precision matrix is $I-\tfrac14J$, since $J^2=3J$ and $(I+J)(I-\tfrac14J)=I+(1-\tfrac14-\tfrac34)J=I$. Thus $K_{12}=-\tfrac14\ne0$, agreeing with the conclusion from conditional covariance.

Check Your Understanding

Use the conditional covariance formula first, and use the precision criterion only when its full-partition assumptions apply.

For a jointly Gaussian vector, what matrix must vanish for $X_A\perp X_B\mid X_C$?
If $\operatorname{Cov}(X,Y)=0$ but $\operatorname{Cov}(X,Z)=\operatorname{Cov}(Y,Z)=1$ and $\operatorname{Var}(Z)=2$, what is $\operatorname{Cov}(X,Y\mid Z)$?
In the precision-matrix theorem, why must $C$ contain all coordinates outside $A\cup B$?
For a positive-definite Gaussian vector, what conditional independence does $K_{ij}=0$ express when conditioning on all other coordinates?
Why does the zero-cross-covariance criterion require the Gaussian assumption?

Conditional Independence in Gaussian Vectors

What You'll Learn