Worked Gaussian Conditioning Problems With Three Covariance Blocks

Three Gaussian Blocks: The Conditioning Question

A common multivariate problem has three jointly Gaussian blocks: two blocks of interest, \(X\) and \(Y\), and an observed block, \(Z\). The goal is to find the distribution of \((X,Y)\) after observing \(Z\), then determine whether \(X\) and \(Y\) are conditionally independent. The key calculation is the covariance left over after the linear information in \(Z\) has been removed.

The blocks can be scalar or vector-valued, and their dimensions need not agree. Write their means as \(\mu_X,\mu_Y,\mu_Z\), and let \(\Sigma_{UV}=\operatorname{Cov}(U,V)\). Thus \(\Sigma_{UV}\) has one row per component of \(U\) and one column per component of \(V\); in particular, \(\Sigma_{VU}=\Sigma_{UV}^{\mathsf T}\). Assume throughout the main formulas that \(\Sigma_{ZZ}\) is positive definite. The blocks \(X\) and \(Y\) need not be independent marginally, and their conditional covariance need not be zero.

Definition: For jointly Gaussian blocks \(X,Y,Z\) with positive definite \(\Sigma_{ZZ}\), the conditional residual covariance blocks are the blocks of the covariance matrix of \((X,Y)\) after conditioning on \(Z\). In particular,

\(\Sigma_{XX\mid Z}=\Sigma_{XX}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZX}\),
\(\Sigma_{YY\mid Z}=\Sigma_{YY}-\Sigma_{YZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}\),
\(\Sigma_{XY\mid Z}=\Sigma_{XY}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}\).

The phrase “residual covariance” is useful because each formula subtracts the covariance explained by predicting the relevant block linearly from \(Z\). The order of the factors matters: for example, \(\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}\) is a matrix with the same dimensions as \(\Sigma_{XY}\). Reversing the order generally gives an undefined product or a different matrix.

Deriving the Conditional Distribution

The Gaussian conditioning formula gives both the conditional mean and the full conditional covariance. Its structure is easiest to see by subtracting from each target block its best linear predictor based on \(Z\).

Theorem: Let \((X,Y,Z)\) be jointly Gaussian with \(\Sigma_{ZZ}\) positive definite. For each \(z\), the conditional distribution of \((X,Y)\) given \(Z=z\) is Gaussian, with mean blocks

\(E[X\mid Z=z]=\mu_X+\Sigma_{XZ}\Sigma_{ZZ}^{-1}(z-\mu_Z)\),
\(E[Y\mid Z=z]=\mu_Y+\Sigma_{YZ}\Sigma_{ZZ}^{-1}(z-\mu_Z)\),

and covariance

\(\begin{pmatrix}\Sigma_{XX\mid Z}&\Sigma_{XY\mid Z}\\\Sigma_{YX\mid Z}&\Sigma_{YY\mid Z}\end{pmatrix}\),

where \(\Sigma_{YX\mid Z}=\Sigma_{XY\mid Z}^{\mathsf T}\). This covariance does not depend on the observed value \(z\).

Proof. Center the variables temporarily, and define the residual blocks \(R_X=X-\Sigma_{XZ}\Sigma_{ZZ}^{-1}Z\) and \(R_Y=Y-\Sigma_{YZ}\Sigma_{ZZ}^{-1}Z\). These are linear transformations of a jointly Gaussian vector, so \((R_X,R_Y,Z)\) is jointly Gaussian. Direct covariance calculation gives

\operatorname{Cov}(R_X,Z) =\Sigma_{XZ}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZZ}=0, \qquad \operatorname{Cov}(R_Y,Z)=0.

The residual pair \((R_X,R_Y)\) is therefore independent of \(Z\), since jointly Gaussian blocks with zero cross-covariance are independent. Its covariance blocks follow by expanding the residual covariances. For instance,

\begin{aligned} \operatorname{Cov}(R_X,R_Y) &=\operatorname{Cov}\bigl(X-\Sigma_{XZ}\Sigma_{ZZ}^{-1}Z,\, Y-\Sigma_{YZ}\Sigma_{ZZ}^{-1}Z\bigr)\\ &=\Sigma_{XY}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}. \end{aligned}

The same expansion gives \(\operatorname{Cov}(R_X)=\Sigma_{XX\mid Z}\) and \(\operatorname{Cov}(R_Y)=\Sigma_{YY\mid Z}\). Given \(Z=z\), add the now-fixed predictor \(\mu_X+\Sigma_{XZ}\Sigma_{ZZ}^{-1}(z-\mu_Z)\) back to \(R_X\), and likewise for \(Y\). This yields the stated conditional means and covariance. \(\square\)

Conditional Independence From the Cross Block

Once the conditional covariance has been calculated, the independence question is a matrix question about its off-diagonal block. This criterion is specific to the jointly Gaussian setting: zero covariance alone does not establish independence for arbitrary distributions.

Theorem: Under the conditions above, \(X\) and \(Y\) are conditionally independent given \(Z\) if and only if

\(\Sigma_{XY\mid Z}=\Sigma_{XY}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}=0\).

The zero on the right is a zero matrix, not merely a zero scalar summary of the block.

Proof. The residual pair \((R_X,R_Y)\) in the preceding derivation is jointly Gaussian and independent of \(Z\). Its cross-covariance is exactly \(\Sigma_{XY\mid Z}\). If this matrix is zero, the Gaussian residual blocks \(R_X\) and \(R_Y\) are independent. Since the predictors given \(Z\) are fixed shifts, \(X\) and \(Y\) are independent conditional on \(Z\). Conversely, if \(X\) and \(Y\) are conditionally independent given \(Z\), their conditional cross-covariance is zero. The conditional Gaussian formula shows that this covariance is \(\Sigma_{XY\mid Z}\), so the matrix must be zero. \(\square\)

This also explains how to read a computed conditional covariance matrix. Its diagonal blocks describe the remaining uncertainty in each target block; the off-diagonal block describes dependence between the blocks after conditioning. The entire conditional covariance must be positive semidefinite, but positive diagonal blocks by themselves do not imply conditional independence.

Worked Scalar Problems

Worked Example: Independent Residuals After Conditioning

Let \(Z\sim N(0,1)\), and let \(\varepsilon_X\sim N(0,4)\) and \(\varepsilon_Y\sim N(0,3)\) be mutually independent and independent of \(Z\). Set \(X=2Z+\varepsilon_X\) and \(Y=3Z+\varepsilon_Y\). Find the conditional distribution of \((X,Y)\) given \(Z=z\), and decide whether the blocks are conditionally independent.

First calculate the marginal covariance blocks from the model: \(\operatorname{Var}(X)=2^2(1)+4=8\), \(\operatorname{Var}(Y)=3^2(1)+3=12\), \(\operatorname{Cov}(X,Y)=2\cdot3\cdot1=6\), \(\operatorname{Cov}(X,Z)=2\), and \(\operatorname{Cov}(Y,Z)=3\).

Since \(\operatorname{Var}(Z)=1\), the conditional means are \(E[X\mid Z=z]=2z\) and \(E[Y\mid Z=z]=3z\). The conditional variances are \(\operatorname{Var}(X\mid Z)=8-2(1)^{-1}2=4\) and \(\operatorname{Var}(Y\mid Z)=12-3(1)^{-1}3=3\). In the second calculation, the contribution removed is \(3^2\operatorname{Var}(Z)=9\), so the residual variance is \(12-9=3\).

The conditional cross-covariance is \(6-2(1)^{-1}3=0\). Hence \((X,Y)\mid Z=z\sim N\bigl((2z,3z)^{\mathsf T},\begin{pmatrix}4&0\\0&3\end{pmatrix}\bigr)\), and \(X\) and \(Y\) are conditionally independent given \(Z\). For example, at \(z=1.5\), their conditional means are \(3\) and \(4.5\), with the same conditional variances \(4\) and \(3\).

Worked Example: Marginal Association That Remains After Conditioning

Let \(Z\sim N(0,1)\). Independently of \(Z\), let the jointly Gaussian errors \((\varepsilon_X,\varepsilon_Y)\) have mean zero and covariance \(\begin{pmatrix}2&1\\1&3\end{pmatrix}\). This is a valid positive definite covariance matrix because its leading diagonal entry is \(2>0\) and its determinant is \(2\cdot3-1^2=5>0\). Set \(X=Z+\varepsilon_X\) and \(Y=2Z+\varepsilon_Y\).

The marginal variances are \(\operatorname{Var}(X)=1+2=3\) and \(\operatorname{Var}(Y)=2^2+3=7\). Their covariance is \(1\cdot2\cdot\operatorname{Var}(Z)+\operatorname{Cov}(\varepsilon_X,\varepsilon_Y)=2+1=3\). Also, \(\operatorname{Cov}(X,Z)=1\) and \(\operatorname{Cov}(Y,Z)=2\).

The conditional means are \(z\) and \(2z\). The conditional variances are \(3-1^2=2\) and \(7-2^2=3\), while the conditional cross-covariance is \(3-(1)(2)=1\). Thus \((X,Y)\mid Z=z\sim N\bigl((z,2z)^{\mathsf T},\begin{pmatrix}2&1\\1&3\end{pmatrix}\bigr)\). The nonzero conditional cross-covariance shows that \(X\) and \(Y\) are not conditionally independent. Conditioning removes their shared contribution through \(Z\), but the correlated errors still associate them.

A Vector Example With Three Covariance Blocks

Worked Example: Computing a Conditional Cross-Covariance Matrix

Let \(Z\sim N(0,2I_2)\), and define \(H_X=\begin{pmatrix}1&1\\0&1\end{pmatrix}\) and \(H_Y=\begin{pmatrix}1&0\\1&1\end{pmatrix}\). Let \((\varepsilon_X,\varepsilon_Y)\) be jointly Gaussian, independent of \(Z\), with covariance blocks \(\operatorname{Var}(\varepsilon_X)=2I_2\), \(\operatorname{Var}(\varepsilon_Y)=3I_2\), and \(\operatorname{Cov}(\varepsilon_X,\varepsilon_Y)=I_2\). This joint error covariance is positive definite: for vectors \(u,v\), its quadratic form is \(2\|u\|^2+2u^{\mathsf T}v+3\|v\|^2\), which is positive unless both vectors are zero. Set \(X=H_XZ+\varepsilon_X\) and \(Y=H_YZ+\varepsilon_Y\).

The marginal covariance of \(Z\) is \(2I_2\). Multiplying the loading matrices gives \(H_XH_X^{\mathsf T}=\begin{pmatrix}2&1\\1&1\end{pmatrix}\), \(H_YH_Y^{\mathsf T}=\begin{pmatrix}1&1\\1&2\end{pmatrix}\), and \(H_XH_Y^{\mathsf T}=\begin{pmatrix}1&2\\0&1\end{pmatrix}\). Therefore the three covariance blocks for \(X,Y\) and their covariances with \(Z\) are

\begin{aligned} \Sigma_{XX}&=2H_XH_X^{\mathsf T}+2I_2 =\begin{pmatrix}6&2\\2&4\end{pmatrix},& \Sigma_{YY}&=2H_YH_Y^{\mathsf T}+3I_2 =\begin{pmatrix}5&2\\2&7\end{pmatrix},\\ \Sigma_{XY}&=2H_XH_Y^{\mathsf T}+I_2 =\begin{pmatrix}3&4\\0&3\end{pmatrix},& \Sigma_{XZ}&=2H_X=\begin{pmatrix}2&2\\0&2\end{pmatrix},\\ \Sigma_{YZ}&=2H_Y=\begin{pmatrix}2&0\\2&2\end{pmatrix},& \Sigma_{ZZ}&=2I_2. \end{aligned}

Because \(\Sigma_{ZZ}^{-1}=\tfrac12 I_2\), the conditional cross-covariance is

\begin{aligned} \Sigma_{XY\mid Z} &=\begin{pmatrix}3&4\\0&3\end{pmatrix} -\begin{pmatrix}2&2\\0&2\end{pmatrix} \left(\tfrac12 I_2\right) \begin{pmatrix}2&2\\0&2\end{pmatrix}\\ &=\begin{pmatrix}3&4\\0&3\end{pmatrix} -\begin{pmatrix}2&4\\0&2\end{pmatrix} =I_2. \end{aligned}

The conditional diagonal blocks are \(\Sigma_{XX\mid Z}=2I_2\) and \(\Sigma_{YY\mid Z}=3I_2\), since the loading contributions \(2H_XH_X^{\mathsf T}\) and \(2H_YH_Y^{\mathsf T}\) are removed by conditioning. Thus the conditional covariance of \((X,Y)\) is \(\begin{pmatrix}2I_2&I_2\\I_2&3I_2\end{pmatrix}\). Its nonzero off-diagonal block means \(X\) and \(Y\) are not conditionally independent. For a particular observation \(Z=z\), the conditional means are \(H_Xz\) and \(H_Yz\); the covariance just computed is the same for every \(z\).

Common Pitfall: Testing the Wrong Covariance

A zero marginal covariance block \(\Sigma_{XY}\) is not the criterion for conditional independence, and a nonzero marginal block does not rule it out. The relevant quantity is always \(\Sigma_{XY}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}\). In the first scalar example, the marginal covariance is \(6\), but the conditional cross-covariance is zero because all of that association is explained by \(Z\). In the second example, the marginal covariance is \(3\), and subtracting the contribution \(2\) leaves a conditional covariance of \(1\).

Another frequent error is to subtract the square of a covariance without including the variance of the conditioning variable. In general, the scalar adjustment is \(\operatorname{Cov}(X,Z)\operatorname{Var}(Z)^{-1}\operatorname{Cov}(Z,Y)\). Only when \(\operatorname{Var}(Z)=1\) does this reduce to the product of the two covariances. In vector problems, keep the full matrix product and check its dimensions before simplifying.

Key takeaway: For three jointly Gaussian blocks, condition by subtracting the covariance explained through \(Z\). The conditional cross-covariance block—not the marginal cross-covariance—decides whether the two target blocks are conditionally independent.

Check Your Understanding

Use the block formulas and explain each covariance adjustment.

In the first scalar example, why is the conditional variance of \(Y\) equal to \(12-9\), rather than \(12-3^2\operatorname{Var}(Y)\)?
Suppose scalar jointly Gaussian \(X,Y,Z\) have \(\operatorname{Var}(Z)=4\), \(\operatorname{Cov}(X,Z)=2\), \(\operatorname{Cov}(Y,Z)=6\), and \(\operatorname{Cov}(X,Y)=5\). Calculate \(\operatorname{Cov}(X,Y\mid Z)\).
For vector blocks, what are the dimensions of \(\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY}\) relative to \(\Sigma_{XY}\)?
Can \(X\) and \(Y\) be conditionally independent given \(Z\) even when their marginal covariance is nonzero? Explain using the conditional cross-block formula.
Why does a zero conditional cross-covariance establish conditional independence here, but not for arbitrary random variables?