Conditioning Gaussian Vectors by Block Matrix Algebra

Conditioning One Gaussian Block on Another

For a jointly Gaussian vector, conditioning on one block changes the mean of the other block according to their cross-covariance. The conditional covariance is a Schur complement: it measures the uncertainty that remains after the part linearly associated with the observed block is removed. Block matrix algebra gives both formulas, and a residual argument explains why they describe the conditional distribution, not just its first two moments.

Let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space, and let \(X\) and \(Y\) be jointly Gaussian random vectors with values in \(\mathbb{R}^p\) and \(\mathbb{R}^q\). Write their means and joint covariance in block form as

Here \(A=\operatorname{Cov}(X)\), \(B=\operatorname{Cov}(Y)\), and \(C=\operatorname{Cov}(X,Y)\). Suppose \(B\) is positive definite, so \(B^{-1}\) exists. In particular, \(Y\) has a nonsingular Gaussian distribution. Conditioning on a specified value \(Y=y\) is understood through a regular conditional distribution; since a single value of a continuously distributed \(Y\) usually has probability zero, it is not ordinary conditioning on a positive-probability event.

The expression \(CB^{-1}(y-\mu_Y)\) adjusts the mean using the observed deviation of \(Y\) from its mean. The covariance subtracts the part of \(X\)'s covariance accounted for by linear association with \(Y\). The proof makes this interpretation precise.

Proof. Define the residual vector \(Z=X-\mu_X-CB^{-1}(Y-\mu_Y)\). It is an affine transformation of the jointly Gaussian vector \((X,Y)\), so \((Z,Y)\) is jointly Gaussian. Its cross-covariance is \[ \operatorname{Cov}(Z,Y) =C-CB^{-1}B=0, \] and its covariance is \[ \operatorname{Cov}(Z) =A-CB^{-1}C^T-CB^{-1}C^T+CB^{-1}BB^{-1}C^T =A-CB^{-1}C^T. \] For jointly Gaussian vectors, zero cross-covariance implies independence: their joint characteristic function has the form \(\exp(i t^T m-\tfrac12 t^T\Sigma t)\); when the cross-covariance blocks are zero, this expression factors into the product of the two marginal characteristic functions. Thus \(Z\) and \(Y\) are independent. Since \(X=\mu_X+CB^{-1}(Y-\mu_Y)+Z\), conditioning on \(Y=y\) leaves the independent residual \(Z\) with its original Gaussian law. This gives the stated conditional mean and covariance. The argument also applies when the covariance of \(Z\) is singular, since a singular Gaussian vector still has the stated characteristic function. \(\square\)

The Schur Complement and Covariance Validity

The matrix \(S=A-CB^{-1}C^T\) is called the Schur complement of \(B\) in the joint covariance matrix. Its role can be seen from an exact factorization. This also verifies that the formula gives a valid covariance matrix, including when the original joint Gaussian vector is singular.

Proof. Because \(B\) is symmetric, the rightmost factor is the transpose of the leftmost factor. Multiplying the right two matrices first gives \[ \begin{bmatrix}S&0\\0&B\end{bmatrix} \begin{bmatrix}I&0\\B^{-1}C^T&I\end{bmatrix} = \begin{bmatrix}S&0\\B^{-1}C^T&B\end{bmatrix}. \] Multiplying on the left by the first matrix then yields top-left \(S+CB^{-1}C^T=A\), top-right \(C\), bottom-left \(C^T\), and bottom-right \(B\), as required. To check positive semidefiniteness, write the factorization as \(L D L^T\), where \(L\) is the left factor and \(D\) is the block diagonal middle factor. The matrix \(L\) is invertible, with inverse \(\begin{bmatrix}I&-CB^{-1}\\0&I\end{bmatrix}\). For any vector \(v\), \(v^TLDL^Tv=(L^Tv)^TD(L^Tv)\). Since \(L^T\) maps vectors bijectively to vectors, this quadratic form is nonnegative for every \(v\) exactly when \(D\) is positive semidefinite. The block diagonal matrix \(D\) is positive semidefinite exactly when both \(S\) and \(B\) are; \(B\) is already positive definite. Hence the joint matrix is positive semidefinite exactly when \(S\) is. \(\square\)

This factorization also explains why the theorem allows a singular conditional covariance: positive semidefiniteness requires \(S\) to have no negative eigenvalues, but does not require it to be positive definite. If \(B\) is singular, the inverse in the formulas does not exist. Conditioning then requires handling the support constraints of the observed block; the inverse formulas above do not apply as written.

Worked Calculations

Interpretation and a Common Mistake

The conditional mean is affine in the observed value \(y\), while the conditional covariance is constant across observed values. If \(C=0\), then the formula gives the original mean and covariance for \(X\); the residual proof also shows that \(X\) and \(Y\) are independent in this jointly Gaussian setting. If \(C\) is nonzero, the observation shifts the conditional mean, and the Schur complement quantifies the remaining uncertainty.

Common mistake: forgetting to center the observed block. In the scalar example above, using \(2+\tfrac34(1)=\tfrac{11}4\) mistakenly uses \(y\) instead of \(y-\mu_Y\). The correct calculation is \(2+\tfrac34(1-(-1))=2+\tfrac32=\tfrac72\). Centering is essential whenever the conditioning block has nonzero mean. Another useful check is matrix dimensions: \(C\) is \(p\) by \(q\), so \(CB^{-1}C^T\) is \(p\) by \(p\), the same dimension as \(A\).