How Conditioning Reduces Gaussian Covariance

Why Compare Conditional and Marginal Covariance?

Covariance describes the spread of a random vector across repeated draws. Conditioning on an observed vector adds information, so for jointly Gaussian variables it cannot increase variance in any linear direction. The precise statement is a matrix inequality: the conditional covariance is no larger than the marginal covariance in positive-semidefinite order. This is not an entry-by-entry comparison of matrix elements; it compares the variance of every linear combination.

The amount of reduction is controlled by the cross-covariance between the vector being predicted and the vector observed. If that cross-covariance is zero, conditioning does not change covariance. If it is nonzero, at least one direction has a strictly reduced variance, though other directions can remain unchanged. The distinction matters in multivariate work, where a nonzero update need not reduce uncertainty in every direction.

Setup and Conditional Covariance

Let \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\) be jointly Gaussian, allowing either vector or their joint distribution to be degenerate. Write their means and covariance blocks as

where \(A=\operatorname{Cov}(X)\), \(B=\operatorname{Cov}(Y)\), and \(C=\operatorname{Cov}(X,Y)\). The full block matrix is positive semidefinite because it is a covariance matrix. For a symmetric matrix, positive semidefinite means that every quadratic form is nonnegative.

If \(B\) is nonsingular, the Gaussian conditional-normal formula gives

This covariance does not depend on the particular value \(y\). If \(B\) is singular, use the Moore–Penrose inverse \(B^+\), obtained by replacing each nonzero eigenvalue of \(B\) by its reciprocal and leaving zero eigenvalues at zero. The singular-block conditional-normal formula is

for \(y\) in the support of \(Y\), up to the usual almost-sure qualification for conditional distributions. The joint covariance condition ensures that \(C^T u\) lies in the range of \(B\) for every \(u\in\mathbb{R}^p\); thus the formula is compatible with directions of \(Y\) having zero variance. These conditional-normal formulas are the block formulas for Gaussian conditioning, including the singular-block case.

The Covariance-Reduction Theorem

The nonsingular case is included by taking \(B^+=B^{-1}\). The result compares covariance matrices in Loewner order, not their individual entries.

Proof. The Moore–Penrose inverse \(B^+\) is positive semidefinite: in an orthonormal eigenbasis of \(B\), its eigenvalues are nonnegative. Therefore, for every \(v\in\mathbb{R}^p\),

Thus \(CB^+C^T\) is positive semidefinite. Subtracting the conditional covariance formula from \(A\) proves \(\operatorname{Cov}(X\mid Y)\preceq A\).

For the equality claims, \(C^Tv\) belongs to the range of \(B\). On that range, \(B^+\) has strictly positive eigenvalues. Consequently,

This proves the direction-specific equality statement. The whole reduction matrix is zero exactly when its quadratic form is zero for every \(v\), which, by the equivalence just shown, occurs exactly when \(C^Tv=0\) for every \(v\), or \(C=0\).

Finally, the reduction is positive definite exactly when \(v^T CB^+C^Tv>0\) for every nonzero \(v\). By the same equivalence, this holds exactly when \(C^Tv\ne0\) for every nonzero \(v\). That is the condition that the linear map \(v\mapsto C^Tv\) has trivial kernel, equivalently that \(C\) has full row rank \(p\). \(\square\)

Worked Examples

What “Reduction” Does and Does Not Mean

The theorem guarantees a reduction in every directional variance: for each fixed \(v\), the conditional variance of \(v^TX\) is no larger than its marginal variance. It does not say that every off-diagonal covariance entry becomes smaller in numerical value. Such entries can change in either direction; positive-semidefinite order is a comparison of quadratic forms, not a comparison entry by entry.

The rank of \(C\) describes how many independent directions in \(X\) are affected. If \(C\ne0\), some direction has a strict reduction. If \(C\) does not have full row rank, there are nonzero directions \(v\) with \(C^Tv=0\), and those directions retain their marginal variance. If \(C\) has full row rank, every nonzero direction has a strict variance reduction.

This distinction is useful when interpreting Gaussian prediction and measurement models. An observation can substantially reduce uncertainty overall while leaving some contrasts or linear combinations unchanged. Conversely, a nonzero update to a covariance matrix does not by itself establish strict reduction in all directions.

Common Pitfall

A tempting but incorrect argument is: “Conditioning reduces uncertainty, so every entry of the conditional covariance matrix must be less than the corresponding marginal entry.” The theorem does not support this conclusion. For example, a covariance entry is not itself a directional variance; it can move upward even when the whole covariance reduction matrix is positive semidefinite.

The correction is to form the difference \(A-\operatorname{Cov}(X\mid Y)\) and test its quadratic forms. Here the difference is \(CB^+C^T\), and \(v^TCB^+C^Tv\geq0\) for every \(v\). That is the exact mathematical meaning of “conditioning reduces Gaussian covariance.”