Why Compare Conditional and Marginal Covariance?
Conditioning on observed information can make a prediction more precise, but “more precise” has an exact multivariate meaning. For a vector of quantities of interest, its conditional covariance should be compared with its marginal covariance in the positive-semidefinite order. This comparison tells us not only whether individual variances decrease, but also how uncertainty changes for every linear combination of the quantities.
For jointly Gaussian vectors, the conditional covariance does not depend on the observed value. The difference between the marginal and conditional covariance is itself positive semidefinite, and it gives a precise measure of the information supplied by the observation. The results below also cover linearly dependent observations, for which the observation covariance is singular.
Conditional Covariance and Its Reduction
Let \(X\) be a \(p\)-dimensional Gaussian random vector and \(Y\) a \(q\)-dimensional Gaussian random vector, jointly Gaussian; either vector may be degenerate. Write their means and covariance blocks as
Thus the joint covariance is the block matrix with blocks \(A,B,B^\mathsf{T},C\). When \(C\) is singular, \(C^+\) denotes its Moore–Penrose inverse: if \(C=U\operatorname{diag}(\lambda_1,\ldots,\lambda_q)U^\mathsf{T}\) is an orthogonal eigenvalue decomposition, \(C^+\) has the same eigenvectors, replaces each positive \(\lambda_i\) by \(1/\lambda_i\), and leaves each zero eigenvalue equal to zero.
Here \(\operatorname{Var}(X\mid Y)\) means the conditional covariance matrix, whose \((i,j)\) entry is the conditional covariance of \(X_i\) and \(X_j\) given \(Y\). The assertion that it is constant means that for almost every observed value \(y\), the conditional distribution has the same covariance matrix. The conditional mean can still depend on \(y\).
Proof by Gaussian Residualization
Set \(Y_0=Y-\mu_Y\), and form the residual
First we verify the covariance identities needed for this residual. If \(v\) is in the null space of \(C\), then \(\operatorname{Var}(v^\mathsf{T}Y_0)=v^\mathsf{T}Cv=0\), so \(v^\mathsf{T}Y_0=0\) almost surely. It follows that \(\operatorname{Cov}(X,v^\mathsf{T}Y_0)=Bv=0\). Therefore every column of \(B^\mathsf{T}\) is in the range of \(C\), and
The defining eigenvalue properties of \(C^+\) also give \(C^+CC^+=C^+\). We can now calculate the residual’s covariance with the observation:
The pair \((R,Y)\) is jointly Gaussian because it is a linear transformation of the jointly Gaussian pair \((X,Y)\). For jointly Gaussian vectors, zero cross-covariance implies independence, including when a vector is degenerate. Thus \(R\) is independent of \(Y\). Its covariance is
where the last equality uses \(C^+CC^+=C^+\). Rearranging the definition of \(R\) gives \(X=\mu_X+BC^+Y_0+R\). Given \(Y\), the first two terms are fixed and the independent residual \(R\) retains its unconditional covariance. Hence \(\operatorname{Var}(X\mid Y)=\operatorname{Var}(R)=A-BC^+B^\mathsf{T}\).
Finally, \(BC^+B^\mathsf{T}\) is positive semidefinite: for every \(u\),
because \(C^+\) is positive semidefinite. Subtracting this matrix from \(A\) proves the claimed covariance inequality. \(\square\)
What the Reduction Means for a Prediction
Take any fixed vector \(a\in\mathbb{R}^p\), and consider the scalar quantity \(T=a^\mathsf{T}X\). Its marginal variance and conditional variance are the corresponding quadratic forms in the covariance matrices. Substituting the theorem’s formula gives the exact reduction.
To verify the variance identity on the second line, the conditional mean obtained from the residual representation is \(E[X\mid Y]=\mu_X+BC^+Y_0\). Therefore \(E[T\mid Y]=a^\mathsf{T}\mu_X+a^\mathsf{T}BC^+Y_0\), whose variance is \(a^\mathsf{T}BC^+B^\mathsf{T}a\), using \(C^+CC^+=C^+\) and \(B=BC^+C\). The reduction is thus the variance of the part of the prediction explained by the observation, not merely a numerical difference without interpretation.
Worked Example: A Scalar Prediction from One Observation
Suppose \(X\) and \(Y\) are jointly Gaussian with zero means, \(\operatorname{Var}(X)=9\), \(\operatorname{Var}(Y)=4\), and \(\operatorname{Cov}(X,Y)=3\). Find the conditional variance of \(X\) given \(Y\), and the amount by which observing \(Y\) reduces the variance.
Here \(A=9\), \(C=4\), and \(B=3\), so \(C^{-1}=1/4\). The conditional variance is
The prior variance is \(9=36/4\), so the reduction is \(36/4-27/4=9/4\). This agrees with the variance of the conditional mean: \(E[X\mid Y]=(3/4)Y\), and \(\operatorname{Var}((3/4)Y)=(9/16)4=9/4\). The specified joint covariance is valid because its determinant is \(9(4)-3^2=27>0\), with positive diagonal entries.
Uncertainty in Linear Contrasts
A linear contrast is any scalar \(a^\mathsf{T}X\), where the entries of \(a\) specify the combination of components being studied. For several contrasts collected as the rows of a matrix \(L\), apply the same result to \(LX\). Its conditional covariance and the matrix reduction are
The matrix inequality means that every linear combination of the contrasts has no greater conditional variance than marginal variance. A particular contrast can gain little information even when other directions in \(X\) gain much more. The relevant quantity is its covariance with the observations, \(B^\mathsf{T}a\), rather than the overall size of the covariance reduction matrix.
Worked Example: Reduction for a Contrast of Two Components
Let \(X=(X_1,X_2)^\mathsf{T}\) and \(Y\) be jointly Gaussian, with
Find the conditional covariance of \(X\), then calculate the variance reduction for the contrast \(T=X_1-X_2\). The conditional covariance is
This is positive definite: its diagonal entries are positive and its determinant is \((16\cdot36-1)/25=23>0\). For \(a=(1,-1)^\mathsf{T}\), the marginal contrast variance is
The contrast’s covariance with \(Y\) is \(a^\mathsf{T}B=2-3=-1\), so the variance reduction is \((a^\mathsf{T}B)^2/C=1/5\). Consequently,
Direct calculation from the conditional covariance gives \(16/5+36/5-2(-1/5)=54/5\), as required. The joint covariance is valid: the conditional covariance just computed is positive definite, and \(C=5>0\).
Singular Observations and Redundant Information
An observation vector can contain exact linear duplicates. In that case, \(C^{-1}\) does not exist, but the Moore–Penrose inverse in the theorem still gives the reduction. The formula depends on the information in the observation, not on how many coordinates are used to encode it.
Worked Example: Two Redundant Gaussian Measurements
Let \(Y_1\sim N(0,1)\), let \(R\sim N(0,1)\) be independent of \(Y_1\), and define \(X=2Y_1+R\) and \(Y=(Y_1,2Y_1)^\mathsf{T}\). Find the conditional variance of \(X\) given \(Y\), including the calculation with the singular covariance matrix.
Since \(X=2Y_1+R\), its variance is \(4+1=5\). The observation covariance and the cross-covariance are
The two entries of \(Y\) are linearly dependent, so \(C\) has rank one. Writing \(C=uu^\mathsf{T}\) for \(u=(1,2)^\mathsf{T}\), its Moore–Penrose inverse is \(C^+=uu^\mathsf{T}/(u^\mathsf{T}u)^2=C/25\). Thus
The conditional variance is therefore \(5-4=1\). This matches the representation: \(Y\) determines \(Y_1\), so it determines \(2Y_1\), while the independent residual \(R\) remains unobserved. The second coordinate of \(Y\) adds no information beyond the first.
Common Mistake: Comparing Covariance Entries Separately
A frequent error is to conclude that conditioning reduces covariance because every diagonal entry decreases, or to assume that each off-diagonal covariance must decrease in absolute value. Neither claim is what the theorem says. The guaranteed statement is that the whole matrix difference \(A-\operatorname{Var}(X\mid Y)\) is positive semidefinite. Individual off-diagonal entries can move in either direction, and checking the diagonal entries alone does not establish the matrix inequality.
The correct test is to consider every contrast. For each vector \(a\), the theorem gives \(a^\mathsf{T}\operatorname{Var}(X\mid Y)a \leq a^\mathsf{T}Aa\). This also explains why a contrast can have zero reduction even if some components of \(X\) are strongly associated with \(Y\): its own covariance \(a^\mathsf{T}B\) with the observation may be zero.
Check Your Understanding
Use the covariance-order result and the contrast formula to answer these questions.
- What does \(M\preceq N\) mean in terms of the variances of all linear combinations?
- For a contrast \(a^\mathsf{T}X\), express its variance reduction using \(a\), \(B\), and \(C^+\).
- Why can the theorem use \(C^+\) when the observation covariance is singular?
- Under what condition does a scalar Gaussian contrast have zero variance reduction after conditioning on \(Y\)?
- Why does a reduction in the full covariance matrix not imply that every off-diagonal covariance decreases in absolute value?