Residual Covariance from Block Matrix Identities

Why Residual Covariance Has a Block-Matrix Form

When several target variables are adjusted for the same controls, the residuals are still a vector: each observation has one residual for each target. Their covariance matrix summarizes both the remaining variation in each target and the remaining association between targets. The covariance blocks of the original variables determine this matrix directly, without fitting a separate scalar regression for every pair.

The key operation is a Schur complement. Here it arises from a simple calculation: subtract from the target vector its best linear prediction from the controls, then calculate the covariance of what remains. The resulting formula also explains why the residual covariance matrix cannot have a negative variance in any linear combination of the residuals.

Setup and Definition

Let \(X\) be a \(p\)-dimensional random vector of target variables and \(Z\) a \(q\)-dimensional random vector of controls. Assume their joint distribution has finite second moments. For clarity, take both vectors to be centered, so \(E[X]=0\) and \(E[Z]=0\). Centering is equivalent to including an intercept when forming linear predictions.

Write the joint covariance matrix in blocks:

Assume \(\Sigma_{ZZ}\) is positive definite, hence invertible. This ensures that the linear projection coefficients on the controls are uniquely determined. The coefficient matrix for predicting \(X\) linearly from \(Z\) is \(\Sigma_{XZ}\Sigma_{ZZ}^{-1}\).

This is a population definition. It describes the residuals from the best linear prediction in mean-square error, not necessarily the residuals from a correctly specified conditional-mean model. The distinction matters when the joint distribution is not Gaussian or when the conditional mean is nonlinear.

Deriving the Residual Covariance

Set \(B=\Sigma_{XZ}\Sigma_{ZZ}^{-1}\), so \(R_X=X-BZ\). Covariance is bilinear, and the covariance blocks give \(\operatorname{Cov}(X,Z)=\Sigma_{XZ}\) and \(\operatorname{Cov}(Z,X)=\Sigma_{ZX}\). Expanding the covariance therefore gives:

To prove the formula, use \(R_X=X-BZ\) and expand each term:

Since \(\Sigma_{ZZ}\) is symmetric, its inverse is symmetric. Thus \(B^{\mathsf T}=\Sigma_{ZZ}^{-1}\Sigma_{ZX}\). Substituting \(B=\Sigma_{XZ}\Sigma_{ZZ}^{-1}\), the second and third terms are each \(\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZX}\). The final term is also that same matrix:

Consequently, one copy is subtracted twice and added once, leaving \(\operatorname{Cov}(R_X)=\Sigma_{XX}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZX}\). This proves the result. \(\square\)

The same coefficient matrix makes the residual uncorrelated with the controls. Indeed, \(\operatorname{Cov}(R_X,Z)=\Sigma_{XZ}-B\Sigma_{ZZ}=0\). This is the normal-equation property of linear projection. It does not, by itself, imply that \(R_X\) and \(Z\) are independent.

The Schur Complement and a Positivity Check

For a block matrix with invertible lower-right block, the Schur complement of \(\Sigma_{ZZ}\) in \(\Sigma\) is, by definition, \(\Sigma_{XX}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZX}\). The derivation above shows that this block-matrix expression is exactly the covariance of the residual vector.

A block factorization gives a second useful result. It shows why the Schur complement must be positive semidefinite when it comes from a covariance matrix.

To verify both claims, let \(S=\Sigma_{XX}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZX}\) and form the block matrix \(L=\begin{pmatrix}I_p&-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\\0&I_q\end{pmatrix}\). Direct block multiplication gives:

The matrix \(\Sigma\) is a covariance matrix, so it is positive semidefinite: for every vector \(v\), \(v^{\mathsf T}\Sigma v\) is the variance of a linear combination of the components and is nonnegative. For any vector \(a\in\mathbb{R}^p\), apply positive semidefiniteness of \(L\Sigma L^{\mathsf T}\) to the vector \((a^{\mathsf T},0^{\mathsf T})^{\mathsf T}\). The resulting quadratic form is \(a^{\mathsf T}Sa\), so \(a^{\mathsf T}Sa\geq 0\) for every \(a\). Hence \(S\) is positive semidefinite.

For the determinant identity, \(L\) is block triangular with identity matrices on its diagonal, so \(\det(L)=1\). Therefore \(\det(L\Sigma L^{\mathsf T})=\det(\Sigma)\). The block-diagonal form has determinant \(\det(S)\det(\Sigma_{ZZ})\), proving the stated identity. \(\square\)

Worked Covariance Calculations

Interpretation and Common Pitfall

The term \(\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZX}\) is the covariance contribution removed by linear adjustment. It depends on the covariance among the controls through \(\Sigma_{ZZ}^{-1}\), as well as on the target-control covariances. For multiple targets, the adjustment changes both the diagonal residual variances and the off-diagonal residual covariances.

A common mistake is to call the Schur complement the conditional covariance matrix in every distribution. The formula always gives the covariance of the linear residuals under the stated second-moment conditions. It equals the conditional covariance \(\operatorname{Cov}(X\mid Z)\), which does not depend on the realized value of \(Z\), in the jointly Gaussian case. Outside that setting, conditional covariance can depend on \(Z\), and the conditional mean can be nonlinear; the residual covariance formula alone does not describe either feature.

Another practical warning is that the inverse in the formula is not available when \(\Sigma_{ZZ}\) is singular. This can happen when controls are redundant or linearly dependent. One should not silently use an ordinary inverse in that case: the projection can instead be formulated on the span of the controls, using a suitable generalized inverse, but that requires handling the rank-deficient case explicitly.