Schur Complements and Multiple Correlation

Why a Schur Complement Appears in Multiple Correlation

A covariance matrix for a response and several predictors records both their separate variances and their joint relationships. Its block structure contains a useful answer to a prediction question: after choosing the best linear combination of the predictors, how much response variance remains? The answer is a scalar Schur complement. It is also the residual variance of the best linear predictor, so it gives a direct route from covariance blocks to R-squared.

This is a statement about linear prediction. It does not, in general, say that the remaining variance is the conditional variance of the response given the predictors. Those quantities coincide under additional assumptions, such as joint normality, but the linear-residual identity below requires only finite second moments and a nonsingular predictor covariance matrix.

Covariance Blocks and the Schur Complement

Let \(X=(X_1,\ldots,X_p)'\) be a random vector of predictors and let \(Y\) be a scalar response. Suppose all components have finite second moments, and write \(A=\operatorname{Cov}(X,X)\), \(c=\operatorname{Cov}(X,Y)\), and \(s=\operatorname{Var}(Y)>0\). Assume \(A\) is positive definite. The joint covariance matrix then has the block form below.

To see why this quantity is tied to prediction, consider an arbitrary linear predictor with coefficient vector \(a\in\mathbb{R}^p\): \(E[Y]+a'(X-E[X])\). Its residual is \(Y-E[Y]-a'(X-E[X])\). Centering makes the constant term explicit, while leaving the covariance structure unchanged.

Deriving the Linear Residual Variance

The theorem follows by expanding the residual variance and completing the quadratic form. For any \(a\), bilinearity of covariance gives \(\operatorname{Var}(Y-a'X)=s-2a'c+a'Aa\); centering the variables does not change this variance. Set \(b=A^{-1}c\), so \(Ab=c\). Then

Indeed, expanding the last term gives \(a'Aa-2a'Ab+b'Ab\). Since \(Ab=c\) and \(b'Ab=c'A^{-1}c\), this expression plus \(s-c'A^{-1}c\) is exactly \(s-2a'c+a'Aa\). Positive definiteness of \(A\) implies \((a-b)'A(a-b)\geq 0\), with equality only when \(a=b\). Therefore the minimum is attained uniquely at \(b\), and its value is \(s-c'A^{-1}c\). This proves the result. \(\square\)

The same argument establishes that the Schur complement is nonnegative: it is the variance of the residual at \(a=b\). It is zero precisely when the centered response equals \(b'(X-E[X])\) almost surely, since a random variable with variance zero is almost surely constant and this residual has mean zero. Thus a zero Schur complement means exact linear prediction, not merely a strong association.

From Residual Variance to R-Squared

The variance removed by the optimal linear predictor is \(c'A^{-1}c\), and the variance remaining is \(s-c'A^{-1}c\). The proportion of response variance explained by the linear predictor is therefore \((c'A^{-1}c)/s\). This is the squared multiple correlation, commonly denoted \(R^2\); the multiple correlation \(R\) is its nonnegative square root.

The second equality follows directly from the residual-variance formula: \(\sigma_{\mathrm{res}}^2=s-c'A^{-1}c\). Divide by \(s>0\) and rearrange to obtain \(c'A^{-1}c/s=1-\sigma_{\mathrm{res}}^2/s\). The same expression agrees with the maximum squared correlation over nonzero linear combinations \(a'X\). In fact, when \(a'Aa>0\),

For the inequality, apply Cauchy–Schwarz to \(A^{1/2}a\) and \(A^{-1/2}c\): \((a'c)^2\leq(a'Aa)(c'A^{-1}c)\). If \(c\ne 0\), equality is attained by taking \(a=A^{-1}c\); if \(c=0\), every such squared correlation is zero. Thus the covariance-block expression is also the largest squared correlation with a linear combination of the predictors.

To verify this identity, multiply \(\Sigma\) on the left by the block lower-triangular matrix \(\begin{pmatrix}I&0\\-c'A^{-1}&1\end{pmatrix}\), whose determinant is one. The product is \(\begin{pmatrix}A&c\\0&s-c'A^{-1}c\end{pmatrix}\). Its determinant is the product of the diagonal-block determinants, namely \(\det(A)(s-c'A^{-1}c)\). Since the left multiplier has determinant one, this is also \(\det(\Sigma)\). \(\square\)

This factorization is a matrix identity; the prediction interpretation comes from the residual-variance proof. In particular, if the joint covariance matrix is singular while \(A\) remains positive definite, the Schur complement is zero and the response is an exact linear combination of the predictors almost surely.

Worked Examples

Interpretation and a Common Pitfall

The Schur complement separates response variation into a part captured by the best linear predictor and a remaining linear residual part. Because \(A\) is positive definite and the joint covariance matrix is a covariance matrix, the residual variance is nonnegative. It follows that \(0\leq R^2\leq 1\). A value of zero means \(c=0\), so no linear combination of the predictors has nonzero covariance with \(Y\); a value of one means the response is exactly linearly determined by the predictors almost surely.

• Pitfall: treating the residual variance as conditional variance without assumptions. The Schur complement always gives the variance of the best linear residual under the stated second-moment conditions. It does not generally give \(\operatorname{Var}(Y\mid X)\), which may depend on the realized predictor values. Joint normality is one setting in which the conditional variance is constant and equals this Schur complement.
• Pitfall: reversing the covariance vector. With \(c=\operatorname{Cov}(X,Y)\) as a column vector, the formula is \(s-c'A^{-1}c\). Omitting one transpose or using the wrong covariance block can produce a dimensionally invalid expression.
• Pitfall: interpreting \(R^2\) as a causal or nonlinear measure. Here \(R^2\) describes the fraction of variance captured by linear prediction from \(X\). It does not by itself establish causation or summarize every possible nonlinear relationship.

The practical value of the identity is that one can read the linear prediction problem directly from covariance information. It provides a variance interpretation for the Schur complement, a matrix check through the determinant factorization, and a precise meaning for multiple \(R^2\): the complement of the fraction of response variance left in the optimal linear residual.