Multiple Correlation from Covariance Blocks

From Covariance Blocks to Multiple Correlation

A multiple correlation summarizes how strongly one response variable is linearly associated with a collection of predictors. The covariance matrix of the response and predictors contains all the second-moment information needed for this summary. The calculation is not generally a sum of squared pairwise correlations: the covariance among the predictors also matters.

This tutorial develops the population calculation from those covariance blocks. The central formula uses the inverse predictor covariance matrix to account for dependence among predictors. Its assumptions are important: the response must have positive variance, and the predictor covariance matrix must be invertible for the formula in that form.

Definition: Let $Y$ be a scalar random variable with $0<\operatorname{Var}(Y)=\sigma_Y^2<\infty$, and let $X=(X_1,\ldots,X_p)^T$ be a random vector with finite second moments. Write $\Sigma_X=\operatorname{Cov}(X,X)$ for the $p\times p$ predictor covariance matrix and $c=\operatorname{Cov}(X,Y)$ for the $p$-vector of response–predictor covariances. If $\Sigma_X$ is positive definite, the squared multiple correlation of $Y$ with $X$ is $$ R^2_{Y\cdot X}=\frac{c^T\Sigma_X^{-1}c}{\sigma_Y^2}. $$ The multiple correlation itself is the nonnegative square root $R_{Y\cdot X}=\sqrt{R^2_{Y\cdot X}}$.

In block form, the joint covariance matrix is

\operatorname{Cov}\begin{pmatrix}X\\Y\end{pmatrix} = \begin{pmatrix} \Sigma_X & c\\ c^T & \sigma_Y^2 \end{pmatrix}.

The entries of $c$ are $\operatorname{Cov}(X_j,Y)$, while $\Sigma_X$ records both the variances of individual predictors and their covariances with one another. The denominator puts the explained variance on the scale of the response variance.

Why the Formula Uses the Inverse Covariance Matrix

For a coefficient vector $b\in\mathbb{R}^p$, consider the centered linear combination $b^T(X-E[X])$ as a predictor of the centered response $Y-E[Y]$. Its covariance with the response is $b^Tc$, and its variance is $b^T\Sigma_Xb$. The best linear combination is the one whose residual variance is smallest. The following derivation shows how the covariance blocks determine that combination and the resulting multiple correlation.

Theorem: Suppose $E[X]$ and $E[Y]$ exist, all variances and covariances are finite, $\operatorname{Var}(Y)>0$, and $\Sigma_X$ is positive definite. Among all $b\in\mathbb{R}^p$, the residual variance $$ \operatorname{Var}\big((Y-E[Y])-b^T(X-E[X])\big) $$ is minimized at $b_*=\Sigma_X^{-1}c$. The minimum residual variance is $\sigma_Y^2-c^T\Sigma_X^{-1}c$, so the fraction of response variance explained by this best linear combination is $R^2_{Y\cdot X}$.

Proof. Expanding the variance using bilinearity of covariance gives, for any $b$,

\operatorname{Var}\big((Y-E[Y])-b^T(X-E[X])\big) =\sigma_Y^2-2b^Tc+b^T\Sigma_Xb.

Because $\Sigma_X$ is positive definite, it is invertible. Set $b_*=\Sigma_X^{-1}c$, so that $\Sigma_Xb_*=c$. Completing the square around $b_*$ yields

\begin{aligned} \sigma_Y^2-2b^Tc+b^T\Sigma_Xb &=\sigma_Y^2-c^T\Sigma_X^{-1}c +(b-b_*)^T\Sigma_X(b-b_*). \end{aligned}

The last term is nonnegative and equals zero only when $b=b_*$, by positive definiteness. Thus $b_*$ uniquely minimizes the residual variance, and the minimum is $\sigma_Y^2-c^T\Sigma_X^{-1}c$. The reduction in variance from the response-only prediction $E[Y]$ is therefore $c^T\Sigma_X^{-1}c$. Dividing that reduction by $\sigma_Y^2$ gives the stated fraction explained, $R^2_{Y\cdot X}$. $\square$

This derivation also gives a useful validity check. Since a variance cannot be negative, the minimum residual variance must be nonnegative. Consequently the covariance blocks must satisfy the following bound.

Result: Under the theorem’s assumptions, $$ 0\leq c^T\Sigma_X^{-1}c\leq \sigma_Y^2, \qquad 0\leq R^2_{Y\cdot X}\leq 1. $$

Proof. The inverse of a positive definite matrix is positive definite, so $c^T\Sigma_X^{-1}c\geq 0$, with equality when $c=0$. The minimizing residual variance derived above is a variance and hence is nonnegative: $\sigma_Y^2-c^T\Sigma_X^{-1}c\geq 0$. Combining these inequalities and dividing by the positive number $\sigma_Y^2$ proves the result. $\square$

Worked Calculations from Covariance Blocks

Each calculation follows the same sequence: identify $\sigma_Y^2$, $\Sigma_X$, and $c$; solve for $\Sigma_X^{-1}c$; take the quadratic form $c^T\Sigma_X^{-1}c$; and divide by the response variance. Checking the resulting value against the bounds is a simple way to catch arithmetic or transcription errors.

Worked Example: Two Correlated Predictors

Suppose the response and two predictors have covariance blocks $$ \sigma_Y^2=16,\qquad \Sigma_X=\begin{pmatrix}4&1\\1&9\end{pmatrix},\qquad c=\begin{pmatrix}3\\6\end{pmatrix}. $$ Compute the squared multiple correlation.

The determinant of $\Sigma_X$ is $4(9)-1(1)=35>0$, and its diagonal entries are positive, so this symmetric $2\times2$ matrix is positive definite. Its inverse is $$ \Sigma_X^{-1} =\frac{1}{35}\begin{pmatrix}9&-1\\-1&4\end{pmatrix}. $$ Multiplying by $c$, $$ \Sigma_X^{-1}c =\frac{1}{35} \begin{pmatrix}9(3)-1(6)\\-1(3)+4(6)\end{pmatrix} =\frac{1}{35}\begin{pmatrix}21\\21\end{pmatrix} =\begin{pmatrix}0.6\\0.6\end{pmatrix}. $$ Therefore, $$ c^T\Sigma_X^{-1}c =\begin{pmatrix}3&6\end{pmatrix} \begin{pmatrix}0.6\\0.6\end{pmatrix} =1.8+3.6=5.4, \qquad R^2_{Y\cdot X}=\frac{5.4}{16}=0.3375. $$ Thus the squared multiple correlation is $0.3375$, and the multiple correlation is $\sqrt{0.3375}\approx0.581$. The corresponding minimum residual variance is $16-5.4=10.6$, which is nonnegative as required.

Worked Example: Standardized Variables with Correlated Predictors

Suppose $Y,X_1,X_2$ each have variance one, the predictors have correlation $0.5$, and their correlations with $Y$ are $0.6$ and $0.5$. Then $$ \sigma_Y^2=1,\qquad \Sigma_X=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix},\qquad c=\begin{pmatrix}0.6\\0.5\end{pmatrix}. $$ The determinant of $\Sigma_X$ is $1-0.5^2=0.75>0$, so it is positive definite. Its inverse is $$ \Sigma_X^{-1}=\frac{1}{0.75}\begin{pmatrix}1&-0.5\\-0.5&1\end{pmatrix}. $$ First multiply: $$ \begin{pmatrix}1&-0.5\\-0.5&1\end{pmatrix} \begin{pmatrix}0.6\\0.5\end{pmatrix} = \begin{pmatrix}0.6-0.25\\-0.3+0.5\end{pmatrix} = \begin{pmatrix}0.35\\0.20\end{pmatrix}. $$ Thus $$ R^2_{Y\cdot X} =c^T\Sigma_X^{-1}c =\frac{0.6(0.35)+0.5(0.20)}{0.75} =\frac{0.21+0.10}{0.75} =\frac{0.31}{0.75} \approx0.4133. $$ The multiple correlation is approximately $\sqrt{0.4133}=0.643$. In this example, accounting for the predictor covariance is essential: the formula uses the full predictor covariance matrix, not just the two response–predictor correlations.

Worked Example: Uncorrelated Predictors with Unequal Variances

Suppose there are three predictors with $$ \sigma_Y^2=4,\qquad \Sigma_X=\begin{pmatrix}1&0&0\\0&4&0\\0&0&9\end{pmatrix},\qquad c=\begin{pmatrix}0.3\\1\\1.5\end{pmatrix}. $$ The inverse predictor covariance matrix is $$ \Sigma_X^{-1}=\begin{pmatrix}1&0&0\\0&0.25&0\\0&0&1/9\end{pmatrix}. $$ Hence $$ \Sigma_X^{-1}c =\begin{pmatrix}0.3\\0.25\\1.5/9\end{pmatrix} =\begin{pmatrix}0.3\\0.25\\1/6\end{pmatrix}, $$ and $$ c^T\Sigma_X^{-1}c =0.3(0.3)+1(0.25)+1.5(1/6) =0.09+0.25+0.25=0.59. $$ Dividing by the response variance gives $$ R^2_{Y\cdot X}=\frac{0.59}{4}=0.1475. $$ The minimum residual variance is $4-0.59=3.41$, so this value is consistent with the nonnegativity check. The unequal predictor variances are handled directly by the diagonal entries of $\Sigma_X^{-1}$.

How to Read and Check the Calculation

The quadratic form $c^T\Sigma_X^{-1}c$ is the amount of response variance accounted for by the best linear combination of the predictors. It has the same units as a variance: the entries of $c$ have units of predictor times response, and the inverse covariance matrix adjusts for the predictor scales and dependencies. Dividing by $\sigma_Y^2$ makes the result dimensionless.

The formula is a population quantity: its inputs are variances and covariances of the joint distribution. If those quantities are replaced by sample estimates, the same algebra produces a sample calculation, but the resulting quantity is then based on estimated blocks. The interpretation of the formula does not by itself settle questions about sampling uncertainty or testing.

A positive definite $\Sigma_X$ is the condition that makes the inverse in this formula well-defined and the minimizing coefficient vector unique. If predictors are exactly linearly dependent, $\Sigma_X$ is singular and this inverse-based expression cannot be used as written. That case requires a separate treatment rather than silently attempting to invert a singular matrix.

Common Mistake: Adding Pairwise Correlations Squared

A tempting shortcut is to square each correlation between $Y$ and a predictor and add the results. That does not generally give the squared multiple correlation. The shortcut ignores covariance among predictors, even though shared variation changes how much each predictor contributes to the best linear combination. The standardized example above makes this visible: the pairwise squared correlations sum to $0.6^2+0.5^2=0.61$, whereas the covariance-block calculation gives approximately $0.4133$.

Another useful check is that the answer must lie between zero and one under the stated assumptions. A value outside that interval usually signals an incorrect inverse, a mismatch between the covariance blocks, or a division error. The blocks must all refer to the same variables and the same scale; for example, covariances from one set of units should not be combined with a predictor covariance matrix from another scaling.

Read the blocks.
Record the scalar response variance $\sigma_Y^2$, the predictor covariance matrix $\Sigma_X$, and the response–predictor covariance vector $c$.

Check the assumptions.
Confirm that $\sigma_Y^2>0$ and that $\Sigma_X$ is invertible; for the theorem and formula here, it is assumed positive definite.

Compute the quadratic form.
Solve $\Sigma_Xu=c$, then calculate $c^Tu$. This is often simpler and more reliable than writing out the full inverse.

Normalize and check.
Divide $c^Tu$ by $\sigma_Y^2$, then verify that the result lies in $[0,1]$.

Key takeaway: With a positive definite predictor covariance matrix, the squared multiple correlation is $c^T\Sigma_X^{-1}c/\sigma_Y^2$. The inverse accounts for dependence and scale among predictors; the normalized result is the proportion of response variance explained by their best linear combination.

Check Your Understanding

Use the covariance-block formula and the assumptions stated above to answer the following.

In the block covariance matrix, what are the dimensions of $\Sigma_X$ and $c$ when there are $p$ predictors?
Why does the formula divide $c^T\Sigma_X^{-1}c$ by $\sigma_Y^2$?
For standardized $Y,X_1,X_2$, suppose $\Sigma_X=\begin{pmatrix}1&0\\0&1\end{pmatrix}$ and $c=(0.3,0.4)^T$. Compute $R^2_{Y\cdot X}$.
What does positive definiteness of $\Sigma_X$ ensure for the inverse-based calculation?
Why is adding the squared pairwise correlations between $Y$ and its predictors not generally a valid calculation of the squared multiple correlation?