Multiple Correlation Versus Regression Coefficients

One Predictor Set, Two Kinds of Summary

A regression model with several predictors produces both a scalar summary of fit and a vector of coefficients. These answer different questions. The multiple correlation describes how strongly the response is linearly associated with the predictor set as a whole; regression coefficients describe a particular representation of the linear prediction in the chosen predictor coordinates. The distinction matters whenever a predictor is recorded in different units or the predictors are replaced by other linear combinations.

For example, changing a length measurement from metres to centimetres multiplies its numerical values by 100. The fitted response need not change, but the coefficient attached to that numerical measurement must change to compensate. By contrast, the multiple correlation and its square, \(R^2\), describe the same fitted response and therefore do not change under an invertible re-expression of the predictors.

Population Definitions

Let \(Y\) be a scalar response and \(X=(X_1,\ldots,X_p)^\mathsf{T}\) a vector of predictors. Assume that \(Y\) and every component of \(X\) have finite second moments, that \(\operatorname{Var}(Y)>0\), and that the predictor covariance matrix is positive definite. Write the centered variables as \(Y-E[Y]\) and \(X-E[X]\). The population linear regression with an intercept is the best linear prediction in mean squared error; centering lets us describe its slope coefficients separately from its intercept.

Definition: Let \(\Sigma=\operatorname{Cov}(X,X)\), \(c=\operatorname{Cov}(X,Y)\), and \(\sigma_Y^2=\operatorname{Var}(Y)\). The population slope vector for the linear regression of \(Y\) on \(X\) is \(\beta=\Sigma^{-1}c\), and the fitted centered response is \(\hat Y_{\mathrm{lin}}=\beta^\mathsf{T}(X-E[X])\). Define \(R^2=\operatorname{Var}(\hat Y_{\mathrm{lin}})/\sigma_Y^2\), and define the multiple correlation as \(R=\sqrt{R^2}\), taking the nonnegative square root. Thus, \(R^2\), not \(R\), is the proportion of response variance explained by the linear predictor.

The intercept in the uncentered regression is \(E[Y]-\beta^\mathsf{T}E[X]\). In what follows, the symbols \(X\) and \(Y\) in covariance calculations can be read as centered variables; centering does not affect covariances or slopes. Positive definiteness of \(\Sigma\) guarantees a unique slope vector. If the predictors are linearly dependent, the coefficient vector may not be unique, so the coordinate-change statements about unique coefficients require additional care.

Result: With these assumptions, \(R^2=c^\mathsf{T}\Sigma^{-1}c/\sigma_Y^2\), and \(0\leq R^2\leq1\). If the fitted response has positive variance, \(R=\operatorname{Corr}(Y,\hat Y_{\mathrm{lin}})\). If it has zero variance, \(R=0\) by the definition above.

Why the Multiple Correlation Is a Scalar Summary

The coefficients depend on the predictor coordinates, but the fitted response is determined by the linear information in the predictors. The next result makes the distinction precise. Let \(A\) be any invertible \(p\times p\) matrix and define new predictors \(Z=AX\). Such a transformation can rescale variables, reorder them, or replace them by invertible linear combinations; it retains exactly the same predictor span.

Theorem: Under the population assumptions above, regression on \(Z=AX\) gives coefficients \(\gamma=A^{-\mathsf{T}}\beta\). The fitted response is unchanged, and \(R^2\) and \(R\) are unchanged.

Proof. The covariance matrix and response covariance vector for \(Z\) are \(\Sigma_Z=A\Sigma A^\mathsf{T}\) and \(c_Z=Ac\). Since \(A\) and \(\Sigma\) are invertible, the regression coefficients in the new coordinates are

\gamma=\Sigma_Z^{-1}c_Z =(A^\mathsf{T})^{-1}\Sigma^{-1}A^{-1}Ac =A^{-\mathsf{T}}\beta.

Therefore \(\gamma^\mathsf{T}Z=\beta^\mathsf{T}X\), so the fitted response is identical. The explained variance formula also gives

c_Z^\mathsf{T}\Sigma_Z^{-1}c_Z =c^\mathsf{T}A^\mathsf{T}(A^\mathsf{T})^{-1} \Sigma^{-1}A^{-1}Ac =c^\mathsf{T}\Sigma^{-1}c.

Dividing by the unchanged \(\sigma_Y^2\) proves that \(R^2\) is unchanged, and hence its nonnegative square root \(R\) is unchanged. \(\square\)

This invariance concerns transformations of the predictors, not arbitrary transformations of the response. It also requires an invertible change of coordinates: discarding a predictor or replacing two predictors by a single sum can lose information and change the set of available linear predictions.

Worked Example: Changing Predictor Units

Suppose \(X_1\) and \(X_2\) are centered, have variance 1, and are uncorrelated. Let \(\varepsilon\) be centered and uncorrelated with both predictors, with variance 3, and set \(Y=2X_1+X_2+\varepsilon\). Find the regression coefficients and multiple correlation, then rescale the predictors.

Here \(\Sigma\) is the \(2\times2\) identity matrix and \(c=(2,1)^\mathsf{T}\). Thus \(\beta=\Sigma^{-1}c=(2,1)^\mathsf{T}\). Since the three terms in \(Y\) are pairwise uncorrelated,

\operatorname{Var}(Y)=2^2(1)+1^2(1)+3=8, \qquad \operatorname{Var}(\hat Y_{\mathrm{lin}})=2^2(1)+1^2(1)=5.

Consequently \(R^2=5/8=0.625\) and \(R=\sqrt{5/8}\approx0.791\). Now measure the first predictor in units 100 times as large and the second in units 0.1 times as large: let \(U=100X_1\) and \(V=0.1X_2\). The same fitted response is \(0.02U+10V\), so the new coefficients are \(0.02\) and \(10\), rather than \(2\) and \(1\).

The fitted variance is still \(0.02^2(10000)+10^2(0.01)=4+1=5\), because \(\operatorname{Var}(U)=10000\), \(\operatorname{Var}(V)=0.01\), and the transformed predictors remain uncorrelated. The response variance is still 8. Therefore \(R^2=5/8\) and \(R\approx0.791\), despite the substantial change in the numerical coefficients.

The Multiple Correlation and the Fitted Response

The formula for \(R^2\) can be understood through the regression residual. Center the variables and write \(e=Y-\beta^\mathsf{T}X\). By the definition \(\beta=\Sigma^{-1}c\), \(\operatorname{Cov}(X,e)=c-\Sigma\beta=0\). Thus the fitted response and residual are uncorrelated. This gives a variance decomposition and also explains why the multiple correlation is nonnegative.

Result: The fitted response and residual satisfy \(\operatorname{Var}(Y)=\operatorname{Var}(\hat Y_{\mathrm{lin}})+\operatorname{Var}(e)\). If \(\operatorname{Var}(\hat Y_{\mathrm{lin}})>0\), then \(\operatorname{Corr}(Y,\hat Y_{\mathrm{lin}})=\sqrt{\operatorname{Var}(\hat Y_{\mathrm{lin}})/\operatorname{Var}(Y)}=R\).

Derivation. Since \(\operatorname{Cov}(X,e)=0\), we have \(\operatorname{Cov}(\beta^\mathsf{T}X,e)=0\). As \(Y=\beta^\mathsf{T}X+e\), it follows that

\operatorname{Var}(Y) =\operatorname{Var}(\beta^\mathsf{T}X)+\operatorname{Var}(e) =\beta^\mathsf{T}\Sigma\beta+\operatorname{Var}(e).

Also, \(\beta^\mathsf{T}\Sigma\beta=c^\mathsf{T}\Sigma^{-1}c\), because \(\beta=\Sigma^{-1}c\) and \(\Sigma\) is symmetric. This proves the stated formula for \(R^2\). The variance decomposition shows that the explained-variance ratio is between zero and one. Finally, when the fitted variance is positive, \(\operatorname{Cov}(Y,\hat Y_{\mathrm{lin}})=\operatorname{Var}(\hat Y_{\mathrm{lin}})\), so

\operatorname{Corr}(Y,\hat Y_{\mathrm{lin}}) =\frac{\operatorname{Var}(\hat Y_{\mathrm{lin}})} {\sqrt{\operatorname{Var}(Y)\operatorname{Var}(\hat Y_{\mathrm{lin}})}} =R.

If the fitted variance is zero, then \(\beta=0\) under positive definiteness of \(\Sigma\), so \(c=0\) and the multiple correlation is defined as zero. \(\square\)

Worked Example: Mixing Predictor Coordinates

Use the same population model as in the previous example, but replace the predictors by \(W_1=X_1+X_2\) and \(W_2=X_1-X_2\). Find the new coefficients and verify the explained variance.

The transformation matrix is invertible, and \(X_1=(W_1+W_2)/2\), \(X_2=(W_1-W_2)/2\). Substituting into the linear part of the response gives

2X_1+X_2 =2\frac{W_1+W_2}{2}+\frac{W_1-W_2}{2} =1.5W_1+0.5W_2.

Thus the new coefficient vector is \((1.5,0.5)^\mathsf{T}\), not \((2,1)^\mathsf{T}\). To check it from covariances, \(\operatorname{Var}(W_1)=\operatorname{Var}(W_2)=2\) and \(\operatorname{Cov}(W_1,W_2)=0\). Moreover, \(c_W=(\operatorname{Cov}(W_1,Y),\operatorname{Cov}(W_2,Y))^\mathsf{T}=(3,1)^\mathsf{T}\), so \(\Sigma_W^{-1}c_W=(1.5,0.5)^\mathsf{T}\).

The fitted variance is \(1.5^2(2)+0.5^2(2)=4.5+0.5=5\). Since \(\operatorname{Var}(Y)=8\), \(R^2=5/8\) and \(R=\sqrt{5/8}\), just as before. The coefficients changed because the coordinates changed; the linear predictor and its fit did not.

Checking the Distinction in a Sample

The same idea applies to ordinary least squares in a finite data set when the design matrix, including an intercept, has full column rank. An invertible change of predictor columns changes the coefficient representation but leaves the fitted-value vector unchanged. The next example verifies this directly with exact sums of squares.

Worked Example: Rescaling Predictors in Four Observations

Consider four centered observations with predictors \(x=(-1,-1,1,1)\), \(z=(-1,1,-1,1)\), and response \(y=(-2,-2,0,4)\). Both predictors sum to zero, their cross-product is zero, and each has sum of squares 4. With an intercept in the model, the intercept estimate is zero. The least-squares slopes are

\hat\beta_x=\frac{\sum_i x_i y_i}{\sum_i x_i^2} =\frac{8}{4}=2, \qquad \hat\beta_z=\frac{\sum_i z_i y_i}{\sum_i z_i^2} =\frac{4}{4}=1.

The fitted values are \(2x+z=(-3,-1,1,3)\), and the residuals are \(y-\hat y=(1,-1,-1,1)\). The response sum of squares is \(4+4+0+16=24\), while the residual sum of squares is \(1+1+1+1=4\). Therefore

R^2=1-\frac{4}{24}=\frac{5}{6}, \qquad R=\sqrt{\frac{5}{6}}\approx0.913.

Now use \(x^*=10x\) and \(z^*=0.5z\). The new slopes are \(\hat\beta_{x^*}=80/400=0.2\) and \(\hat\beta_{z^*}=2/1=2\): the cross-products with \(y\) are 80 and 2, and the predictor sums of squares are 400 and 1. Yet \(0.2x^*+2z^*=2x+z\), so the fitted values, residuals, \(R^2=5/6\), and \(R\approx0.913\) are unchanged.

Interpretation and a Common Pitfall

A coefficient is expressed in units of response per unit of its predictor, conditional on the other included predictors in the linear model. Its value therefore depends on the measurement scale and, more generally, on the coordinates used to describe the predictor span. A multiple correlation is a nonnegative scalar summary of the linear fit to the response using the whole predictor set. It does not identify the direction or meaning of each coefficient, and neither the coefficients nor the multiple correlation alone establish a causal relationship.

A frequent mistake is to say, “The multiple correlation is the proportion of variance explained, so it equals \(R^2\).” The correction is precise: \(R^2\) is the proportion of response variance explained, while \(R=\sqrt{R^2}\) is the multiple correlation. Both remain unchanged under invertible predictor-coordinate changes, but they are not the same quantity. Another mistake is to interpret a larger coefficient after rescaling as stronger association. The units changed; compare fitted values or a clearly defined scale-free summary rather than raw coefficient magnitudes.

Invertibility is essential to the invariance claim. If one replaces \(X_1\) and \(X_2\) by only \(X_1+X_2\), the transformation discards a direction in the predictor space. The available linear predictions may then be more limited, and \(R^2\) can decrease. Invariance applies when the transformed coordinates retain all of the original predictor information, not whenever predictors are altered in any way.

Key takeaway: \(R^2\) is the proportion of response variance explained by the predictor set, and \(R=\sqrt{R^2}\) is the multiple correlation. Both are invariant under invertible changes of predictor coordinates. Regression coefficients, by contrast, depend on the units and coordinates used to express the predictors.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

If a predictor is multiplied by 100 and the fitted response is kept the same, how must its coefficient change?
What is the distinction between \(R\) and \(R^2\), and which one is the proportion of response variance explained?
Why do an invertible linear transformation of the predictors and the corresponding coefficient change leave fitted values unchanged?
In the four-observation example, verify that the response sum of squares equals the fitted sum of squares plus the residual sum of squares.
Would replacing two predictors by their sum alone necessarily preserve \(R^2\)? Explain why or why not.