Multiple Correlation as a Projection Length

Why a Projection Defines Multiple Correlation

With several predictors, their linear combinations form a space of possible fitted patterns. Multiple correlation summarizes how much the response aligns with that space, without assigning the summary to any one predictor. The geometry is especially useful: the fitted response is the projection of the observed response onto the predictor space, and the multiple correlation is the length of that projection relative to the response’s length.

We work with $n$ observations. Let $y\in\mathbb{R}^n$ be the response vector after subtracting its sample mean, and let $Z$ be the matrix whose columns are the predictors after subtracting their sample means. Thus the entries of $y$ and of each column of $Z$ sum to zero. Centering is equivalent to including an intercept and then considering the centered fitted values. The Euclidean inner product is $u^\mathsf{T}v$, and the squared length of a vector $u$ is $u^\mathsf{T}u$.

Definition: The predictor space is the column space $S=\operatorname{col}(Z)$, the set of all linear combinations of the centered predictor columns. Let $P_S$ be the orthogonal projection onto $S$. The projected response is $\hat y=P_Sy$, and its residual is $e=y-\hat y$. If $y\ne 0$ and $\hat y\ne 0$, the sample multiple correlation is the ordinary sample correlation between $y$ and $\hat y$: $$ R_{y\cdot Z}=\operatorname{Corr}(y,\hat y). $$ When $\hat y=0$, that ordinary correlation is undefined; by convention, multiple correlation is set to $0$, consistent with $R_{y\cdot Z}^2=0$. If $y=0$, the response has no variation and multiple correlation is not defined.

The projection is well-defined even if some predictor columns are redundant: it depends on the space they span, not on a particular list of columns. The fitted vector is centered because every vector in $S$ is centered. The projection residual is orthogonal to every vector in $S$, including the fitted vector itself.

From Projection Length to Squared Multiple Correlation

Write $SS_T=y^\mathsf{T}y$ for the total sum of squares about the response mean, $SS_P=\hat y^\mathsf{T}\hat y$ for the projected sum of squares, and $SS_E=e^\mathsf{T}e$ for the residual sum of squares. The subscript $P$ emphasizes that this is the squared length of the projection. In a least-squares fit with an intercept, these are the usual total, regression, and error sums of squares.

Theorem: For $y\ne 0$, the orthogonal projection decomposition gives $$ SS_T=SS_P+SS_E. $$ If $\hat y\ne 0$, then $$ R_{y\cdot Z}=\frac{\|\hat y\|}{\|y\|}, \qquad R_{y\cdot Z}^{\,2}=\frac{SS_P}{SS_T} =1-\frac{SS_E}{SS_T}. $$ If $\hat y=0$, the convention $R_{y\cdot Z}=0$ gives the same squared-sum-of-squares formula.

Proof of the sum-of-squares decomposition. By definition $y=\hat y+e$. Since $\hat y\in S$ and $e$ is orthogonal to $S$, $\hat y^\mathsf{T}e=0$. Expanding the squared length,

y^\mathsf{T}y =(\hat y+e)^\mathsf{T}(\hat y+e) =\hat y^\mathsf{T}\hat y+2\hat y^\mathsf{T}e+e^\mathsf{T}e =SS_P+SS_E.

Proof of the correlation formula. Both $y$ and $\hat y$ are centered, so their sample correlation is their sample covariance divided by their sample standard deviations. The common factors involving $n-1$ cancel, giving

\operatorname{Corr}(y,\hat y) =\frac{y^\mathsf{T}\hat y} {\sqrt{(y^\mathsf{T}y)(\hat y^\mathsf{T}\hat y)}}.

Because $y=\hat y+e$ and $e^\mathsf{T}\hat y=0$, we have $y^\mathsf{T}\hat y=\hat y^\mathsf{T}\hat y$. If $\hat y\ne0$, substitution yields

\operatorname{Corr}(y,\hat y) =\frac{\hat y^\mathsf{T}\hat y} {\sqrt{(y^\mathsf{T}y)(\hat y^\mathsf{T}\hat y)}} =\frac{\|\hat y\|}{\|y\|}.

Squaring gives $R_{y\cdot Z}^{\,2}=SS_P/SS_T$. Dividing the sum-of-squares decomposition by $SS_T>0$ also gives $SS_P/SS_T=1-SS_E/SS_T$. When $\hat y=0$, $SS_P=0$, so the stated convention gives the same squared formula. $\square$

This result also explains why multiple correlation is nonnegative: it is a ratio of lengths. A fitted pattern may point in different directions as predictor coefficients change, but the projection itself has a nonnegative length. In particular, $R^2$ is the fraction of the centered response’s squared length lying in the predictor space; the remaining fraction is the residual squared length.

Worked Calculations

Worked Example: Two Orthogonal Predictors

Consider four observations with centered predictor vectors $x_1=(1,-1,1,-1)^\mathsf{T}$ and $x_2=(1,1,-1,-1)^\mathsf{T}$, and centered response $y=(2.5,-1.5,-0.5,-0.5)^\mathsf{T}$. Find the fitted vector, the residual, and the multiple correlation.

First, $x_1^\mathsf{T}x_1=x_2^\mathsf{T}x_2=4$ and $x_1^\mathsf{T}x_2=0$. The response can be written as $y=x_1+0.5x_2+w$, where $w=(1,-1,-1,1)^\mathsf{T}$. Direct checks give $x_1^\mathsf{T}w=0$, $x_2^\mathsf{T}w=0$, and $w^\mathsf{T}w=4$. Therefore the projection onto the span of the two predictors is

\hat y=x_1+0.5x_2=(1.5,-0.5,0.5,-1.5)^\mathsf{T}, \qquad e=y-\hat y=w.

The projected sum of squares is $SS_P=4+(0.5)^2(4)=5$, since the two predictors are orthogonal. The residual sum of squares is $SS_E=4$, so $SS_T=9$. Hence

R_{y\cdot Z}^{\,2}=\frac{5}{9}, \qquad R_{y\cdot Z}=\frac{\sqrt{5}}{3}\approx 0.745.

The same total is obtained directly: $y^\mathsf{T}y=2.5^2+(-1.5)^2+(-0.5)^2+(-0.5)^2=9$. The residual is orthogonal to both predictors, confirming that it is the projection residual.

Worked Example: Redundant Predictors Still Define a Projection

Let $x_1=(1,-1,1,-1)^\mathsf{T}$, $x_2=2x_1$, and $y=(3,-1,1,-3)^\mathsf{T}$. Although there are two predictor columns, they span only one direction. Find the multiple correlation.

The predictor space is $S=\operatorname{span}(x_1)$. Since $x_1^\mathsf{T}x_1=4$ and $x_1^\mathsf{T}y=3+1+1+3=8$, projection onto this one-dimensional space gives

\hat y=\frac{x_1^\mathsf{T}y}{x_1^\mathsf{T}x_1}x_1 =2x_1=(2,-2,2,-2)^\mathsf{T}.

Thus $e=y-\hat y=(1,1,-1,-1)^\mathsf{T}$. Its inner product with $x_1$ is $1-1-1+1=0$, as required. The squared lengths are $SS_P=2^2(4)=16$, $SS_E=4$, and $SS_T=y^\mathsf{T}y=9+1+1+9=20$. Consequently

R_{y\cdot Z}^{\,2}=\frac{16}{20}=0.8, \qquad R_{y\cdot Z}=\sqrt{0.8}\approx0.894.

No inverse of a predictor cross-product matrix is needed for this definition. The column space and its orthogonal projection remain unambiguous even when the columns are linearly dependent.

Worked Example: A Predictor Space with Zero Fit

Take centered vectors $x_1=(1,-1,1,-1)^\mathsf{T}$, $x_2=(1,1,-1,-1)^\mathsf{T}$, and $y=(1,-1,-1,1)^\mathsf{T}$. Each inner product $x_1^\mathsf{T}y$ and $x_2^\mathsf{T}y$ is zero. Since $x_1$ and $x_2$ are orthogonal and nonzero, $y$ is orthogonal to their entire span, so $\hat y=0$.

Here $SS_T=y^\mathsf{T}y=4$, $SS_P=0$, and $SS_E=4$. The ordinary correlation with $\hat y$ cannot be calculated because the fitted vector has zero variance. Under the stated convention, the multiple correlation is $0$, and the squared formula gives $R^2=SS_P/SS_T=0/4=0$.

Interpretation and Common Pitfalls

The projection-length interpretation makes multiple correlation a measure of the overall alignment between a response and a set of predictors. It does not describe the direction or size of an individual predictor’s coefficient. Different collections of predictor columns can span the same space and therefore yield the same projection and multiple correlation. Adding a predictor can enlarge that space, but the meaning of the resulting change is a separate question; here the key fact is that the projected length is determined by the full predictor space.

Confusing $R$ with $R^2$. Multiple correlation is the nonnegative length ratio $\|\hat y\|/\|y\|$; its square is the proportion $SS_P/SS_T$. Reporting the proportion as $R$ rather than $R^2$ changes the quantity.
Assuming the fitted vector is always correlated with the response. If the projection is zero, the ordinary correlation is undefined because the fitted vector has zero variance. The conventional multiple correlation is nevertheless set to zero, matching the squared-sum-of-squares formula.
Using an uncentered response while omitting the intercept. The stated sums of squares are about the response mean. Centering the response and predictors, or equivalently fitting an intercept and working with centered fitted values, keeps the projection decomposition aligned with those sums of squares.
Thinking every predictor must contribute a distinct direction. Redundant columns do not invalidate the definition. Projection is onto their column space, which may have lower dimension than the number of columns.

Key takeaway: Project the centered response onto the space spanned by the centered predictors. The multiple correlation is the projected length divided by the response length, and its square is the projected sum of squares divided by the total sum of squares.

Check Your Understanding

Use the projection and sum-of-squares definitions to answer these questions.

Why is the residual from an orthogonal projection uncorrelated with every fitted vector in the predictor space?
If $SS_T=50$ and $SS_E=18$, calculate $SS_P$, $R^2$, and $R$.
What convention is used for multiple correlation if the projected response is zero, and why is the ordinary correlation formula unavailable?
Why can linearly dependent predictor columns still define a valid multiple correlation?
Explain why the multiple correlation is nonnegative even if some fitted coefficients are negative.

Multiple Correlation as a Projection Length

What You'll Learn