Least Squares as Orthogonal Projection

Least Squares as a Geometric Problem

In multiple regression, a fitted response vector must be compatible with the predictors in the design matrix. That restriction means not every vector in \(\mathbb{R}^n\) can be a possible fit. The set of all possible fitted vectors is the column space of the design matrix. Ordinary least squares chooses the vector in that space closest to the observed response vector, using ordinary Euclidean distance.

This geometric view gives a direct interpretation of the least-squares criterion: minimizing the sum of squared residuals is the same as finding the shortest distance from the response vector to the column space. It also explains why the fitted vector is a projection, rather than merely a collection of separate predictions.

Definition: Let \(X\) be an \(n \times p\) design matrix and let \(y \in \mathbb{R}^n\) be the response vector. The column space of \(X\), written \(\mathcal{C}(X)\), is the set of all vectors \(Xb\) as \(b\) ranges over \(\mathbb{R}^p\). A vector \(\hat{y} \in \mathcal{C}(X)\) is the orthogonal projection of \(y\) onto \(\mathcal{C}(X)\) if \(y-\hat{y}\) is perpendicular to every vector in \(\mathcal{C}(X)\).

For any coefficient vector \(b\), the corresponding fitted response vector is \(Xb\), and its residual vector is \(y-Xb\). The residual sum of squares is the squared length of that residual:

\operatorname{RSS}(b) = \lVert y-Xb\rVert^2 = (y-Xb)'(y-Xb).

Here, the prime denotes transpose and \(\lVert \cdot \rVert\) is Euclidean length. The distinction between \(b\) and \(Xb\) matters: \(b\) contains coefficients, whereas \(Xb\) is a vector of fitted responses, one for each observation.

From the Normal Equations to a Projection

Assume \(X\) is fixed and has full column rank. Ordinary least squares selects the coefficient vector \(\hat{\beta}\) that minimizes \(\lVert y-Xb\rVert^2\). The normal equations are \(X'X\hat{\beta}=X'y\). Since \(X\) has full column rank, \(X'X\) is invertible, so the coefficients and fitted vector are:

\hat{\beta}=(X'X)^{-1}X'y, \qquad \hat{y}=X\hat{\beta}=X(X'X)^{-1}X'y.

The vector \(\hat{y}\) belongs to \(\mathcal{C}(X)\), because it is \(X\) multiplied by a coefficient vector. The normal equations also give \(X'(y-\hat{y})=X'y-X'X\hat{\beta}=0\). Thus the residual \(e=y-\hat{y}\) is perpendicular to every column of \(X\), and therefore to every linear combination of those columns—that is, to every vector in \(\mathcal{C}(X)\).

Theorem: If \(X\) has full column rank, the least-squares fitted vector \(\hat{y}=X(X'X)^{-1}X'y\) is the unique vector in \(\mathcal{C}(X)\) closest to \(y\). Its residual \(y-\hat{y}\) is perpendicular to the column space.

The orthogonality condition proves that the fit is not just a stationary point of the criterion; it is the closest point. Take any other candidate \(z\in\mathcal{C}(X)\). Since both \(z\) and \(\hat{y}\) are in the column space, their difference \(d=z-\hat{y}\) is also in the column space. The residual \(e=y-\hat{y}\) is perpendicular to \(d\). As \(y-z=e-d\), the Pythagorean theorem gives:

\lVert y-z\rVert^2 =\lVert e-d\rVert^2 =\lVert e\rVert^2+\lVert d\rVert^2 \geq \lVert e\rVert^2 =\lVert y-\hat{y}\rVert^2.

Equality holds only when \(\lVert d\rVert^2=0\), which means \(d=0\) and \(z=\hat{y}\). This proves both that \(\hat{y}\) minimizes squared residual length and that no other fitted vector in the column space ties it. Full column rank additionally makes the coefficient vector representing that fitted vector unique.

Identify the permitted fits.
Every possible fitted response vector has the form \(Xb\), so all permitted fits lie in \(\mathcal{C}(X)\).

Find the least-squares coefficients.
Use the normal equations to obtain \(\hat{\beta}\), then form \(\hat{y}=X\hat{\beta}\).

Check the geometric condition.
Verify that \(X'(y-\hat{y})=0\). The residual is then perpendicular to the column space.

Conclude that the fit is closest.
Any other fit adds a column-space displacement to the residual, increasing squared distance by the squared length of that displacement.

Worked Calculations

Worked Example: A Regression Line as a Projection

A hypothetical pricing analyst uses an intercept and one numeric rating as predictors for responses \(y=(1,2,2)'\). The rating values are \(0,1,2\), so the design matrix is \(X=\begin{pmatrix}1&0\\1&1\\1&2\end{pmatrix}\). Calculate the least-squares fitted vector and residual sum of squares. Check that the fitted vector is the projection. Method: solve the normal equations. Trap avoided: reporting the coefficients as though they were fitted responses.

First calculate the cross-products:

X'X= \begin{pmatrix} 3&3\\ 3&5 \end{pmatrix}, \qquad X'y= \begin{pmatrix} 1+2+2\\ 0(1)+1(2)+2(2) \end{pmatrix} = \begin{pmatrix}5\\6\end{pmatrix}.

The determinant of \(X'X\) is \(3(5)-3(3)=6\), so it is invertible. Its inverse and the coefficient estimate are:

(X'X)^{-1}=\frac{1}{6} \begin{pmatrix}5&-3\\-3&3\end{pmatrix}, \qquad \hat{\beta} =\frac{1}{6} \begin{pmatrix}5&-3\\-3&3\end{pmatrix} \begin{pmatrix}5\\6\end{pmatrix} = \begin{pmatrix}7/6\\1/2\end{pmatrix}.

Thus the fitted responses, in rating order \(0,1,2\), are:

\hat{y}=X\hat{\beta} = \begin{pmatrix}7/6\\7/6+1/2\\7/6+2(1/2)\end{pmatrix} = \begin{pmatrix}7/6\\5/3\\13/6\end{pmatrix}.

Subtracting fitted values from observed responses gives \(e=y-\hat{y}=(-1/6,\,1/3,\,-1/6)'\). Therefore:

\operatorname{RSS} =\left(-\frac16\right)^2+\left(\frac13\right)^2+\left(-\frac16\right)^2 =\frac{1}{36}+\frac{4}{36}+\frac{1}{36} =\frac16.

For the projection check, the residual's inner product with the intercept column is \(-1/6+1/3-1/6=0\). Its inner product with the rating column is \(0(-1/6)+1(1/3)+2(-1/6)=0\). It is consequently perpendicular to both columns and to their span. As a further check, \(\lVert y\rVert^2=9\), \(\lVert\hat{y}\rVert^2=53/6\), and \(53/6+1/6=9\), as required for perpendicular components. The coefficients are \(7/6\) and \(1/2\); the fitted vector is \((7/6,5/3,13/6)'\).

Worked Example: Projection with Orthogonal Design Columns

A hypothetical service team models four response measurements \(y=(3,1,2,0)'\) using two columns \(x_1=(1,1,1,1)'\) and \(x_2=(1,-1,1,-1)'\). Find the projection of \(y\) onto their span and its residual sum of squares. Method: use the orthogonality and squared lengths of the columns to solve the normal equations. Trap avoided: assuming orthogonal columns have unit length.

The two columns are perpendicular because \(x_1'x_2=1-1+1-1=0\). Each has squared length \(4\). The inner products with \(y\) are \(x_1'y=3+1+2+0=6\) and \(x_2'y=3-1+2-0=4\). The normal equations therefore give:

\hat{\beta}_1=\frac{x_1'y}{x_1'x_1}=\frac64=\frac32, \qquad \hat{\beta}_2=\frac{x_2'y}{x_2'x_2}=\frac44=1.

The fitted vector and residual are:

\hat{y}=\frac32x_1+x_2 = \begin{pmatrix}5/2\\1/2\\5/2\\1/2\end{pmatrix}, \qquad e=y-\hat{y} = \begin{pmatrix}1/2\\1/2\\-1/2\\-1/2\end{pmatrix}.

The residual sum of squares is \(4(1/2)^2=1\). To check the projection, \(x_1'e=1/2+1/2-1/2-1/2=0\), and \(x_2'e=1/2-1/2-1/2+1/2=0\). Also, \(\lVert y\rVert^2=14\), \(\lVert\hat{y}\rVert^2=13\), and \(13+1=14\). The fit is the perpendicular projection, and the unexplained squared length is \(1\).

Worked Example: Projecting onto a Plane in Three Dimensions

A hypothetical analyst has three responses \(y=(1,2,4)'\) and two predictor columns \(x_1=(1,0,1)'\) and \(x_2=(0,1,1)'\). Find the projection onto the plane spanned by those columns and compare its residual sum of squares with that from the candidate coefficient vector \(b=(1,2)'\). Method: calculate the least-squares projection and apply the squared-distance comparison. Trap avoided: assuming that any vector in the predictor plane is equally close to the response.

With \(X=(x_1\ \ x_2)\), the cross-products are:

X'X= \begin{pmatrix}2&1\\1&2\end{pmatrix}, \qquad X'y= \begin{pmatrix}1+4\\2+4\end{pmatrix} = \begin{pmatrix}5\\6\end{pmatrix}, \qquad (X'X)^{-1}=\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}.

Hence \(\hat{\beta}=(4/3,\,7/3)'\), and:

\hat{y}=\frac43x_1+\frac73x_2 = \begin{pmatrix}4/3\\7/3\\11/3\end{pmatrix}, \qquad e=y-\hat{y} = \begin{pmatrix}-1/3\\-1/3\\1/3\end{pmatrix}.

The residual sum of squares is \(1/9+1/9+1/9=1/3\). Its inner products with the columns are \(x_1'e=-1/3+1/3=0\) and \(x_2'e=-1/3+1/3=0\), confirming perpendicularity. The proposed alternative \(b=(1,2)'\) gives \(Xb=(1,2,3)'\), with residual \((0,0,1)'\) and RSS \(1\). This exceeds the projection's RSS by \(1-1/3=2/3\). In fact, the difference between the alternative fit and the projection is \((-1/3,-1/3,-2/3)'\), whose squared length is \(1/9+1/9+4/9=2/3\). The Pythagorean decomposition explains the entire increase.

Assumptions and Interpretation

The projection result is a statement about minimizing squared distance for a given design matrix and response vector. It does not require normally distributed errors, constant error variance, or independent errors. Those assumptions matter for particular statistical conclusions about a regression model, but they are not needed to establish the least-squares geometric result.

Full column rank is needed for the displayed inverse formula and for a unique coefficient vector. The geometric idea is slightly broader: even if columns are dependent, the set \(\mathcal{C}(X)\) is still a subspace and has a unique closest point to \(y\). However, different coefficient vectors can then represent that same fitted vector. The full-rank setting avoids that coefficient ambiguity.

The practical importance of the geometry is that it distinguishes two quantities that are easy to conflate: the response vector decomposes into the fitted component in the column space and the residual component perpendicular to it. The fitted component is the part the specified predictors can represent under the linear model. The residual component is the remaining discrepancy for the observed responses.

Common Exam Trap

A frequent error is to use the inner product with a predictor column as the coefficient without dividing by that column's squared length. In the orthogonal-columns example, \(x_1'y=6\), but the coefficient is not \(6\): because \(x_1'x_1=4\), it is \(6/4=3/2\). Likewise, \(x_2'y=4\) gives coefficient \(4/4=1\). The shortcut \(\hat{\beta}_j=x_j'y\) works only when the columns are mutually orthogonal and each has unit squared length. More generally, solve the normal equations or use the full-rank formula.

A second check is to keep dimensions straight. The coefficient vector has \(p\) entries; the fitted response and residual vectors each have \(n\) entries. The projection is \(X\hat{\beta}\), not \(\hat{\beta}\) itself.

Key takeaway: Least squares chooses the vector in the design-matrix column space that is closest to the observed response. The normal equations make the residual perpendicular to that space, and the Pythagorean theorem shows that every other candidate fit has at least as large a squared distance.

Check Your Understanding

For each problem, identify the projection or compare candidate fits using squared Euclidean distance.

Project \(y=(2,0,4)'\) onto the span of \(u=(1,1,1)'\). Give the fitted vector, residual, and RSS. (Answer: fitted vector \((2,2,2)'\), residual \((0,-2,2)'\), RSS \(8\).)
Let \(x_1=(1,1,1,1)'\), \(x_2=(1,-1,1,-1)'\), and \(y=(4,2,0,2)'\). Find the least-squares coefficients and fitted vector. (Answer: coefficients \((2,0)'\), fitted vector \((2,2,2,2)'\).)
For \(x_1=(1,0,1)'\), \(x_2=(0,1,1)'\), and \(y=(2,1,4)'\), calculate the least-squares coefficients and RSS, then state whether the residual is perpendicular to both columns. (Answer: coefficients \((7/3,4/3)'\), RSS \(1/3\), and the residual is perpendicular to both columns.)
Project \(y=(3,1,2,0)'\) onto the span of \(u=(1,1,0,0)'\) and \(v=(0,0,1,1)'\). Give the fitted vector, residual, and RSS. (Answer: fitted vector \((2,2,1,1)'\), residual \((1,-1,1,-1)'\), RSS \(4\).)
Let \(\mathcal{C}\) be the span of \((1,1,1)'\), and let \(y=(1,0,2)'\). Compare the RSS for the candidate \(z=(2,2,2)'\) with the minimum RSS. (Answer: the projection is \((1,1,1)'\) with minimum RSS \(2\); the candidate \(z\) has RSS \(5\), an increase of \(3\).)

What You'll Learn