Least Squares as a Geometric Projection
In simple linear regression, the fitted line is often described by its intercept and slope. There is also a useful geometric description: the observed response vector is projected onto the space of all vectors that can be written as a constant plus a multiple of the predictor vector. The projection gives the fitted values; the difference between the observed vector and that projection gives the residuals.
This viewpoint makes the least-squares criterion concrete. Among all possible fitted vectors from a line with an intercept, the least-squares fit is the one closest to the observed response vector, where closeness is measured by squared Euclidean distance. It also explains why the residual vector is orthogonal to both the constant vector and the predictor vector.
The word “line” has two related uses here. In a scatterplot of \(x\) against \(y\), the fitted relationship is a line, \(\hat y_i=\hat\beta_0+\hat\beta_1x_i\). In the \(n\)-dimensional space of response vectors, all fitted vectors of this form make up \(\operatorname{span}\{\mathbf{1},\mathbf{x}\}\), which is generally a two-dimensional subspace. The projection is onto that subspace, not generally onto a one-dimensional subspace.
The projection description is algebraic and applies to the observed data without a probability distribution assumption. If the data are also described by the regression model \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), that model gives a statistical interpretation, but the calculation of the least-squares projection itself does not require assumptions about the distribution of the errors.
From the Least-Squares Criterion to Projection
A vector in the fitted-value space has the form \(b_0\mathbf{1}+b_1\mathbf{x}\). The ordinary least-squares fit chooses \(b_0\) and \(b_1\) to minimize the squared distance from \(\mathbf{y}\) to this vector. For any candidate coefficients, that squared distance is the residual sum of squares.
At the minimizing coefficients, the residual vector must be orthogonal to each direction in the fitted-value space. The following derivation shows why.
Writing \(e_i=y_i-b_0-b_1x_i\), the derivative is \(\frac{\partial Q}{\partial b_0}=-2\sum_i e_i\). At the minimum it is zero, so \(\sum_i e_i=0\), or \(\mathbf{1}^\mathsf{T}\mathbf{e}=\mathbf{0}\).
The derivative is \(\frac{\partial Q}{\partial b_1}=-2\sum_i x_i e_i\). Setting it to zero gives \(\sum_i x_i e_i=0\), or \(\mathbf{x}^\mathsf{T}\mathbf{e}=\mathbf{0}\).
The two equations say the residual vector is perpendicular to both spanning vectors, and therefore to every vector in \(\operatorname{span}\{\mathbf{1},\mathbf{x}\}\). Thus \(\hat{\mathbf{y}}\) is the orthogonal projection of \(\mathbf{y}\) onto that space.
Equivalently, form the design matrix \(X=[\mathbf{1}\ \mathbf{x}]\), whose columns are the constant and predictor vectors. When the predictor values are not all equal, \(X\) has two linearly independent columns. The normal equations and projection formula are
The matrix \(P_X=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\) is called the projection matrix, so \(\hat{\mathbf{y}}=P_X\mathbf{y}\). This formula requires the two columns of \(X\) to be linearly independent. If all predictor values are equal, the intercept and slope cannot be uniquely determined from the usual two-parameter fit, and this inverse formula does not apply.
Using Centering to Calculate the Projection
For a simple regression with an intercept, the fitted slope and intercept can be calculated from centered observations. Define \(x_{c,i}=x_i-\bar{x}\) and \(y_{c,i}=y_i-\bar{y}\). The slope is the inner product of the centered predictor and response vectors divided by the squared length of the centered predictor vector. The intercept then makes the fitted line pass through \((\bar{x},\bar{y})\).
This calculation is another way to obtain the projection. The constant component of the fit is \(\bar{y}\mathbf{1}\); the additional component in the centered predictor direction is \(\hat\beta_1\mathbf{x}_c\), where \(\mathbf{x}_c=\mathbf{x}-\bar{x}\mathbf{1}\). Since \(\mathbf{1}^\mathsf{T}\mathbf{x}_c=0\), these directions are perpendicular, which can make the geometry and arithmetic easier to check.
Worked Example: Projecting a Response Vector from Centered Data
A hypothetical analyst records a response \(y\) and predictor \(x\) for three independent locations: \((1,4)\), \((3,5)\), and \((5,9)\). Find the fitted vector and residual vector for the least-squares line with an intercept, then check the projection conditions. Treat these as fixed observations; no interest rate, mortality assumption, or probability distribution is involved or needed.
Method. Use the centered-vector formula for the slope, then calculate the intercept and fitted values. This method avoids fitting a line through the origin when the model includes an intercept.
The vectors are \(\mathbf{x}=(1,3,5)^\mathsf{T}\) and \(\mathbf{y}=(4,5,9)^\mathsf{T}\). Their means are \(\bar{x}=(1+3+5)/3=3\) and \(\bar{y}=(4+5+9)/3=6\). Therefore, the centered vectors are \(\mathbf{x}_c=(-2,0,2)^\mathsf{T}\) and \(\mathbf{y}_c=(-2,-1,3)^\mathsf{T}\).
The numerator for the slope is \((-2)(-2)+(0)(-1)+(2)(3)=4+0+6=10\). The denominator is \((-2)^2+0^2+2^2=4+0+4=8\). Thus \(\hat\beta_1=10/8=1.25\), and \(\hat\beta_0=6-(1.25)(3)=2.25\).
The fitted values are \(2.25+(1.25)(1)=3.5\), \(2.25+(1.25)(3)=6\), and \(2.25+(1.25)(5)=8.5\). Hence \(\hat{\mathbf{y}}=(3.5,6,8.5)^\mathsf{T}\). Subtracting fitted values from observed values gives \(\mathbf{e}=(4-3.5,5-6,9-8.5)^\mathsf{T}=(0.5,-1,0.5)^\mathsf{T}\).
Check the two orthogonality conditions: \(\mathbf{1}^\mathsf{T}\mathbf{e}=0.5-1+0.5=0\), and \(\mathbf{x}^\mathsf{T}\mathbf{e}=(1)(0.5)+(3)(-1)+(5)(0.5)=0.5-3+2.5=0\). The residual is perpendicular to both spanning vectors, confirming the projection.
Worked Example: Applying the Projection in Centered Coordinates
A hypothetical study records \((x,y)=(2,5),(4,6),(6,8),(8,9)\). Find the fitted vector by projecting \(\mathbf{y}\) onto the space spanned by \(\mathbf{1}\) and \(\mathbf{x}\). The observations are fixed; no interest rate, mortality assumption, or probability distribution is specified or needed.
Method. Use the perpendicular basis consisting of \(\mathbf{1}\) and the centered predictor vector. This makes the projection a sum of two simple projections.
The means are \(\bar{x}=(2+4+6+8)/4=5\) and \(\bar{y}=(5+6+8+9)/4=7\). Thus \(\mathbf{x}_c=(-3,-1,1,3)^\mathsf{T}\) and \(\mathbf{y}_c=(-2,-1,1,2)^\mathsf{T}\). The dot product is \(\mathbf{x}_c^\mathsf{T}\mathbf{y}_c=6+1+1+6=14\), and \(\mathbf{x}_c^\mathsf{T}\mathbf{x}_c=9+1+1+9=20\). Therefore \(\hat\beta_1=14/20=0.7\), while \(\hat\beta_0=7-(0.7)(5)=3.5\).
The projection of \(\mathbf{y}\) onto the constant direction is \(7\mathbf{1}\). Its component in the centered predictor direction is \((14/20)\mathbf{x}_c=0.7(-3,-1,1,3)^\mathsf{T}=(-2.1,-0.7,0.7,2.1)^\mathsf{T}\). Adding the components gives \(\hat{\mathbf{y}}=(4.9,6.3,7.7,9.1)^\mathsf{T}\).
The residual vector is \((0.1,-0.3,0.3,-0.1)^\mathsf{T}\). Its entries sum to \(0.1-0.3+0.3-0.1=0\). Its dot product with \(\mathbf{x}\) is \((2)(0.1)+(4)(-0.3)+(6)(0.3)+(8)(-0.1)=0.2-1.2+1.8-0.8=0\). Both checks hold, so the fitted vector is in the required space and the residual is perpendicular to it.
Worked Example: Checking a Projection with Fractional Coefficients
For a hypothetical set of three fixed observations, the predictor and response vectors are \(\mathbf{x}=(1,2,4)^\mathsf{T}\) and \(\mathbf{y}=(3,5,8)^\mathsf{T}\). Find the least-squares fitted vector exactly and verify that its residual is orthogonal to the fitted-value space. No interest or mortality assumption or probability distribution is part of this calculation.
Method. Calculate the centered slope and intercept, retaining fractions until the end. Checking both residual dot products confirms that the result is the projection.
The means are \(\bar{x}=(1+2+4)/3=7/3\) and \(\bar{y}=(3+5+8)/3=16/3\). The centered vectors are \(\mathbf{x}_c=(-4/3,-1/3,5/3)^\mathsf{T}\) and \(\mathbf{y}_c=(-7/3,-1/3,8/3)^\mathsf{T}\). Their dot product is \(28/9+1/9+40/9=69/9=23/3\), and the squared length of \(\mathbf{x}_c\) is \(16/9+1/9+25/9=42/9=14/3\). Therefore, \(\hat\beta_1=(23/3)/(14/3)=23/14\).
The intercept is \(\hat\beta_0=16/3-(23/14)(7/3)=16/3-23/6=32/6-23/6=3/2\). The fitted values are \(3/2+23/14=44/14\), \(3/2+46/14=67/14\), and \(3/2+92/14=113/14\). Thus \(\hat{\mathbf{y}}=(44/14,67/14,113/14)^\mathsf{T}\), and subtracting from \(\mathbf{y}=(42/14,70/14,112/14)^\mathsf{T}\) gives \(\mathbf{e}=(-2/14,3/14,-1/14)^\mathsf{T}\).
The residual sum is \((-2+3-1)/14=0\). Its dot product with the predictor is \((1)(-2/14)+(2)(3/14)+(4)(-1/14)=(-2+6-4)/14=0\). These two zero dot products establish that the residual is orthogonal to the entire fitted-value space.
Common Mistake: Omitting the Constant Direction
When a model includes an intercept, its fitted-value space contains \(\mathbf{1}\) as well as \(\mathbf{x}\). A frequent error is to project \(\mathbf{y}\) onto \(\operatorname{span}\{\mathbf{x}\}\) alone. That fits a line through the origin, which is a different least-squares problem.
For the first worked example, using the through-origin slope would give \((\mathbf{x}^\mathsf{T}\mathbf{y})/(\mathbf{x}^\mathsf{T}\mathbf{x})=(1\cdot4+3\cdot5+5\cdot9)/(1^2+3^2+5^2)=64/35\), with intercept zero. Its residuals would be \(4-64/35=76/35\), \(5-192/35=-17/35\), and \(9-320/35=-5/35\). They sum to \(54/35\), not zero. The correct fit with an intercept has slope \(1.25\), intercept \(2.25\), and residuals that sum to zero and are orthogonal to the predictor.
The projection view is useful because it gives direct checks on a calculation. For an ordinary least-squares fit with an intercept, verify that the residual vector has zero dot product with the constant vector and with the predictor vector. A failure in either check indicates an arithmetic error, an omitted model component, or a different fitting criterion.
Check Your Understanding
For each problem, identify the fitted-value space and use projection or orthogonality ideas as appropriate.
- A regression with an intercept uses predictor vector \((1,2,3)^\mathsf{T}\). State two vectors that span its fitted-value space. (Answer: \(\mathbf{1}=(1,1,1)^\mathsf{T}\) and \(\mathbf{x}=(1,2,3)^\mathsf{T}\).)
- For \(\mathbf{x}=(1,2,3)^\mathsf{T}\) and \(\mathbf{y}=(2,4,5)^\mathsf{T}\), calculate the least-squares slope and intercept. (Answer: \(\hat\beta_1=1.5,\ \hat\beta_0=2/3\).)
- A fitted regression with an intercept has residuals \((1,-2,1)^\mathsf{T}\) and predictor vector \((1,2,4)^\mathsf{T}\). Check the two orthogonality conditions. (Answer: The residual sum is \(1-2+1=0\), and the predictor dot product is \(1-4+4=1\), so the residual is not orthogonal to the fitted-value space.)
- For \(\mathbf{x}=(1,3,5)^\mathsf{T}\) and \(\mathbf{y}=(2,5,8)^\mathsf{T}\), find the fitted vector and verify the residual sum. (Answer: \(\hat\beta_1=1.5,\ \hat\beta_0=0.5,\ \hat{\mathbf{y}}=(2,5,8)^\mathsf{T}\); the residual sum is \(0\).)
- For \(\mathbf{x}=(2,4,6)^\mathsf{T}\), explain why the usual two-parameter projection formula using \((X^\mathsf{T}X)^{-1}\) does not apply. (Answer: The predictor is constant, so \(\mathbf{x}=4\mathbf{1}\); the two columns of \(X\) are linearly dependent and \(X^\mathsf{T}X\) is singular.)