Why Residual Geometry Matters
A residual is the observed response minus its fitted value. The earlier material on regression diagnostics introduced residuals as quantities used to examine model adequacy. Their geometry adds an important qualification: ordinary least squares (OLS) forces some relationships between residuals, predictors, and fitted values to be exactly zero in the observed sample. Those zero relationships are properties of the fitting procedure, not proof that the regression assumptions are satisfied.
This distinction matters when reading diagnostic plots. For example, residuals can have zero sample linear association with the fitted values and still display a clear curved pattern. The zero association is built into OLS; the shape of the residuals may still reveal structure that the fitted linear model has not captured.
The Normal Equations and Orthogonality
The model used for fitting is \(\mathbf{y}=X\boldsymbol{\beta}+\boldsymbol{\varepsilon}\), where \(X\) contains the included predictors and, if specified, a column of ones for the intercept. The OLS calculation is a deterministic minimization for the observed \(X\) and \(\mathbf{y}\); the geometric identities below do not require normally distributed errors. Assume first that \(X\) has full column rank, so \(X^\mathsf{T}X\) is invertible.
For any coefficient vector \(\boldsymbol{\beta}\), the residual sum of squares is \((\mathbf{y}-X\boldsymbol{\beta})^\mathsf{T}(\mathbf{y}-X\boldsymbol{\beta})\). At its minimum, the derivative with respect to each coefficient is zero. This gives the normal equations:
Since \(\mathbf{e}=\mathbf{y}-X\hat{\boldsymbol{\beta}}\), the normal equations state that \(X^\mathsf{T}\mathbf{e}=\mathbf{0}\). Each entry of this vector is the inner product of the residuals with one column of \(X\). Thus, residuals are orthogonal to every included predictor column and to the intercept column if one is included.
The fitted vector is \(\hat{\mathbf{y}}=X\hat{\boldsymbol{\beta}}\), so it is a linear combination of the columns of \(X\). Because the residual vector is orthogonal to each of those columns, it is also orthogonal to their linear combination, the fitted vector. In matrix form, the projection matrix is \(P=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\), with \(\hat{\mathbf{y}}=P\mathbf{y}\) and \(\mathbf{e}=(I-P)\mathbf{y}\). The fitted values are the projection of the observed responses onto the column space of \(X\); the residuals are the part perpendicular to that space.
Consequences When an Intercept Is Included
With an intercept, the normal equation for the column of ones is \(\mathbf{1}^\mathsf{T}\mathbf{e}=0\), or \(\sum_i e_i=0\). Equivalently, the sample mean of the residuals is zero. The normal equation for an included predictor \(x_j\) is \(\sum_i x_{ij}e_i=0\). Since the residual mean is zero, subtracting the predictor mean does not change this inner product:
Therefore, the sample covariance between the residuals and each included predictor is zero. Similarly, fitted values are linear combinations of included predictor columns, so \(\sum_i \hat{y}_i e_i=0\). With an intercept, the residual mean is zero as well, making their sample covariance with fitted values zero. If both variables have nonzero sample variance, their sample Pearson correlation is consequently zero.
These statements concern linear inner products in the observed sample. They do not say that residuals are independent of predictors, that residuals have constant variance, or that the mean response is correctly described by the included terms. Nor do they force residuals to be orthogonal to an omitted variable or to a nonlinear transformation that is not a column of \(X\).
Worked Example: Checking Simple-Regression Residual Geometry
A hypothetical portfolio team records a response \(y\) at three predictor values \(x=0,1,2\). The observed responses are \(1,2,4\), respectively. Fit an OLS line with an intercept and verify the residual constraints. No probability distribution is needed for this geometric calculation.
Method. For simple OLS with an intercept, the slope is \(S_{xy}/S_{xx}\), and the intercept is \(\bar{y}-\hat{\beta}_1\bar{x}\). First, \(\bar{x}=(0+1+2)/3=1\) and \(\bar{y}=(1+2+4)/3=7/3\). Then \(S_{xx}=(-1)^2+0^2+1^2=2\), and \(S_{xy}=(-1)(1-7/3)+0(2-7/3)+1(4-7/3)=4/3+0+5/3=3\).
Thus \(\hat{\beta}_1=3/2\) and \(\hat{\beta}_0=7/3-(3/2)(1)=5/6\). The fitted line is \(\hat{y}=5/6+(3/2)x\). Its fitted values are \(5/6, 7/3, 23/6\), so the residuals are \(1/6,-1/3,1/6\).
Check the constraints. Their sum is \(1/6-1/3+1/6=0\). Their inner product with \(x\) is \(0(1/6)+1(-1/3)+2(1/6)=0\). Their inner product with the fitted values is \((5/6)(1/6)+(7/3)(-1/3)+(23/6)(1/6)=5/36-7/9+23/36=0\).
Check the sums of squares. The residual sum of squares is \(1/36+1/9+1/36=1/6\). The total sum of squares is \((1-7/3)^2+(2-7/3)^2+(4-7/3)^2=14/3\). The regression sum of squares is \((5/6-7/3)^2+(7/3-7/3)^2+(23/6-7/3)^2=9/2\), and \(9/2+1/6=14/3\).
The arithmetic verifies both the normal-equation constraints and the sums-of-squares decomposition. The common mistake to avoid is checking only that residuals sum to zero: that verifies the intercept equation but not the predictor or fitted-value equations.
Why the Sums of Squares Decompose
With an intercept, let \(\bar{y}\) be the sample response mean. The vector of deviations from that mean can be written as \(\mathbf{y}-\bar{y}\mathbf{1}=(\hat{\mathbf{y}}-\bar{y}\mathbf{1})+\mathbf{e}\). The fitted deviations belong to the column space of \(X\), because both \(\hat{\mathbf{y}}\) and \(\bar{y}\mathbf{1}\) belong to it. The residuals are orthogonal to that space. Squaring the length of this vector sum therefore introduces no cross-product term:
These are the total, regression, and error sums of squares, respectively, under the usual intercept-model definitions. The decomposition is a geometric consequence of orthogonality. It is not an additional assumption about the error distribution.
Worked Example: Multiple Predictors and a Valid OLS Fit
Consider four observations with predictor pairs \((x_1,x_2)=(-1,-1),(-1,1),(1,-1),(1,1)\) and responses \(10,6,12,12\). Fit the stated candidate equation \(\hat{y}=10+2x_1-x_2\), then check whether its residuals satisfy the OLS normal equations for a model with an intercept and both predictors.
Method. Evaluate the equation at each pair. The fitted values are \(10+2(-1)-(-1)=9\), \(10+2(-1)-1=7\), \(10+2(1)-(-1)=13\), and \(10+2(1)-1=11\). Subtracting fitted values from observed responses gives residuals \(1,-1,-1,1\).
For the intercept column, the residual inner product is \(1-1-1+1=0\). For \(x_1\), it is \((-1)(1)+(-1)(-1)+(1)(-1)+(1)(1)=-1+1-1+1=0\). For \(x_2\), it is \((-1)(1)+(1)(-1)+(-1)(-1)+(1)(1)=-1-1+1+1=0\). The fit is therefore the OLS fit: its residuals satisfy all normal equations, and the design columns are linearly independent.
The fitted-value inner product is \(9(1)+7(-1)+13(-1)+11(1)=9-7-13+11=0\). The residual sum of squares is \(1^2+(-1)^2+(-1)^2+1^2=4\). The common mistake this calculation avoids is evaluating a fitted equation incorrectly and then using the incorrect values to claim orthogonality; each prediction must be checked against its predictor pair.
What Orthogonality Does—and Does Not—Show in a Diagnostic Plot
A residual-versus-fitted plot places \(\hat{y}_i\) on the horizontal axis and \(e_i\) on the vertical axis. In an intercept model, the plotted residuals have zero sample linear association with the fitted values by construction. That fact alone does not mean the plotted points have no structure. A bend, grouping, or other systematic arrangement can coexist with zero covariance because positive and negative contributions can cancel in the overall linear sum.
The same caution applies to residuals versus an included predictor: the sample linear association is zero, but a nonlinear relationship can remain. Residuals also need not be orthogonal to an omitted predictor or to a transformation such as \(x^2\) unless that term is included in the design matrix. Diagnostic interpretation must therefore consider the pattern of points, not simply whether a linear correlation is zero.
Worked Example: A Curved Relationship Despite Zero Linear Association
For predictor values \(x=0,1,2,3\), suppose the observed responses are \(y=0,1,4,9\). Fit a straight line with an intercept and examine whether its residuals reveal structure, even though they are orthogonal to the included predictor and fitted values.
Method. The means are \(\bar{x}=3/2\) and \(\bar{y}=7/2\). The centered sums are \(S_{xx}=(-3/2)^2+(-1/2)^2+(1/2)^2+(3/2)^2=5\) and \(S_{xy}=(-3/2)(-7/2)+(-1/2)(-5/2)+(1/2)(1/2)+(3/2)(11/2)=15\). Thus the slope is \(15/5=3\), and the intercept is \(7/2-3(3/2)=-1\). The fitted line is \(\hat{y}=-1+3x\), with fitted values \(-1,2,5,8\).
The residuals are \(1,-1,-1,1\). They sum to zero, and their inner product with \(x\) is \(0(1)+1(-1)+2(-1)+3(1)=0\). Their inner product with fitted values is \((-1)(1)+2(-1)+5(-1)+8(1)=0\). Nevertheless, in order of increasing \(x\), the residuals go positive, negative, negative, positive: the pattern is curved rather than randomly scattered around zero. Also, their inner product with the omitted \(x^2\) values is \(0(1)+1(-1)+4(-1)+9(1)=4\), not zero.
The correct conclusion is not “zero correlation, so the line is adequate.” OLS forces the zero linear inner products, while the residual pattern and its association with an omitted nonlinear term show that those identities do not rule out curvature. This example avoids the common mistake of treating a forced sample correlation as a complete diagnostic.
A No-Intercept Caution
The zero-sum property depends on including an intercept. In a no-intercept regression, the normal equations still make residuals orthogonal to the included predictor columns, and hence to fitted values, but there is no intercept column to force their sum to zero. For example, with \(x=(1,2,3)\), \(y=(1,2,2)\), and a model \(\hat{y}=bx\), the OLS slope is \(b=(1\cdot1+2\cdot2+3\cdot2)/(1^2+2^2+3^2)=11/14\). The fitted values are \(11/14,22/14,33/14\), and the residuals are \(3/14,6/14,-5/14\). Their sum is \(4/14=2/7\), not zero, while their inner product with \(x\) is \(3/14+12/14-15/14=0\). The incorrect calculation “OLS residuals always sum to zero” confuses a property of an intercept model with a property of all OLS models.
Check Your Understanding
For each problem, use the stated model form and verify which orthogonality constraints apply.
- For \(x=(0,1,2)\) and \(y=(2,3,6)\), fit a line with an intercept. Give the fitted values and residuals, and check the residual sum and the inner product with \(x\). (Answer: \(\hat{y}=5/3+2x\); fitted values \(5/3,11/3,17/3\); residuals \(1/3,-2/3,1/3\); both the residual sum and \(x^\mathsf{T}\mathbf{e}\) are \(0\).)
- For \(x=(-2,-1,1,2)\) and \(y=(4,1,2,5)\), fit a line with an intercept and verify the residual constraints. (Answer: \(\hat{y}=3+(3/10)x\); fitted values \(12/5,27/10,33/10,18/5\); residuals \(8/5,-17/10,-13/10,7/5\); their sum and inner product with \(x\) are both \(0\).)
- A model with an intercept has fitted values \((2,4,6,8)\) and residuals \((1,-1,-1,1)\), in that observation order. Find the observed responses and check the residual sum and fitted-value inner product. Can the residuals still show a pattern? (Answer: responses \((3,3,5,9)\); residual sum \(0\); fitted-value inner product \(2-4-6+8=0\); yes, the ordered residuals show a positive, negative, negative, positive pattern despite those identities.)
- For a no-intercept model with \(x=(1,2,3)\) and \(y=(1,2,2)\), calculate the fitted values and residual sum. Which orthogonality condition still holds? (Answer: slope \(11/14\); fitted values \((11/14,22/14,33/14)\); residuals \((3/14,6/14,-5/14)\), whose sum is \(2/7\); the residual inner product with \(x\), and therefore with fitted values, is \(0\).)