Residuals Are Orthogonal to Predictor Values

Introduction

A fitted line is chosen to balance its errors according to the least-squares rule. That choice imposes exact algebraic relationships between the predictor values and the residuals. One of these relationships is that the sum of the predictor values multiplied by their residuals is zero. It holds for the fitted sample, even when the points do not lie exactly on a line.

The word orthogonal describes this zero sum of products. It does not mean that every residual is zero, nor does it establish that the predictor and the unexplained part of the response are statistically independent. The result is a property of how the fitted line is calculated.

Setup and Definitions

Suppose we have $n$ observed pairs $(x_i,y_i)$, where $x_i$ is the predictor value and $y_i$ is the response value for observation $i$. Fit a straight line with an intercept, written $\hat{y}_i=b_0+b_1x_i$. Here, $b_0$ is the fitted intercept, $b_1$ is the fitted slope, and $\hat{y}_i$ is the fitted response at $x_i$. The residual is the observed response minus its fitted response: $e_i=y_i-\hat{y}_i$.

The line is a least-squares line: among straight lines with an intercept, it is chosen to minimize the sum of squared residuals, $\sum_{i=1}^n e_i^2$. We assume the predictor values are not all identical, so a nonzero slope denominator is available. The property below is an algebraic consequence of the least-squares fit; it does not require normally distributed errors or other probability assumptions.

Definition: The predictor values and residuals are orthogonal in the sample when their sum of products is zero: $$ \sum_{i=1}^n x_i e_i=0. $$ Equivalently, the vector of observed predictor values has zero inner product with the vector of residuals. This definition concerns the fitted data, not a claim about a population relationship.

A useful route to the result begins with the centered predictor values $x_i-\bar{x}$, where $\bar{x}$ is the sample mean of the predictor values. The least-squares slope with an intercept is

b_1=\frac{\sum_{i=1}^n (x_i-\bar{x})(y_i-\bar{y})} {\sum_{i=1}^n (x_i-\bar{x})^2},

where $\bar{y}$ is the sample mean of the responses. The denominator is positive because the predictor values are not all identical.

Why the Products Sum to Zero

Because the fitted line includes an intercept, its coefficients satisfy $b_0=\bar{y}-b_1\bar{x}$. Substituting this into the residual definition expresses each residual using centered values:

e_i=y_i-b_0-b_1x_i =(y_i-\bar{y})-b_1(x_i-\bar{x}).

Multiply each residual by its centered predictor value and sum. The first resulting sum is the numerator of the least-squares slope equation; the second is its denominator. Thus the slope equation makes the difference equal to zero:

\begin{aligned} \sum_{i=1}^n (x_i-\bar{x})e_i &=\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y}) -b_1\sum_{i=1}^n(x_i-\bar{x})^2\\ &=\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y}) -\frac{\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})} {\sum_{i=1}^n(x_i-\bar{x})^2} \sum_{i=1}^n(x_i-\bar{x})^2\\ &=0. \end{aligned}

To obtain the result for the raw predictor values $x_i$, write each one as its centered value plus the mean. The zero-sum residual property for a least-squares line with an intercept is $\sum_{i=1}^n e_i=0$. Therefore,

\sum_{i=1}^n x_i e_i =\sum_{i=1}^n\bigl((x_i-\bar{x})+\bar{x}\bigr)e_i =\sum_{i=1}^n(x_i-\bar{x})e_i+\bar{x}\sum_{i=1}^n e_i =0+\bar{x}(0)=0.

This proves the stated result. The centered version follows directly from the slope equation; the raw version follows by combining it with the zero-sum residual property.

Theorem: For a least-squares straight-line fit with an intercept, if the predictor values are not all identical, then $$ \sum_{i=1}^n (x_i-\bar{x})e_i=0 \qquad\text{and}\qquad \sum_{i=1}^n x_i e_i=0. $$ The first equality says centered predictor values and residuals have zero sum of products. The second is the residual–predictor orthogonality property in raw predictor units.

Worked Calculations

Worked Example: Checking Orthogonality for Three Observations

Suppose a hypothetical set of observations has predictor values $1,2,3$ hours of practice and responses $2,3,5$ points on a task. Fit a least-squares line with an intercept, then check the sum of predictor–residual products.

The means are $\bar{x}=2$ hours and $\bar{y}=10/3$ points. The centered cross-product sum is

\sum (x_i-\bar{x})(y_i-\bar{y}) =(-1)(-4/3)+(0)(-1/3)+(1)(5/3)=3,

and the centered predictor sum of squares is

\sum (x_i-\bar{x})^2=(-1)^2+0^2+1^2=2.

So the slope is $b_1=3/2=1.5$ points per hour, and the intercept is $b_0=\bar{y}-b_1\bar{x}=10/3-(3/2)(2)=1/3$ point. The fitted line is $\hat{y}=1/3+1.5x$. Its fitted values are $11/6,\ 10/3,\ 29/6$ points, giving residuals $1/6,\ -1/3,\ 1/6$ points.

The products of the raw predictor values and residuals are

\sum x_i e_i =(1)(1/6)+(2)(-1/3)+(3)(1/6) =1/6-2/3+1/2=0.

The residuals are not individually zero, but their predictor-weighted products cancel exactly.

Worked Example: Checking with a Different Predictor Scale

For a second hypothetical data set, let $x$ be distance in kilometres and $y$ be a score in points. The four observed pairs are $(2,10), (4,12), (6,11), (8,15)$. Find the fitted line and verify the property.

The means are $\bar{x}=5$ kilometres and $\bar{y}=12$ points. Using the centered slope formula,

b_1= \frac{(-3)(-2)+(-1)(0)+(1)(-1)+(3)(3)} {(-3)^2+(-1)^2+(1)^2+(3)^2} =\frac{6+0-1+9}{9+1+1+9} =\frac{14}{20}=0.7.

The intercept is $b_0=12-(0.7)(5)=8.5$ points, so $\hat{y}=8.5+0.7x$. The fitted values at $x=2,4,6,8$ kilometres are $9.9,11.3,12.7,14.1$ points. The residuals are $0.1,0.7,-1.7,0.9$ points.

Now multiply each predictor value by its residual and add:

\sum x_i e_i =(2)(0.1)+(4)(0.7)+(6)(-1.7)+(8)(0.9) =0.2+2.8-10.2+7.2=0.

The cancellation holds although the residuals have different magnitudes and signs. The products have units of kilometre-points; their sum is zero in those units.

Worked Example: Using Centered Products to Verify the Raw Sum

Consider a third hypothetical set of pairs, with predictor values $10,20,30$ units and responses $4,7,8$ points. This example makes the distinction between centered and raw predictor values explicit.

Here, $\bar{x}=20$ and $\bar{y}=19/3$. The centered cross-product sum is

$$ (-10)(-7/3)+(0)(2/3)+(10)(5/3) =70/3+0+50/3=40, $$

and the centered predictor sum of squares is $100+0+100=200$. Therefore $b_1=40/200=0.2$ points per predictor unit, and $b_0=19/3-(0.2)(20)=7/3$ points. The fitted responses at the three predictor values are $13/3,\ 19/3,\ 25/3$, so the residuals are $-1/3,\ 2/3,\ -1/3$ points.

The centered products sum to zero:

\sum (x_i-\bar{x})e_i =(-10)(-1/3)+(0)(2/3)+(10)(-1/3) =10/3+0-10/3=0.

For the raw products, the calculation is $\sum x_i e_i=(10)(-1/3)+(20)(2/3)+(30)(-1/3)=-10/3+40/3-30/3=0$. The mean contribution is zero because the residuals sum to $-1/3+2/3-1/3=0$.

What Orthogonality Does—and Does Not—Tell You

The zero sum is a constraint on the residuals as a group. A positive residual at a relatively large predictor value contributes a positive product; a negative residual at that value contributes a negative product. Products at other observations can balance these contributions. The theorem guarantees the total is zero, not that each contribution vanishes.

The result also explains a useful distinction: $\sum (x_i-\bar{x})e_i=0$ concerns deviations of predictor values from their mean, while $\sum x_i e_i=0$ concerns the raw values. For an intercept-including fit, the zero-sum residual property makes these statements equivalent. Centering changes each product sum by $\bar{x}\sum e_i$, which is zero here.

Orthogonality is not a claim that residuals are independent of predictor values in a broader probabilistic sense. It does not show that a straight line describes every feature of the data well, that the residuals are all small, or that changing the predictor causes a change in the response. It is an exact arithmetic property of the least-squares line fitted to these observations.

Common Mistake

Common mistake: “The products $x_i e_i$ sum to zero, so every product must be zero; therefore every residual must be zero.” The conclusion does not follow. A sum can be zero because positive and negative terms cancel. In the first worked example, the residuals are $1/6,-1/3,1/6$, yet the weighted products sum to zero. Orthogonality is a property of the total, not a statement that the fit passes through every observed point.

A related error is to interpret the identity as evidence that the predictor and residuals are independent or unrelated in every sense. The fitted data satisfy one particular sum-of-products equation because of the least-squares slope calculation. That equation alone does not establish those broader claims.

Check Your Understanding

Use the definitions and result above to answer these questions.

State the condition on the predictor values needed to use the centered least-squares slope formula.
Why does the centered sum $\sum (x_i-\bar{x})e_i$ equal zero for the fitted least-squares line?
What additional property connects the centered sum of products to $\sum x_i e_i$?
Does $\sum x_i e_i=0$ imply that every residual is zero? Explain.
Does the orthogonality property establish that the predictor and residuals are statistically independent? Why or why not?