Residuals Are Orthogonal to Fitted Values

Why Residuals and Fitted Values Matter

A least-squares regression replaces each observed response with a fitted value from the estimated line. The difference between the observed response and its fitted value is the residual. A useful property of an ordinary least-squares fit with an intercept is that the residuals have zero inner product with the fitted values. This is an exact algebraic identity for the fitted data, not an approximation and not a statement that requires normally distributed errors.

The identity is useful when checking regression calculations. It also makes clear why the intercept matters to the derivation: the intercept and slope normal equations together impose two conditions on the residuals. We will derive the result and verify it numerically.

Definition: For observed pairs \((x_i,y_i)\), \(i=1,\ldots,n\), a simple least-squares line with an intercept has fitted values \(\hat y_i=\hat\beta_0+\hat\beta_1x_i\) and residuals \(e_i=y_i-\hat y_i\). The inner product of the residual and fitted-value vectors is \(\sum_{i=1}^n e_i\hat y_i\). They are called orthogonal when this sum is zero.

The model being fit here is a straight line with an intercept, and the estimates \(\hat\beta_0,\hat\beta_1\) are chosen to minimize the sum of squared residuals. At least two distinct predictor values are needed for a unique slope and intercept. No distributional assumption about the responses or errors is needed for the identity. If a different fitting procedure is used, the least-squares normal equations below need not hold.

Deriving the Identity from the Normal Equations

For a candidate intercept \(b_0\) and slope \(b_1\), the residual sum of squares is the sum of squared differences between the observed responses and the candidate fitted values. At the least-squares estimates, this function is minimized.

Q(b_0,b_1)=\sum_{i=1}^{n}(y_i-b_0-b_1x_i)^2

Write \(e_i=y_i-\hat\beta_0-\hat\beta_1x_i\) for the residual at the minimizing estimates. Differentiating \(Q\) with respect to each coefficient and setting the derivatives equal to zero gives the normal equations:

\frac{\partial Q}{\partial b_0}=-2\sum_{i=1}^{n}(y_i-b_0-b_1x_i), \qquad \frac{\partial Q}{\partial b_1}=-2\sum_{i=1}^{n}x_i(y_i-b_0-b_1x_i)

\sum_{i=1}^{n} e_i=0, \qquad \sum_{i=1}^{n} x_i e_i=0

The first equation says the residuals sum to zero; the second says their inner product with the predictor values is zero. Since every fitted value is the intercept plus the slope times its predictor, the inner product with fitted values is a linear combination of those two sums.

Theorem: In simple ordinary least squares with an intercept, the residuals have zero inner product with the fitted values:

\begin{aligned} \sum_{i=1}^{n}e_i\hat y_i &=\sum_{i=1}^{n}e_i(\hat\beta_0+\hat\beta_1x_i)\\ &=\hat\beta_0\sum_{i=1}^{n}e_i+\hat\beta_1\sum_{i=1}^{n}x_i e_i\\ &=\hat\beta_0(0)+\hat\beta_1(0)=0. \end{aligned}

This derivation explains why the result holds: the fitted values have the form of a constant plus a multiple of the predictor, and the normal equations make the residual inner product with each of those parts zero. In a calculation, the result can be checked by multiplying each residual by its corresponding fitted value and adding the products.

Worked Calculations

Worked Example: Checking Orthogonality for Three Observations

A hypothetical analyst records a process setting \(x\) and a response \(y\) for three trials: \((0,1)\), \((1,2)\), and \((2,4)\). Fit the least-squares line with an intercept, then calculate the residual–fitted-value inner product. Use the centered-sums method for the slope.

Method. First calculate the means and centered sums. Here \(\bar x=(0+1+2)/3=1\) and \(\bar y=(1+2+4)/3=7/3\). The predictor centered sum of squares is

S_{xx}=(-1)^2+0^2+1^2=2.

The centered cross-product is

S_{xy}=(-1)(1-7/3)+0(2-7/3)+1(4-7/3) =\frac{4}{3}+0+\frac{5}{3}=3.

Therefore, \(\hat\beta_1=S_{xy}/S_{xx}=3/2\), and \(\hat\beta_0=\bar y-\hat\beta_1\bar x=7/3-3/2=5/6\). The fitted values are \(5/6\), \(7/3\), and \(23/6\). Subtracting each fitted value from its observed response gives residuals \(1/6\), \(-1/3\), and \(1/6\).

Now multiply corresponding residuals and fitted values:

\sum e_i\hat y_i =\left(\frac{1}{6}\right)\left(\frac{5}{6}\right) +\left(-\frac{1}{3}\right)\left(\frac{7}{3}\right) +\left(\frac{1}{6}\right)\left(\frac{23}{6}\right) =\frac{5}{36}-\frac{28}{36}+\frac{23}{36}=0.

As a check, the residuals sum to \(1/6-1/3+1/6=0\), and their predictor-weighted sum is \(0(1/6)+1(-1/3)+2(1/6)=0\). Both normal equations hold. Common mistake avoided: using the observed responses in place of fitted values in the inner product. The identity being checked is specifically \(\sum e_i\hat y_i=0\).

Worked Example: Verifying the Result When the Predictor Is Not Centered

A hypothetical laboratory relates a measured response to a setting recorded as \(x=1,2,3\). The responses are \(y=3,5,5\). Find the least-squares line with an intercept and check the inner product directly.

Method. The means are \(\bar x=2\) and \(\bar y=13/3\). The centered sums are \(S_{xx}=(-1)^2+0^2+1^2=2\) and

S_{xy}=(-1)(3-13/3)+0(5-13/3)+1(5-13/3) =\frac{4}{3}+\frac{2}{3}=2.

Hence \(\hat\beta_1=2/2=1\), and \(\hat\beta_0=13/3-1(2)=7/3\). The fitted values at \(x=1,2,3\) are \(10/3,13/3,16/3\). The residuals are \(3-10/3=-1/3\), \(5-13/3=2/3\), and \(5-16/3=-1/3\).

The requested inner product is

\sum e_i\hat y_i =-\frac{10}{9}+\frac{26}{9}-\frac{16}{9}=0.

Check the normal equations independently: \(\sum e_i=-1/3+2/3-1/3=0\), while \(\sum x_ie_i=1(-1/3)+2(2/3)+3(-1/3)=-1/3+4/3-1=0\). Common mistake avoided: treating a predictor that does not start at zero as a reason the identity should fail. The fitted line includes its intercept, and the normal equations apply to these original predictor values.

Worked Example: Checking Every Term in a Four-Observation Fit

A hypothetical study records \((x,y)=(-1,2),(0,1),(1,3),(2,4)\). Fit a line with an intercept and verify the residual–fitted-value inner product. This example checks both the coefficient calculation and the final sum.

Method. The means are \(\bar x=(−1+0+1+2)/4=1/2\) and \(\bar y=(2+1+3+4)/4=5/2\). The centered sums are

S_{xx}=\left(-\frac{3}{2}\right)^2+\left(-\frac{1}{2}\right)^2 +\left(\frac{1}{2}\right)^2+\left(\frac{3}{2}\right)^2=5,

S_{xy} =\left(-\frac{3}{2}\right)\left(-\frac{1}{2}\right) +\left(-\frac{1}{2}\right)\left(-\frac{3}{2}\right) +\left(\frac{1}{2}\right)\left(\frac{1}{2}\right) +\left(\frac{3}{2}\right)\left(\frac{3}{2}\right)=4.

Thus \(\hat\beta_1=4/5=0.8\), and \(\hat\beta_0=5/2-(0.8)(1/2)=2.1\). The fitted values, in the order of the data, are \(1.3,2.1,2.9,3.7\). The residuals are \(0.7,-1.1,0.1,0.3\).

The four products are \(0.7(1.3)=0.91\), \((-1.1)(2.1)=-2.31\), \(0.1(2.9)=0.29\), and \(0.3(3.7)=1.11\). Their sum is \(0.91-2.31+0.29+1.11=0\). As a further check, the residual sum is \(0.7-1.1+0.1+0.3=0\), and the predictor-weighted residual sum is \((-1)(0.7)+0(-1.1)+1(0.1)+2(0.3)=0\). Common mistake avoided: rounding the fitted values or residuals early and then interpreting a small nonzero computed sum as a failure of the identity. Here the displayed decimals are exact; in other problems, retain sufficient precision until the final check.

How to Use the Identity Correctly

For a fitted line with an intercept, the result is an unweighted sum over the observations. It does not mean that each individual product \(e_i\hat y_i\) is zero; positive and negative products may cancel. Nor does the result say that residuals are orthogonal to the observed responses. Since \(y_i=\hat y_i+e_i\), the observed-response inner product is

\sum_{i=1}^{n}e_i y_i =\sum_{i=1}^{n}e_i\hat y_i+\sum_{i=1}^{n}e_i^2 =\sum_{i=1}^{n}e_i^2.

Thus, unless all residuals are zero, \(\sum e_i y_i\) is positive rather than zero. This is a quick way to detect the common substitution of \(y_i\) for \(\hat y_i\).

When auditing a calculation, use this sequence:

Confirm the fitting setup.
Check that the line was obtained by ordinary least squares and includes an intercept.

Compute each residual consistently.
For every observation, use \(e_i=y_i-\hat y_i\), with the same observation order for \(y_i\), \(x_i\), and \(\hat y_i\).

Check both normal equations.
Calculate \(\sum e_i\) and \(\sum x_ie_i\). Each should be zero, subject only to rounding in reported values.

Check the fitted-value inner product.
Calculate \(\sum e_i\hat y_i\). It should be zero, with any small discrepancy explained by rounding rather than by a different target sum.

The identity is a computational property of the least-squares fit, not evidence that the regression line is a good description of the data. A poor linear fit still satisfies the normal equations when it is calculated correctly. Conversely, a noticeably nonzero inner product using unrounded values can indicate an arithmetic error, inconsistent residuals, or a fit that was not obtained by the stated least-squares method. This algebraic check is aligned with the SOA VEE Mathematical Statistics syllabus.

Check Your Understanding

For each problem, distinguish the two normal-equation checks from the final residual–fitted-value inner product.

A least-squares line with an intercept is fit to \((0,2),(1,3),(2,5)\). What is \(\sum e_i\hat y_i\), regardless of the individual fitted values? (Answer: \(0\))
For an intercept fit, suppose \(\sum e_i=0\), \(\sum x_ie_i=0\), \(\hat\beta_0=4\), and \(\hat\beta_1=-2\). Evaluate \(\sum e_i\hat y_i\) using the fitted-value equation. (Answer: \(0\))
For \(x=1,2,4\) and \(y=3,4,8\), the fitted line with an intercept is \(\hat y=1+(12/7)x\). The residuals in that order are \(2/7,-3/7,1/7\). Verify the inner product. (Answer: \((38-93+55)/49=0\))
An intercept fit has \(\sum e_i=0\) and \(\sum x_ie_i=0\). Its fitted values are \(\hat y_i=3+0.5x_i\). Express \(\sum e_i\hat y_i\) in terms of the two normal-equation sums, then evaluate it. (Answer: \(3\sum e_i+0.5\sum x_ie_i=0\))
A fit has fitted values \(3,5,7\) and residuals \(-1/2,1,-1/2\), in matching order. Find both \(\sum e_i\hat y_i\) and \(\sum e_i y_i\), where \(y_i=\hat y_i+e_i\). (Answer: \(\sum e_i\hat y_i=0\) and \(\sum e_i y_i=3/2\))

What You'll Learn