Residual Orthogonality to Predictor Values

Why Predictor-Weighted Residuals Cancel

A least-squares line is selected to minimize the sum of squared residuals. Its residuals are therefore not arbitrary: their values satisfy equations that follow from the minimization. One important consequence is that the residuals, when multiplied by their corresponding predictor values and added, sum to zero. This identity is useful for checking a fitted line and for understanding the centered-data calculations used in regression.

The result concerns a simple linear least-squares fit that includes both an intercept and a slope. It is an exact algebraic property of the fitted data, not a claim that each residual is zero or that the residuals are independent of the predictor in every possible sense.

Definition: For observations \((x_i,y_i)\), \(i=1,\ldots,n\), a fitted simple linear regression line has fitted values \(\hat y_i=a+bx_i\). The residual for observation \(i\) is \(e_i=y_i-\hat y_i\). The least-squares estimates \(a\) and \(b\) minimize \(Q(a,b)=\sum_{i=1}^{n}[y_i-(a+bx_i)]^2\).

Here, \(x_i\) and \(y_i\) are the observed predictor and response values, while \(a\) and \(b\) are the fitted intercept and slope. No probability distribution or assumption about the errors is needed for the identities below. To have a unique slope estimate, the observed predictor values must not all be equal.

Deriving the Slope Normal Equation

The slope normal equation follows directly from minimizing the sum of squared residuals. At a least-squares minimum, the partial derivative of \(Q(a,b)\) with respect to \(b\) is zero. Since \(e_i=y_i-a-bx_i\), differentiating each squared residual gives:

\frac{\partial Q}{\partial b} =\sum_{i=1}^{n}2e_i(-x_i) =-2\sum_{i=1}^{n}x_i e_i.

Setting this derivative equal to zero gives the slope normal equation. The intercept normal equation follows in the same way by differentiating with respect to \(a\). Together, they are:

\sum_{i=1}^{n}x_i e_i=0 \qquad\text{and}\qquad \sum_{i=1}^{n}e_i=0.

The first equation is the focus here: at the least-squares fit, positive and negative residual contributions cancel after each residual is weighted by its predictor value. The second equation is the residual-sum identity for a fit with an intercept. The centered-data identity follows by combining these two equations.

Theorem: In a simple least-squares regression fit that includes an intercept and a slope, the residuals satisfy \(\sum_{i=1}^{n}x_i e_i=0\). Because the intercept equation also gives \(\sum_{i=1}^{n}e_i=0\), the centered-data identity \(\sum_{i=1}^{n}(x_i-\bar x)e_i=0\) holds, where \(\bar x=\frac{1}{n}\sum_{i=1}^{n}x_i\).

Why the Centered-Data Identity Follows

Centering a predictor means subtracting its sample mean from every observed predictor value. The centered-data identity is an immediate algebraic consequence of the two normal equations:

\sum_{i=1}^{n}(x_i-\bar x)e_i =\sum_{i=1}^{n}x_i e_i-\bar x\sum_{i=1}^{n}e_i =0-\bar x(0) =0.

Thus, the weighted residual sum is zero both when the weights are the original predictor values and when they are the centered predictor values. The centered version is helpful because it describes cancellation around the mean predictor value: residuals paired with predictor values above and below \(\bar x\) balance in the weighted sum.

This conclusion depends on fitting the intercept as well as the slope. The intercept equation supplies \(\sum e_i=0\), which is needed to move from the uncentered form to the centered form. A calculation involving an arbitrary line, rather than a least-squares fit, does not in general satisfy these equations.

Worked Example: Checking Both Forms of the Identity

A hypothetical service team records the number of completed inspections \(x\) and the minutes required for a final review \(y\) for three work batches. The data are \((1,2)\), \((2,3)\), and \((3,5)\). Find the least-squares line, calculate its residuals, and verify both residual identities.

Method. Calculate the slope from centered sums, then obtain the intercept from \(\bar y=a+b\bar x\). This method avoids trying to infer the identity from rounded fitted values.

The sample means are \(\bar x=(1+2+3)/3=2\) and \(\bar y=(2+3+5)/3=10/3\). The centered predictor sum of squares is \(S_{xx}=(-1)^2+0^2+1^2=2\). The centered cross-product sum is \(S_{xy}=(-1)(-4/3)+0(-1/3)+1(5/3)=4/3+5/3=3\). Therefore \(b=S_{xy}/S_{xx}=3/2\), and \(a=\bar y-b\bar x=10/3-(3/2)(2)=1/3\).

The fitted line is \(\hat y=1/3+(3/2)x\). At \(x=1\), the fitted value is \(11/6\) and the residual is \(2-11/6=1/6\). At \(x=2\), the fitted value is \(10/3\) and the residual is \(3-10/3=-1/3\). At \(x=3\), the fitted value is \(29/6\) and the residual is \(5-29/6=1/6\).

The predictor-weighted residual sum is \(1(1/6)+2(-1/3)+3(1/6)=1/6-4/6+3/6=0\). The residual sum is \(1/6-1/3+1/6=0\), as required by the intercept equation. Since \(\bar x=2\), the centered weighted sum is \((-1)(1/6)+0(-1/3)+1(1/6)=0\).

Check. Both identities hold using the unrounded residuals. A common mistake is to calculate with rounded fitted values and then interpret a small nonzero total as a failure of the identity; the exact fractions show why calculations should retain precision until the final check.

Using the Identity as a Calculation Check

The weighted residual sum is a useful audit of a regression calculation. Once the fitted line and residuals have been computed, multiply each residual by the predictor from the same observation and add the products. A correct least-squares calculation should give zero, apart from any small discrepancy caused by rounding.

This check is distinct from merely confirming that the residuals sum to zero. The unweighted sum tests the intercept normal equation; the predictor-weighted sum tests the slope normal equation. Either check can reveal an arithmetic error that the other does not.

Worked Example: Checking a Fit with Nonnegative Predictors

For a hypothetical equipment calibration, a technician records predictor settings \(x=0,2,4\) and responses \(y=3,5,8\). Fit the least-squares line and check the predictor-weighted residual identity.

Method. Use the centered-sum slope formula and then compute residuals observation by observation. This makes the slope equation visible in the final audit.

The means are \(\bar x=(0+2+4)/3=2\) and \(\bar y=(3+5+8)/3=16/3\). The centered predictor values are \(-2,0,2\), so \(S_{xx}=(-2)^2+0^2+2^2=8\). The centered response values are \(-7/3,-1/3,8/3\), giving \(S_{xy}=(-2)(-7/3)+0(-1/3)+2(8/3)=14/3+16/3=10\). Thus \(b=10/8=5/4\), and \(a=16/3-(5/4)(2)=16/3-5/2=17/6\).

The fitted line is \(\hat y=17/6+(5/4)x\). The fitted values at \(x=0,2,4\) are \(17/6,16/3,47/6\). Subtracting these from the observed responses gives residuals \(1/6,-1/3,1/6\), respectively.

The weighted sum is \(0(1/6)+2(-1/3)+4(1/6)=0-2/3+2/3=0\). The centered sum is \((-2)(1/6)+0(-1/3)+2(1/6)=-1/3+1/3=0\). The residuals also sum to \(1/6-1/3+1/6=0\).

Check. The intercept, slope, and all three residuals give both normal-equation checks. The common mistake avoided here is to check only the unweighted residual sum and assume that this alone verifies the slope calculation; the weighted check is separate.

What the Identity Does—and Does Not—Say

A zero weighted sum is a statement about the complete collection of fitted residuals. It does not say that every residual is zero, nor does it say that each residual is individually unrelated to its predictor. Rather, the positive and negative products \(x_i e_i\) cancel in the sum.

It is also important not to apply the identity to residuals from any line. For example, consider predictor values \(0,2,4\) and a proposed residual list \(1,-3,2\). The residuals sum to \(1-3+2=0\), but the predictor-weighted sum is \(0(1)+2(-3)+4(2)=-6+8=2\), not zero. These residuals could not be the residuals of a least-squares line with an intercept for those predictor values: they pass the unweighted check but fail the slope normal equation.

Worked Example: Centering Predictors Below and Above Zero

A hypothetical laboratory compares an instrument setting \(x\) with a response \(y\). The observations are \((-1,4)\), \((1,1)\), and \((3,5)\). Find the least-squares line and verify the centered-data identity, including when some predictor values are negative.

Method. Compute centered sums to find the slope, then calculate the fitted values and residuals. The common mistake to avoid is treating negative predictor values as if their weighted contributions should be omitted or made positive.

The means are \(\bar x=(-1+1+3)/3=1\) and \(\bar y=(4+1+5)/3=10/3\). The centered predictor values are \(-2,0,2\), so \(S_{xx}=4+0+4=8\). The centered response values are \(2/3,-7/3,5/3\). Hence \(S_{xy}=(-2)(2/3)+0(-7/3)+2(5/3)=-4/3+10/3=2\), and \(b=2/8=1/4\). The intercept is \(a=10/3-(1/4)(1)=37/12\).

The fitted line is \(\hat y=37/12+(1/4)x\). At \(x=-1\), the fitted value is \(34/12=17/6\), so \(e=4-17/6=7/6\). At \(x=1\), the fitted value is \(40/12=10/3\), so \(e=1-10/3=-7/3\). At \(x=3\), the fitted value is \(46/12=23/6\), so \(e=5-23/6=7/6\).

The uncentered weighted sum is \((-1)(7/6)+(1)(-7/3)+(3)(7/6)=-7/6-14/6+21/6=0\). Because \(\bar x=1\), the centered weighted sum is \((-2)(7/6)+0(-7/3)+2(7/6)=-14/6+14/6=0\). The residual sum is \(7/6-14/6+7/6=0\).

Check. The negative predictor contributes \(-7/6\) to the uncentered sum, exactly as multiplication requires. Keeping its sign gives zero for both forms of the identity.

Common Mistakes and Scope

Confusing the two equations. The intercept equation is \(\sum e_i=0\); the slope equation is \(\sum x_i e_i=0\). They check different features of the fitted line.
Assuming a zero residual sum proves the weighted identity. The residual list \(1,-3,2\) for \(x=0,2,4\) sums to zero but has weighted sum \(2\), so the slope condition fails.
Using rounded residuals too early. Rounding each fitted value can leave a small nonzero computed sum. Keep exact fractions or adequate precision during the calculation and round only for reporting.
Forgetting the intercept condition in the centered form. The step from \(\sum x_i e_i=0\) to \(\sum(x_i-\bar x)e_i=0\) uses \(\sum e_i=0\), which comes from fitting an intercept.

Key takeaway: For a simple least-squares line fitted with an intercept, the normal equations give \(\sum x_i e_i=0\) and \(\sum e_i=0\). Subtracting \(\bar x\) from each predictor therefore preserves the zero weighted sum: \(\sum(x_i-\bar x)e_i=0\). These are exact checks on the fitted residuals, not properties of arbitrary candidate lines.

Check Your Understanding

For each problem, use exact arithmetic where practical and check the relevant residual identity.

A hypothetical sensor study has data \((0,1),(2,4),(4,6)\). Find the least-squares line and the predictor-weighted residual sum. (Answer: \(\hat y=7/6+(5/4)x\); \(\sum x_i e_i=0\).)
A fitted line for observations with \(x=-2,0,2\) has residuals \(1/3,-2/3,1/3\). Verify the residual sum and both weighted sums. (Answer: \(\sum e_i=0\), \(\sum x_i e_i=0\), and since \(\bar x=0\), \(\sum(x_i-\bar x)e_i=0\).)
For predictor values \(1,3,5\), a proposed residual list is \(2,-5,3\). Does the list satisfy the intercept check? Does it satisfy the slope check? (Answer: The residual sum is \(2-5+3=0\), but the weighted sum is \(1(2)+3(-5)+5(3)=2\), so it fails the slope check.)
For data \((2,3),(4,7),(6,8)\), find the least-squares line and verify the centered-data identity. (Answer: \(\hat y=1+(5/4)x\); the residuals are \(-1/2,1,-1/2\), and \(\sum(x_i-\bar x)e_i=(-2)(-1/2)+0(1)+2(-1/2)=0\).)
Suppose a simple least-squares fit includes an intercept and has \(\sum x_i e_i=0\) and \(\sum e_i=0\). Show the centered-data identity in one line. (Answer: \(\sum(x_i-\bar x)e_i=\sum x_i e_i-\bar x\sum e_i=0-\bar x(0)=0\).)

What You'll Learn