Residual Orthogonality to the Predictor

Why Residual Orthogonality Matters

After fitting a least-squares regression line with an intercept, the residuals have a precise relationship with the observed predictor values: their sample covariance is zero. This is not an assumption about how the data were generated. It is a consequence of the way the line is chosen. Knowing why it holds helps distinguish a property built into the fitting procedure from evidence that a model is appropriate.

Let $x_1,\ldots,x_n$ be observed predictor values and $y_1,\ldots,y_n$ the corresponding responses. A simple linear regression fit with an intercept produces fitted values $\hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_i$. The residual for observation $i$ is $e_i=y_i-\hat{y}_i$. Least squares chooses the two coefficients to minimize the sum of squared residuals.

Definition: For a fitted least-squares regression with an intercept, residuals are orthogonal to the predictor values when their centered cross-product is zero: $$ \sum_{i=1}^{n}(x_i-\bar{x})(e_i-\bar{e})=0. $$ The least-squares fit also has $\bar{e}=0$. Consequently, the centered cross-product can be written as $\sum_{i=1}^{n}(x_i-\bar{x})e_i$, and it is zero. If the predictor and residuals both have positive sample variance, this is equivalent to zero sample correlation.

Here, “orthogonal” is an algebraic description of the observed vectors: after centering, their dot product is zero. It does not say that predictor values and residuals are independent random variables. Residuals are calculated from the same data used to fit the line, and the zero cross-product is enforced by the fitting procedure.

Why Least Squares Guarantees the Result

For fixed observed predictor values, define the residual sum of squares as a function of the two coefficients:

Q(\beta_0,\beta_1)=\sum_{i=1}^{n}(y_i-\beta_0-\beta_1x_i)^2.

At a least-squares minimum, the partial derivatives with respect to both coefficients are zero. Differentiating with respect to the intercept gives the condition that the residuals sum to zero. Differentiating with respect to the slope gives the condition that their cross-product with the predictor values is zero. Together, these conditions establish orthogonality to the centered predictor.

\frac{\partial Q}{\partial \beta_0}=-2\sum_{i=1}^{n}e_i=0 \quad\Longrightarrow\quad \sum_{i=1}^{n}e_i=0, $$ $$ \frac{\partial Q}{\partial \beta_1}=-2\sum_{i=1}^{n}x_ie_i=0 \quad\Longrightarrow\quad \sum_{i=1}^{n}x_ie_i=0.

Since the residuals sum to zero, $\sum (x_i-\bar{x})e_i=\sum x_ie_i-\bar{x}\sum e_i=\sum x_ie_i=0$. Also, subtracting $\bar{e}$ from each residual does not change this centered cross-product because $\sum(x_i-\bar{x})=0$. Thus it equals the numerator of the sample covariance, which is zero.

Define the objective.
For each candidate line, calculate the sum of squared vertical residuals.

Use both fitted coefficients.
The derivative with respect to the intercept makes the residuals sum to zero; the derivative with respect to the slope makes $\sum x_ie_i=0$.

Center the predictor.
Because the residual sum is zero, centering $x_i$ leaves the cross-product at zero.

This argument requires a least-squares fit that includes an intercept. For a nonconstant predictor, the usual simple-regression slope and intercept are uniquely determined. No probability distribution for the errors, and no assumptions of normality or constant variance, are needed for this algebraic result.

Worked Example: Checking Orthogonality From a Small Data Set

A hypothetical analyst relates a device’s weekly usage $y$, in hours, to its age $x$, in years. The three observations are $(1,2)$, $(2,3)$, and $(3,5)$. Fit the least-squares line with an intercept and verify the residual cross-product.

Method and coefficient calculation. First calculate the means: $\bar{x}=(1+2+3)/3=2$ and $\bar{y}=(2+3+5)/3=10/3$. The centered predictor sum of squares is $S_{xx}=(-1)^2+0^2+1^2=2$. The centered cross-product of predictor and response is $S_{xy}=(-1)(2-10/3)+0(3-10/3)+1(5-10/3)=4/3+0+5/3=3$. Therefore, $\hat{\beta}_1=S_{xy}/S_{xx}=3/2$, and $\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}=10/3-3=1/3$.

Fitted values and residuals. At $x=1$, the fitted value is $1/3+3/2=11/6$, so $e_1=2-11/6=1/6$. At $x=2$, the fitted value is $1/3+3=10/3$, so $e_2=3-10/3=-1/3$. At $x=3$, the fitted value is $1/3+9/2=29/6$, so $e_3=5-29/6=1/6$.

Verification. The residual sum is $1/6-1/3+1/6=0$. The predictor-residual cross-product is $1(1/6)+2(-1/3)+3(1/6)=1/6-2/3+1/2=0$. Equivalently, the centered cross-product is $(-1)(1/6)+0(-1/3)+1(1/6)=0$. The calculation checks the algebraic property exactly. The common mistake to avoid is checking only that the residuals sum to zero: that condition alone does not demonstrate orthogonality to the predictor.

What Zero Sample Covariance Means

When the predictor and residuals both vary, the sample covariance is their centered cross-product divided by $n-1$. Thus, under the conditions above, it is zero. Dividing that covariance by the product of the sample standard deviations gives a zero sample correlation. If all residuals are zero, however, their sample standard deviation is zero and the sample correlation is undefined; the centered cross-product is still zero.

The result concerns a particular linear association in the fitted sample. It does not imply that every pattern in a residual plot disappears, that the residuals are independent of the predictor, or that the regression relationship is substantively useful. For example, a nonlinear pattern among residuals can remain even though their sample covariance with $x$ is exactly zero. Orthogonality is guaranteed by fitting, so it is not by itself a diagnostic test for whether the line describes the data well.

Worked Example: From Orthogonality to Sample Covariance

A hypothetical pricing team fits a simple least-squares line with an intercept to four observations. The observed predictors are $x=(0,1,2,3)$, and the responses are $y=(1,2,2,5)$. Calculate the fit, then verify the centered residual cross-product and the sample covariance.

Calculate the line. The means are $\bar{x}=(0+1+2+3)/4=3/2$ and $\bar{y}=(1+2+2+5)/4=5/2$. The centered predictor values are $-3/2,-1/2,1/2,3/2$, so $S_{xx}=9/4+1/4+1/4+9/4=5$. The centered response values are $-3/2,-1/2,-1/2,5/2$, giving $S_{xy}=9/4+1/4-1/4+15/4=6$. It follows that $\hat{\beta}_1=6/5$ and $\hat{\beta}_0=5/2-(6/5)(3/2)=5/2-9/5=7/10$.

Calculate the residuals. The fitted values at $x=0,1,2,3$ are $7/10,19/10,31/10,43/10$. Subtracting these from the observed responses gives $e=(3/10,1/10,-11/10,7/10)$. Their sum is $(3+1-11+7)/10=0$.

Check the covariance. The centered cross-product is $(-3/2)(3/10)+(-1/2)(1/10)+(1/2)(-11/10)+(3/2)(7/10)$ $=-9/20-1/20-11/20+21/20=0$. The sample covariance is this sum divided by $n-1=3$, so it is $0/3=0$. This confirms that the residuals have zero sample covariance with the predictor. The common mistake to avoid is dividing by $n$ instead of $n-1$ and treating the definition of sample covariance as the reason it is zero; either divisor gives zero here, but the sample covariance convention uses $n-1$, while the reason the cross-product vanishes is the least-squares fit.

The Role of the Intercept

The intercept matters because it adds the condition $\sum e_i=0$. Without that condition, a slope fitted through the origin can still make $\sum x_ie_i=0$, but this does not generally make the usual centered sample covariance zero. The following example isolates this distinction.

Worked Example: A Fit Through the Origin Is Different

Suppose a hypothetical analyst deliberately fits a line through the origin to the observations $(1,2)$, $(2,2)$, and $(3,4)$. Calculate its residuals and determine whether their sample covariance with the predictor is zero.

Fit the restricted line. For a line $\hat{y}_i=\hat{\beta}_1x_i$, least squares gives $\hat{\beta}_1=\sum x_iy_i/\sum x_i^2$. Here, $\sum x_iy_i=1(2)+2(2)+3(4)=18$, and $\sum x_i^2=1+4+9=14$. Thus, $\hat{\beta}_1=18/14=9/7$.

Find the residuals. The fitted values are $9/7,18/7,27/7$. The residuals are $2-9/7=5/7$, $2-18/7=-4/7$, and $4-27/7=1/7$. Their sum is $(5-4+1)/7=2/7$, not zero. The slope condition does hold: $\sum x_ie_i=1(5/7)+2(-4/7)+3(1/7)=(5-8+3)/7=0$.

Calculate the sample covariance. Here $\bar{x}=2$ and $\bar{e}=(2/7)/3=2/21$. The centered cross-product equals $\sum(x_i-\bar{x})(e_i-\bar{e})=\sum(x_i-\bar{x})e_i$, because the centered predictor values sum to zero. Its value is $(-1)(5/7)+0(-4/7)+1(1/7)=-4/7$. Therefore, the sample covariance is $(-4/7)/(3-1)=-2/7$, not zero. The common mistake is to treat the through-origin condition $\sum x_ie_i=0$ as if it also implied zero sample covariance. It does not when the residuals have a nonzero mean.

Common Interpretations to Avoid

“The residuals are independent of the predictor.” Zero sample covariance does not establish independence. It records a particular relationship in the fitted data.
“The fit must be good because the residual covariance is zero.” The property follows from least squares with an intercept, even when the data have other systematic patterns.
“The residuals are uncorrelated with the unobserved errors.” Residuals are calculated from observed responses and fitted values; the orthogonality result here concerns residuals and observed predictor values.
“The result holds for every regression fit.” The zero sample covariance conclusion relies on an intercept. A fit constrained through the origin need not have it.
“Zero correlation is always defined.” If the residuals have no variation, their standard deviation is zero, so correlation is undefined even though the cross-product is zero.

Key takeaway: In ordinary least squares with an intercept, minimizing the residual sum of squares forces $\sum e_i=0$ and $\sum x_ie_i=0$. These conditions make the sample covariance between the predictor values and residuals zero. This is an algebraic feature of the fit, not proof of independence or model adequacy.

Check Your Understanding

For each problem, focus on what the least-squares conditions guarantee and on the role of the intercept.

A least-squares line with an intercept is fitted to observations with a nonconstant predictor. If the residuals sum to zero and $\sum x_ie_i=0$, what is $\sum(x_i-\bar{x})e_i$? (Answer: 0)
A fitted line includes an intercept, and its residuals are $1,-2,1$, paired with predictor values $1,2,3$. Calculate the residual sum and the predictor-residual cross-product. (Answer: Residual sum $=0$; cross-product $=1-4+3=0$)
A least-squares fit with an intercept has nonconstant residuals and a nonconstant predictor. What are the sample covariance and sample correlation between the predictor and residuals? (Answer: Both are 0)
A line through the origin has residuals $5/7,-4/7,1/7$ at predictor values $1,2,3$. Given that $\bar{x}=2$, calculate their sample covariance. (Answer: $-2/7$)
A least-squares fit with an intercept has every residual equal to zero. Is the sample correlation between predictor and residuals zero or undefined, and why? (Answer: Undefined, because the residual standard deviation is zero; the centered cross-product is nevertheless zero.)

What You'll Learn