Why OLS Residuals Are Orthogonal to Predictors

Why Residual Orthogonality Matters

In ordinary least squares, fitted values are chosen to make the sum of squared residuals as small as possible. That minimization has an important geometric consequence: once the fit is made, the residual vector has zero inner product with every column of the design matrix. This is true for the intercept column when one is included, and for each predictor column.

The result is useful both as a way to understand least squares and as a practical calculation check. It is an algebraic property of the least squares solution, not a claim that residuals are independent of predictors or that the model assumptions guarantee each observed residual is zero.

Definition: For a response vector \(y\) and design matrix \(X\), a least squares estimate \(\hat{\beta}\) minimizes \((y-X\beta)^{\mathsf T}(y-X\beta)\). The fitted-value vector is \(\hat{y}=X\hat{\beta}\), and the residual vector is \(e=y-\hat{y}\). The residuals are orthogonal to every design-matrix column when \(X^{\mathsf T}e=0\); equivalently, the inner product of \(e\) with each column of \(X\) is zero.

From Minimization to Orthogonality

Let \(X\) be an \(n\)-by-\(p\) design matrix, \(y\) an \(n\)-vector of observed responses, and \(\beta\) a \(p\)-vector of coefficients. Least squares minimizes the residual sum of squares as a function of \(\beta\):

RSS(\beta)=(y-X\beta)^{\mathsf T}(y-X\beta).

At a minimizing coefficient vector, a small change in any coefficient cannot reduce the objective to first order. Differentiating with respect to \(\beta\) and setting the gradient equal to zero gives the normal equations.

\nabla_{\beta}RSS(\beta)=-2X^{\mathsf T}(y-X\beta)=0, \qquad X^{\mathsf T}X\hat{\beta}=X^{\mathsf T}y.

Since \(e=y-X\hat{\beta}\), the same condition can be written directly in terms of the residuals:

X^{\mathsf T}e =X^{\mathsf T}(y-X\hat{\beta}) =X^{\mathsf T}y-X^{\mathsf T}X\hat{\beta} =0.

Theorem: At any least squares solution \(\hat{\beta}\), the residual vector \(e=y-X\hat{\beta}\) satisfies \(X^{\mathsf T}e=0\). Thus, for every column \(x_j\) of \(X\), \(\sum_{i=1}^n x_{ij}e_i=0\). If \(X\) has full column rank, the solution is unique; full rank is not needed for the orthogonality condition to hold at a least squares solution.

The result can also be understood by considering a change in the fitted values that moves along a column of \(X\). If the coefficient of column \(x_j\) changes by a small amount \(t\), the residual vector changes from \(e\) to \(e-tx_j\). The resulting sum of squares is

(e-tx_j)^{\mathsf T}(e-tx_j) =e^{\mathsf T}e-2t x_j^{\mathsf T}e+t^2x_j^{\mathsf T}x_j.

At the minimum, the linear term in \(t\) must be zero: otherwise, choosing a sufficiently small \(t\) with the appropriate sign would reduce the sum of squares. Hence \(x_j^{\mathsf T}e=0\). This argument applies to each design column, including the intercept column if it is present.

No probability distribution for the errors is needed for this result. It follows from least squares minimization alone. Regression assumptions about the conditional mean or error variance are relevant to other conclusions, such as the interpretation and precision of coefficient estimates, but they are not what makes \(X^{\mathsf T}e=0\).

What the Intercept Adds

When a model includes an intercept, its design matrix contains a column of ones, \(\mathbf{1}=(1,\ldots,1)^{\mathsf T}\). Orthogonality to that column gives

\mathbf{1}^{\mathsf T}e=\sum_{i=1}^n e_i=0.

So the residuals sum to zero when the fitted model includes an intercept. This is a consequence of the intercept column's presence, not a universal property of all least squares regressions. More generally, orthogonality to predictor column \(x_j\) gives \(\sum_i x_{ij}e_i=0\), which does not usually imply that the unweighted residuals sum to zero.

For an intercept model, \(\sum_i e_i=0\) also means that the average fitted value equals the average observed response: because \(y_i=\hat{y}_i+e_i\), summing over observations gives \(\sum_i y_i=\sum_i\hat{y}_i\). This equality concerns the sample averages in the fitted data; it does not mean that every fitted value equals its observed response.

Worked Example: Checking Orthogonality in a Simple Regression

For three hypothetical observations, a regression with an intercept uses predictor values \(x=(0,1,2)^{\mathsf T}\) and responses \(y=(1,2,2)^{\mathsf T}\). Find the least squares line and verify that its residual vector is orthogonal to both the intercept column and the predictor column. Method: solve the two normal equations. The calculation is algebraic, so no error distribution or probability parameters are needed. The main mistake to avoid is checking only that the residuals sum to zero.

The design columns are \(\mathbf{1}=(1,1,1)^{\mathsf T}\) and \(x=(0,1,2)^{\mathsf T}\). The normal equations are

\begin{aligned} 3\hat{\beta}_0+3\hat{\beta}_1&=5,\\ 3\hat{\beta}_0+5\hat{\beta}_1&=6. \end{aligned}

Subtract the first equation from the second: \(2\hat{\beta}_1=1\), so \(\hat{\beta}_1=1/2\). Substituting into the first equation gives \(3\hat{\beta}_0+3/2=5\), hence \(\hat{\beta}_0=7/6\). The fitted values are \(7/6,\ 5/3,\ 13/6\). Subtracting each fitted value from its observed response gives

e= \begin{pmatrix} 1-7/6\\ 2-5/3\\ 2-13/6 \end{pmatrix} = \begin{pmatrix} -1/6\\ 1/3\\ -1/6 \end{pmatrix}.

For the intercept column, the inner product is \(-1/6+1/3-1/6=0\). For the predictor column, it is \(0(-1/6)+1(1/3)+2(-1/6)=1/3-1/3=0\). Thus \(X^{\mathsf T}e=0\), as required. The residuals do not all vanish; their orthogonality is the relevant check.

Worked Example: Orthogonality to Every Column in Multiple Regression

A hypothetical response is recorded for four observations with two predictors. The design rows are \((1,-1,-1)\), \((1,-1,1)\), \((1,1,-1)\), and \((1,1,1)\), where the first entry is the intercept. The observed responses are \(y=(10,6,12,12)^{\mathsf T}\). Find the least squares coefficients and verify each column condition. Method: use the normal equations and then take the inner product with each column. No probabilistic distribution is assumed. The trap is to verify only the intercept condition and overlook the predictor conditions.

The three columns are \(\mathbf{1}=(1,1,1,1)^{\mathsf T}\), \(x_1=(-1,-1,1,1)^{\mathsf T}\), and \(x_2=(-1,1,-1,1)^{\mathsf T}\). Each predictor column sums to zero, and their cross-product is \(x_1^{\mathsf T}x_2=1-1-1+1=0\). Their squared lengths are \(4\). Therefore the normal equations separate:

\hat{\beta}_0=\frac{\sum_i y_i}{4}=\frac{40}{4}=10, \quad \hat{\beta}_1=\frac{x_1^{\mathsf T}y}{4} =\frac{-10-6+12+12}{4}=2, \quad \hat{\beta}_2=\frac{x_2^{\mathsf T}y}{4} =\frac{-10+6-12+12}{4}=-1.

The fitted values, in row order, are \(10-2-(-1)=9\), \(10-2+(-1)=7\), \(10+2-(-1)=13\), and \(10+2+(-1)=11\). Hence \(e=(1,-1,-1,1)^{\mathsf T}\). The three inner products are

\mathbf{1}^{\mathsf T}e=1-1-1+1=0,\qquad x_1^{\mathsf T}e=-1+1-1+1=0,\qquad x_2^{\mathsf T}e=-1-1+1+1=0.

Every design column is orthogonal to the residual vector. The check is complete: the residual is not merely centered; it is orthogonal to each predictor column as well.

Worked Example: Why Residuals Need Not Sum to Zero Without an Intercept

Consider a regression through the origin with one predictor, \(x=(1,2,3)^{\mathsf T}\), and responses \(y=(2,1,4)^{\mathsf T}\). Find the least squares coefficient and check the residual inner product. Then determine whether the residuals sum to zero. Method: use the one-column normal equation. The common mistake is to impose the intercept-model sum condition even though the design has no intercept.

Here \(X=x\), so the normal equation gives

\hat{\beta}_1=\frac{x^{\mathsf T}y}{x^{\mathsf T}x} =\frac{1(2)+2(1)+3(4)}{1^2+2^2+3^2} =\frac{16}{14}=\frac{8}{7}.

The fitted values are \(8/7,\ 16/7,\ 24/7\), and the residuals are \(6/7,\ -9/7,\ 4/7\). The predictor inner product is

x^{\mathsf T}e =1(6/7)+2(-9/7)+3(4/7) =(6-18+12)/7=0.

However, the residual sum is \((6-9+4)/7=1/7\), not zero. This does not contradict the theorem: the only design column is \(x\), and there is no column of ones. Orthogonality is required for the columns that are actually in the design matrix.

Common Mistakes and Practical Checks

Assuming residuals always sum to zero. That conclusion follows from orthogonality to a column of ones. Without an intercept, it need not hold.
Checking only the intercept condition. A zero residual sum does not by itself verify orthogonality to any predictor. Check \(\sum_i x_{ij}e_i=0\) for each predictor column.
Confusing orthogonality with independence. The normal equations establish zero sample inner products. They do not establish that residuals are statistically independent of predictors.
Attributing the result to normal errors. The condition follows from least squares minimization and does not require normally distributed errors.
Expecting exact zero after rounded calculations. If coefficients or residuals have been rounded, the computed inner products may be close to, rather than exactly, zero. Use unrounded values when checking.

For an exact arithmetic check, form each residual using the same unrounded fitted coefficients, then compute its inner product with each column of \(X\). For a numerical software output, small discrepancies can result from floating-point precision; a discrepancy that is large relative to the scale of the calculation suggests an error in the fit or the arithmetic.

Key takeaway: The least squares normal equations are exactly the statement \(X^{\mathsf T}e=0\). They make residuals orthogonal to every included design column. An intercept adds the special consequence that residuals sum to zero; without that column, do not assume the sum is zero.

Check Your Understanding

For each problem, use the stated design to decide which residual inner products must be zero.

A regression with an intercept has \(x=(0,1,2)^{\mathsf T}\) and \(y=(3,4,8)^{\mathsf T}\). Find the fitted line and residual vector, then check both column inner products. (Answer: \(\hat{\beta}_0=5/2,\ \hat{\beta}_1=5/2,\ e=(1/2,-1,1/2)^{\mathsf T}\); both inner products are zero.)
A regression through the origin has \(x=(1,2)^{\mathsf T}\) and \(y=(1,3)^{\mathsf T}\). Find the slope and residuals, and state their sum. (Answer: \(\hat{\beta}_1=7/5,\ e=(-2/5,1/5)^{\mathsf T}\), with \(x^{\mathsf T}e=0\) and residual sum \(-1/5\).)
A design matrix has columns \(\mathbf{1}\), \(x_1\), and \(x_2\), and the least squares residual vector is \(e=(2,-1,-1)^{\mathsf T}\). Must the intercept-column inner product be zero? (Answer: Yes. In fact, this vector sums to \(0\), as the normal equations require for a design containing an intercept.)
A full-rank least squares fit has \(X^{\mathsf T}e=0\), but the residuals are not independent of the observed predictors. Is this a contradiction? (Answer: No. Orthogonality is a sample algebraic property and does not imply statistical independence.)
A model has no intercept and design columns \(x_1\) and \(x_2\). What normal-equation checks must its residual vector satisfy, and what cannot be concluded in general? (Answer: \(x_1^{\mathsf T}e=0\) and \(x_2^{\mathsf T}e=0\); one cannot conclude that \(\mathbf{1}^{\mathsf T}e=0\).)

What You'll Learn