Least-Squares Constraints on Residuals

Why Least-Squares Residuals Have Built-In Constraints

Residuals are central to regression diagnostics, but they are not arbitrary differences between observed and fitted values. Ordinary least squares chooses the fitted coefficients to minimize the sum of squared residuals. That minimization imposes exact algebraic constraints on the residuals: with an intercept, they sum to zero, and their inner product with each included predictor column is zero.

These properties help explain what a residual plot can and cannot show. For example, a residual plot from a fitted model must balance around zero overall. That balance is a consequence of the fitting procedure, not proof that the model describes the data well. The same distinction applies to the linear association between residuals and predictors already included in the fit.

The Least-Squares Setup

Suppose there are \(n\) observations. For observation \(i\), let \(y_i\) be the response and let \(x_{ij}\) be the value of predictor \(j\), for \(j=1,\ldots,p\). An ordinary least-squares model with an intercept has fitted values

\hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_{i1}+\cdots+\hat{\beta}_p x_{ip}.

The residual for observation \(i\) is \(e_i=y_i-\hat{y}_i\). Ordinary least squares selects the coefficients to minimize the residual sum of squares, \(\sum_{i=1}^n e_i^2\). Here, assume the usual unweighted least-squares fit: the coefficients are chosen freely to minimize that sum, with no penalty or other constraint imposed on them.

Definition: Two columns of \(n\) numbers are orthogonal if their inner product—the sum of the products of corresponding entries—is zero. Thus the residuals are orthogonal to predictor \(j\) when \(\sum_{i=1}^n x_{ij}e_i=0\). With an intercept, they are also orthogonal to the constant column of ones, which means \(\sum_{i=1}^n e_i=0\).

These are properties of the fitted sample, not assumptions about unobserved errors or the population. No normality, constant-variance, or independence condition is needed for the algebraic constraints. The relevant ingredients are the least-squares minimization and the columns included in the fitted model.

Deriving the Constraints

At a least-squares minimum, changing any freely estimated coefficient slightly cannot reduce the sum of squared residuals. Since the sum of squares is differentiable in each coefficient, its partial derivative with respect to each coefficient must be zero at the minimum.

S(\beta_0,\beta_1,\ldots,\beta_p) =\sum_{i=1}^n\left(y_i-\beta_0-\sum_{j=1}^p\beta_jx_{ij}\right)^2.

Differentiate first with respect to the intercept. At the minimizing coefficients, the residuals are \(e_i=y_i-\hat{y}_i\), so

\frac{\partial S}{\partial\beta_0} =-2\sum_{i=1}^n e_i=0 \quad\Longrightarrow\quad \sum_{i=1}^n e_i=0.

For predictor \(j\), differentiating with respect to its coefficient gives

\frac{\partial S}{\partial\beta_j} =-2\sum_{i=1}^n x_{ij}e_i=0 \quad\Longrightarrow\quad \sum_{i=1}^n x_{ij}e_i=0.

This argument applies to every included predictor column. If the model includes a transformed predictor or a column representing a category, the residuals are orthogonal to that column as well. The constraint concerns the particular columns in the fitted model; it does not automatically apply to predictors or transformations that were not included.

Worked Example: A Simple Regression Fit

Consider hypothetical observations with predictor values \(x=(0,1,2)\) and responses \(y=(1,2,2)\). Fit a straight-line regression with an intercept and check the two residual constraints.

The predictor and response means are \(\bar{x}=1\) and \(\bar{y}=5/3\). The least-squares slope is

\hat{\beta}_1 =\frac{\sum_i(x_i-\bar{x})(y_i-\bar{y})}{\sum_i(x_i-\bar{x})^2} =\frac{1}{2}=0.5.

The intercept is \(\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}=5/3-1/2=7/6\). Thus the fitted values are \(7/6,\ 5/3,\ 13/6\), and the residuals are

e=\left(1-\frac{7}{6},\ 2-\frac{5}{3},\ 2-\frac{13}{6}\right) =\left(-\frac{1}{6},\ \frac{1}{3},\ -\frac{1}{6}\right).

Their sum is \(-1/6+1/3-1/6=0\). Their inner product with the predictor is \(0(-1/6)+1(1/3)+2(-1/6)=0\). Both constraints hold exactly; the values are not merely close to zero because of rounding.

What the Constraints Mean

Because the residuals sum to zero, their sample mean is zero. This is also why, in a model with an intercept, the average fitted value equals the average observed response:

\bar{e}=0 \quad\text{and}\quad \bar{\hat{y}}=\bar{y}.

The predictor constraint has a centered version. For any included predictor \(x_j\),

\sum_{i=1}^n(x_{ij}-\bar{x}_j)e_i =\sum_{i=1}^n x_{ij}e_i-\bar{x}_j\sum_{i=1}^n e_i =0.

So the sample covariance between an included predictor and the residuals is zero. If both variables have nonzero sample standard deviations, their sample correlation is also zero. If all residuals are zero, however, their standard deviation is zero and the correlation is undefined; the zero inner product still holds.

The residuals are also orthogonal to the fitted values. Each fitted value is a linear combination of the intercept column and the included predictor columns. Therefore,

\sum_{i=1}^n \hat{y}_i e_i =\hat{\beta}_0\sum_{i=1}^n e_i +\sum_{j=1}^p\hat{\beta}_j\sum_{i=1}^n x_{ij}e_i =0.

Since the residuals sum to zero, the centered inner product with fitted values is zero too. Thus a simple linear association between residuals and fitted values is ruled out by the fitting procedure itself when the model contains an intercept.

Worked Example: Several Included Predictors

Suppose a hypothetical model includes an intercept and two predictors. For four observations, let the predictor columns be \(x_1=(-1,-1,1,1)\) and \(x_2=(-1,1,-1,1)\). Consider responses \(y=(10,6,12,12)\).

A candidate fit with intercept \(10\), coefficient \(2\) for \(x_1\), and coefficient \(-1\) for \(x_2\) gives fitted values \((9,7,13,11)\). The residuals are therefore \(e=(1,-1,-1,1)\).

To verify that these are the least-squares coefficients, note that the intercept column \((1,1,1,1)\), \(x_1\), and \(x_2\) are pairwise orthogonal. The response has inner products \(40\), \(8\), and \(-4\) with those columns, respectively. Dividing by each column’s squared length, which is \(4\), gives the coefficients \(10,\ 2,\ -1\). These are the least-squares projections onto the three included columns.

The residual constraints can now be checked directly:

\sum_i e_i=1-1-1+1=0,\qquad \sum_i x_{i1}e_i=-1+1-1+1=0,\qquad \sum_i x_{i2}e_i=-1-1+1+1=0.

Thus the residuals are orthogonal to both included predictors and the intercept column. This does not say that they have no possible pattern across observations; it says only that their inner products with these particular fitted columns are zero.

Diagnostic Implications—and a Common Mistake

In a residual plot, the overall residual mean being zero is expected for a least-squares model with an intercept. Likewise, a zero linear sample association between residuals and an included predictor is expected. Neither fact, by itself, is evidence that the model’s assumptions are satisfied or that its form is adequate. Residuals can still vary systematically in ways not captured by these exact linear constraints.

For instance, a zero linear association with a predictor does not rule out a curved relationship between that predictor and the residuals. Nor does orthogonality to included predictors establish that residuals are independent, normally distributed, or equally variable. Those are different questions, assessed using relevant diagnostic evidence.

Common mistake: “The residuals sum to zero and have zero correlation with the included predictor, so the model fits well.” The correction is that these are built-in consequences of ordinary least squares with an intercept. They do not measure whether the residuals have other systematic patterns or whether the model’s assumptions are reasonable.

Worked Example: The Zero-Sum Constraint Is Not a Goodness-of-Fit Check

Suppose an intercept-only model is fitted to the hypothetical responses \(8,\ 10,\ 13,\ 9\). With no predictors, the least-squares fitted value for every observation is the sample mean:

\bar{y}=\frac{8+10+13+9}{4}=10.

The fitted values are all \(10\), giving residuals \((-2,0,3,-1)\). Their sum is \(-2+0+3-1=0\), as required. Yet the residuals are not all small: one is \(3\) response units above the fitted value, and another is \(2\) units below it. The zero sum confirms the intercept constraint, not that predictions are close or that an intercept-only model is suitable for a particular purpose.

Using the Constraints Carefully

The algebra gives a useful reference point when reading regression output. With an intercept, a reported residual mean noticeably different from zero may indicate rounding, a different fitting procedure, or that the model did not include an ordinary freely estimated intercept. For a predictor used in an ordinary least-squares fit, an apparent nonzero sample residual correlation may likewise result from rounding or from a mismatch between the plotted quantity and the column actually fitted.

But the reverse inference is not justified: seeing a zero mean or zero linear correlation does not validate the regression. Treat the constraints as properties of the calculation, then use residual diagnostics to assess patterns and assumptions that the calculation does not force to vanish. The key distinction is between what least squares guarantees algebraically and what the residuals reveal about model adequacy.

Check Your Understanding

Answer each question using the definitions and constraints from this tutorial.

For a least-squares model with an intercept, what derivative condition implies that the residuals sum to zero?
If a predictor column is included in an ordinary least-squares model, what inner product of that column and the residuals must equal zero?
Why does zero sample covariance between an included predictor and the residuals not rule out every possible residual pattern?
Explain why a residual mean of zero is not, by itself, evidence that the fitted model is adequate.
Would the residuals necessarily sum to zero in an ordinary least-squares model that has no intercept? Explain what the intercept derivative contributes to the result.