Computing and Interpreting Regression Residuals

Why Residuals Matter

A fitted regression line gives a predicted response for each predictor value. But a prediction is not usually the same as the observed response for an individual case. The difference between those two values tells us how far that case’s observed response is from the line, and in which direction.

That difference is called a residual. Residuals put individual prediction errors on the same scale as the response, such as points, minutes, or dollars. They also give each observed point a direct connection to its fitted line: the residual is the point’s signed vertical gap from the line.

Calculating a Residual

Suppose a fitted line predicts the response \(y\) from the predictor \(x\) using \(\hat{y}=a+bx\), where \(a\) is the fitted intercept and \(b\) is the fitted slope. For observation \(i\), first substitute its predictor value \(x_i\) to calculate \(\hat{y}_i=a+bx_i\). Then subtract that prediction from its observed response \(y_i\).

The order of subtraction matters. A positive residual means the observed response is greater than the predicted response. A negative residual means the observed response is less than predicted. A residual of zero means the observation falls exactly on the fitted line. In all three cases, the residual has the same units as the response because it is a difference between two response values.

Residuals as Vertical Gaps on a Scatterplot

A scatterplot displays predictor values along the horizontal axis and response values along the vertical axis. An observation \((x_i,y_i)\) is plotted at its predictor and its observed response. The fitted line gives the predicted response \(\hat{y}_i\) at that same horizontal position \(x_i\). The residual compares \(y_i\) with \(\hat{y}_i\), so it is a vertical difference between the point and the line.

If \(e_i>0\), then \(y_i>\hat{y}_i\), and the point lies above the line. If \(e_i<0\), then \(y_i<\hat{y}_i\), and the point lies below the line. If \(e_i=0\), the observed point is on the line. The size of the vertical gap is \(|e_i|\), while the sign records its direction.

A residual is not a horizontal distance to the line, and it is not the shortest geometric distance to the line. It compares two response values at the same \(x_i\). This is why a residual’s units come from the vertical axis, not the horizontal one.

What a Residual Does—and Does Not—Measure

A residual belongs to one particular paired observation and one particular fitted line. If either the observed response or the fitted line changes, the residual may change. It is therefore not a permanent attribute of a person or object. To interpret one correctly, identify the observation, the response, the fitted prediction at that observation’s predictor value, and the response units.

A residual is also distinct from an unobserved model error. The model error is the deviation of an observed response from the underlying population relationship, which is not directly known. A residual instead uses the fitted line estimated from the available data, so it can be calculated from the observed response and the fitted prediction. It is an observable difference, not a direct measurement of an unknown population error.

Residuals are useful because they show how individual observations differ from predictions. A large absolute residual means the observed response is far from the fitted prediction in response units. That fact alone does not determine whether the whole regression is appropriate, nor does it explain why the observation differs. Assessing overall patterns in residuals is a separate task.

A Useful Check for Least-Squares Lines

When a straight line is fitted by ordinary least squares with an intercept, its residuals sum to zero, apart from possible rounding in reported values. This is a property of the fitting procedure, not a rule that every residual must be zero or that every point lies close to the line.

To see why, write the sum of squared residuals for a line with intercept \(a\) and slope \(b\) as \(Q(a,b)=\sum_i (y_i-a-bx_i)^2\). At the least-squares fit, changing the intercept slightly cannot reduce \(Q\). Therefore the derivative of \(Q\) with respect to \(a\) is zero:

It follows that \(\sum_i e_i=0\). In the quiz-score example, the three residuals sum to \(3+(-3)+4=4\), not zero. That line was supplied as a hypothetical fitted equation and was not specified as the ordinary least-squares line for those three observations. The check applies only when the line was fitted by ordinary least squares with an intercept, and exact residuals are used.

Common Mistakes

• Reversing the subtraction. A learner might calculate \(\hat{y}_i-y_i\), then say a positive result means the observation was above the line. The correction is to use \(e_i=y_i-\hat{y}_i\): positive residuals are above the line, and negative residuals are below it.
• Using the wrong case’s values. A prediction at one predictor value must be compared with the observed response for that same observation. Pair each \(x_i\) with its own \(y_i\) before calculating.
• Giving the predictor’s units. A rent residual is measured in dollars per month, not square metres. Since \(y_i\) and \(\hat{y}_i\) are both response values, their difference has response units.
• Calling the residual a horizontal or shortest distance. The residual compares observed and predicted responses at the same predictor value. On the usual scatterplot, it is a signed vertical gap.
• Assuming zero-sum residuals mean a perfect fit. Residuals can add to zero while individual residuals are large, because positive and negative values can cancel. The zero-sum property is not evidence that every observation is predicted exactly.