Residuals as Signed Prediction Errors

Why Compare an Observation With Its Prediction?

A regression line gives a predicted response for a specified explanatory-variable value. But a prediction is not necessarily the response that was observed. To describe the difference for a particular observation, subtract its predicted response from its observed response. The result is a residual: a signed prediction error that records both the size and direction of the difference.

For example, suppose a line predicts a response of 18 units for an observation whose actual response is 21 units. The actual value is 3 units above the prediction, so the residual is positive 3 units. If the actual response were 16 instead, the residual would be negative 2 units. The sign matters: it tells whether the prediction was too low or too high.

Definition: For observation $i$, let $y_i$ be its observed response and let $\hat{y}_i$ be the response predicted by the regression line at that observation’s explanatory-variable value. The residual is $$ e_i = y_i - \hat{y}_i. $$ A residual is measured in the same units as the response variable. A positive residual means the observed response is greater than its prediction; a negative residual means the observed response is less than its prediction.

Calculate the Residual for One Observation

To calculate a residual, you need the observed response and the predicted response for the same observation. If the regression line is written as $\hat{y}=a+bx$, substitute the observation’s explanatory-variable value $x_i$ to find its prediction $\hat{y}_i$. Then subtract that prediction from the observed response $y_i$. The order is important: observed minus predicted, not predicted minus observed.

A residual is a vertical difference on a scatterplot, because it compares two response values at the same explanatory-variable value. It is not the horizontal distance between two explanatory-variable values. The calculation also preserves the measurement scale: if the response is measured in minutes, the residual is in minutes; if the response is measured in kilograms, the residual is in kilograms.

Worked Example: A Positive Residual for a Travel Time

Suppose a hypothetical regression line predicts travel time in minutes from distance in kilometres: $\hat{y}=12+3x$. A trip covering $4$ km takes $26$ minutes. Find and interpret the residual.

First calculate the predicted response at $x=4$:

\hat{y}_i = 12 + 3(4) = 12 + 12 = 24 \text{ minutes}.

The observed response is $y_i=26$ minutes. Therefore,

e_i = y_i - \hat{y}_i = 26 - 24 = 2 \text{ minutes}.

The residual is $2$ minutes. Because it is positive, the observed trip took 2 minutes longer than the line predicted. In this case, the prediction was too low by 2 minutes.

What the Sign Tells You

The sign follows directly from the subtraction in the definition. If the observed response is larger than the predicted response, subtracting the prediction leaves a positive number. If the observed response is smaller, the subtraction gives a negative number. If observed and predicted responses are equal, the residual is zero.

\begin{aligned} y_i > \hat{y}_i &\quad\Longrightarrow\quad e_i > 0,\\ y_i < \hat{y}_i &\quad\Longrightarrow\quad e_i < 0,\\ y_i = \hat{y}_i &\quad\Longrightarrow\quad e_i = 0. \end{aligned}

Equivalently, rearranging the definition gives

y_i = \hat{y}_i + e_i.

This identity is a useful check on both calculation and interpretation: observed response equals predicted response plus residual. A positive residual places the observation above the prediction at its $x$-value, so the line underpredicted the response. A negative residual places it below the prediction, so the line overpredicted the response. A zero residual means the observation lies exactly on the line.

The magnitude of a residual describes how far the observed response is from the prediction, regardless of direction. For instance, residuals of $4$ and $-4$ have the same magnitude, 4 response units, but describe opposite directions. It is therefore incomplete to report only that a residual is “4” unless the sign is clear. In context, give both the signed value and what it says about the prediction.

Worked Example: Residuals for Several Paired Observations

A hypothetical instructor uses the prediction rule $\hat{y}=50+5x$ to predict quiz score $y$, in points, from hours $x$ spent practicing. For three students, the paired observations are $(1,58)$, $(2,57)$, and $(4,75)$, where each pair lists practice hours and observed score. Find each predicted score and residual.

For the first student, $x_1=1$ and $y_1=58$:

\hat{y}_1 = 50+5(1)=55,\qquad e_1=58-55=3 \text{ points}.

The positive residual means the observed score was 3 points above the prediction.

For the second student, $x_2=2$ and $y_2=57$:

\hat{y}_2 = 50+5(2)=60,\qquad e_2=57-60=-3 \text{ points}.

The negative residual means the observed score was 3 points below the prediction.

For the third student, $x_3=4$ and $y_3=75$:

\hat{y}_3 = 50+5(4)=70,\qquad e_3=75-70=5 \text{ points}.

The third observed score was 5 points above its prediction. The residuals are therefore $3$, $-3$, and $5$ points, respectively. The first and second predictions miss by the same number of points, but in opposite directions. Each calculation uses the observed and predicted scores for the same student.

Residuals From a Table

When a data set contains several paired observations, calculate a prediction and a residual for each row. Keep the explanatory-variable value, observed response, prediction, and residual aligned. A table helps avoid accidentally subtracting a prediction for one observation from the response for another.

Worked Example: Completing a Residual Table

A hypothetical model predicts package delivery time $y$, in hours, from route length $x$, in kilometres, using $\hat{y}=1+0.2x$. The observed data are shown below. Find the prediction and residual for each route.

Route length $x$ (km)	Observed time $y$ (hours)	Predicted time $\hat{y}=1+0.2x$ (hours)	Residual $e=y-\hat{y}$ (hours)
5	2.4	2.0	0.4
10	2.7	3.0	-0.3
15	4.0	4.0	0

For the 5-km route, the prediction is $1+0.2(5)=2.0$ hours, so its residual is $2.4-2.0=0.4$ hours. The delivery took 0.4 hours longer than predicted.

For the 10-km route, the prediction is $1+0.2(10)=3.0$ hours, so its residual is $2.7-3.0=-0.3$ hours. The delivery took 0.3 hours less than predicted.

For the 15-km route, the prediction is $1+0.2(15)=4.0$ hours, so its residual is $4.0-4.0=0$ hours. The observed delivery time equals the prediction. All three residuals are in hours, the units of the response.

A Careful Calculation Routine

Residuals are simple to calculate, but a consistent routine prevents the most common errors. The prediction rule must be applied to the observation’s explanatory-variable value, and the subtraction must remain in the defined order.

Identify the paired values.
For the observation of interest, record its explanatory value $x_i$ and its observed response $y_i$.

Calculate its prediction.
Use the given regression line at $x_i$ to obtain $\hat{y}_i$.

Subtract in the correct order.
Compute $e_i=y_i-\hat{y}_i$, keeping enough precision during the calculation to avoid unnecessary rounding.

Check and interpret.
Use $y_i=\hat{y}_i+e_i$ to check the arithmetic, then describe the sign and size in the response variable’s units.

For these calculations, the regression line is treated as a given prediction rule, and each observed response must be paired with its own explanatory value. No probability model is needed to define or calculate a residual. A residual is a descriptive difference for an observed case, not a probability or a statement that the line caused the response.

Common Mistakes and Why Residuals Matter

Reversing the subtraction. Calculating $\hat{y}_i-y_i$ gives the opposite sign. The residual is defined as observed minus predicted: $y_i-\hat{y}_i$.
Reversing the interpretation. A positive residual does not mean the line overpredicted. It means observed is greater than predicted, so the line underpredicted. A negative residual means the line overpredicted.
Dropping the units. A residual is not automatically measured in the units of $x$. It has the response variable’s units, because it is the difference of two response values.
Mixing observations. Each observed response must be compared with the prediction at that observation’s own explanatory-variable value.
Reporting only a magnitude. Saying “the error is 3” leaves direction unclear. A residual of $3$ and one of $-3$ have equal magnitudes but different interpretations.

Residuals give a direct, observation-by-observation account of how a prediction differs from what was observed. They also provide a starting point for examining how a regression line fits a set of data; the detailed properties of the line and the criteria used to choose it are separate questions. For now, the essential ideas are the subtraction order, the response units, and the sign-based interpretation.

Key takeaway: For each observation, calculate $e_i=y_i-\hat{y}_i$. A positive residual means the observed response is above the prediction, a negative residual means it is below, and a zero residual means they match exactly.

Check Your Understanding

For each question, use observed minus predicted when calculating a residual.

A line predicts a blood pressure measurement of 118 mmHg, and the observed value is 123 mmHg. Find the residual and interpret its sign.
A model predicts a response of 42 units, while the observed response is 37 units. What is the residual, and did the model overpredict or underpredict?
At $x=3$, the prediction rule is $\hat{y}=8+2x$, and the observed response is $y=15$. Calculate the prediction and residual.
Two observations have residuals $6$ cm and $-6$ cm. Which prediction was too low, which was too high, and how do the miss sizes compare?
Explain why a residual for a response measured in kilograms is also measured in kilograms, even if the explanatory variable is measured in metres.