Prediction Errors and Residuals

What Does a Prediction Error Measure?

A regression prediction is rarely exactly equal to the outcome later observed. The difference between the two tells us how far the prediction missed for that particular case, and in which direction. To interpret that difference, we need to keep track of whether the case helped fit the regression rule or was held out from fitting.

Suppose a fitted regression rule uses a case’s predictor values to produce a predicted outcome. For an individual case, compare the observed outcome with that prediction. The order of subtraction matters: in this tutorial, the signed difference is always observed minus predicted.

The calculation has the same form for training and held-out cases. The distinction is about how the prediction rule was obtained and which data were used: a training residual comes from a case used to fit the rule, while a held-out prediction error comes from a case excluded from fitting. In both cases, \(\hat{y}\) means the prediction produced by the relevant fitted rule for that individual’s predictor values.

A positive error means the observed outcome was larger than predicted: the prediction was too low, or an underprediction. A negative error means the observed outcome was smaller than predicted: the prediction was too high, or an overprediction. An error of zero means the prediction exactly matched the observed outcome. These are statements about one case, not a summary of all predictions.

Training Residuals and Held-Out Prediction Errors

Let a regression procedure be fitted using a training set. For training case \(i\), write its observed outcome as \(y_i\) and its fitted prediction as \(\hat{y}_i\). Its training residual is

Now consider held-out case \(j\), whose outcome was not used to fit the prediction rule. Once its outcome is observed, its held-out prediction error is

The different labels \(e_i\) and \(u_j\) make it clear which data source is being discussed. Some texts use the word “residual” more broadly for observed-minus-predicted differences on any data. The important practice is to state whether a difference comes from training data or held-out data. The arithmetic alone does not reveal that distinction.

A held-out prediction requires a fitted rule that can generate a prediction from the case’s predictor values. Its observed outcome is then used to calculate the error, not to revise that prediction for the evaluation at hand. The held-out data should not have influenced fitting or choices that adapt the model to those data. This preserves the distinction between evaluating predictions on new cases and describing how well the model fits cases it has already seen.

No normality assumption or other distributional condition is required just to calculate an individual prediction error. The necessary ingredients are a defined outcome, an observed value, and a prediction produced for that same outcome. The prediction and observation must refer to the same case and use the same outcome definition and units. If a rule predicts a transformed outcome, the comparison must be made on a consistent scale.

Why the Sign and Subtraction Order Matter

The sign convention follows directly from the definition. Since \(y-\hat{y}\) is positive exactly when \(y\) is greater than \(\hat{y}\), a positive error indicates underprediction. If \(y\) is less than \(\hat{y}\), the difference is negative, indicating overprediction. Rearranging the definition gives a useful check:

In words, the observed value equals the prediction plus the signed error. If an error has the wrong sign, this identity will not recover the observed outcome. This is a quick way to check a calculation.

A signed error is a vertical difference on the outcome scale: it compares two outcome values for the same predictor values. It is not a distance measured along the fitted line, and it is not a percentage unless a separate calculation explicitly converts it to one. Because the error keeps the outcome’s units, an error of \(3\) minutes means something different from an error of \(3\) kilograms.

Calculating Errors for Several Held-Out Cases

When several cases are held out, calculate a prediction and then a signed error for each case separately. Keep the case identities together with the results: each error describes the miss for one observed outcome, not an abstract property of the model.

What an Individual Error Can—and Cannot—Tell You

An individual error identifies the direction and size of a model’s miss on one case. It can help locate cases for which predictions were notably high or low, and it provides the basic quantity from which broader assessments of prediction performance can be formed. But one error cannot establish how well the model performs across a population or across future cases. A small error for one case does not guarantee small errors elsewhere.

Also, an observed-minus-predicted difference is not automatically a causal effect, a measurement mistake, or the unobservable random disturbance in a theoretical model. It is the realized discrepancy between a particular observed outcome and a particular prediction. Other factors may help explain that discrepancy, but the error alone does not identify them.

Common Mistakes

• Reversing the subtraction. Calculating predicted minus observed reverses the sign. Use \(y-\hat{y}\), then check that observed equals predicted plus error.
• Calling every error a training residual. The same subtraction applies to both, but a held-out case was not used to fit the prediction rule. Label the data status accurately.
• Interpreting a positive error as overprediction. Since observed is larger than predicted when \(y-\hat{y}>0\), a positive error means underprediction.
• Using the predictor’s units for the error. Errors are differences in the outcome, so their units are the outcome’s units.
• Treating one case’s error as a complete model evaluation. It describes that case only; conclusions about general predictive performance require assessment across relevant cases.