Why Plot Errors Against Predictions?
A single prediction metric can summarize performance, but it cannot show where errors occur. For example, positive and negative errors can cancel in the mean, even if a predictor tends to underestimate in one region and overestimate in another. A plot of errors against predicted values makes that structure visible: it shows whether errors are centered near zero across the prediction range, whether their average direction changes, and whether their spread grows or shrinks.
This plot is a descriptive diagnostic. It complements summaries such as mean prediction error, MAE, and RMSE, and it complements the calibration measures and plots used to assess agreement between predictions and outcomes. Its distinctive question is: how do the signed errors behave as the predicted value changes?
The word residual often refers to an observed-minus-fitted value in the data used to fit a regression model. When predictions are assessed on held-out cases, the same subtraction is often called a prediction error. In this tutorial, the plot is about observed-minus-predicted errors; whether the predictions are fitted or held out should be made clear. For an honest assessment of performance on new cases, predictions should be made without using those cases’ outcomes to fit or tune the prediction procedure.
The observed values and predictions must refer to the same outcome, on the same scale and in the same units. For example, if the outcome is time in hours, both \(y_i\), \(\hat{y}_i\), and \(e_i\) are in hours. The plotted points are paired: each error belongs with the prediction that produced it.
How to Read the Plot
Start with the horizontal line \(e=0\). Then look across the range of predicted values. A cloud with no obvious structure and roughly similar vertical spread throughout the range is consistent with errors that do not have a strong systematic pattern against predictions. It does not prove that the predictions are unbiased or accurate; a small, noisy, or unrepresentative validation sample may conceal patterns.
- Local bias: If errors in a region mostly lie above zero, predictions in that region tend to be too low. If they mostly lie below zero, predictions tend to be too high.
- Curvature or another ordered shape: If the center of the error cloud bends or changes direction across predicted values, the average error may depend on the prediction level in a non-constant way. A straight-line summary or a single overall mean may not describe this adequately.
- Changing spread: If errors are tightly clustered in one region and widely dispersed in another, prediction accuracy varies across predicted values. This can occur even when average errors in those regions are close to zero.
These features can overlap. A plot may show both a changing average error and a changing spread. Look at the center and the vertical dispersion separately: the center describes the typical signed direction, while the spread describes how much individual errors vary around that center.
Worked Example: Detecting Local Underprediction and Overprediction
Suppose a hypothetical validation set contains six travel-time predictions. Times are measured in hours. For each case, calculate the observed-minus-predicted error and inspect the pattern across low, middle, and high predictions.
| Predicted time \(\hat{y}_i\) (hours) | Observed time \(y_i\) (hours) | Error \(e_i=y_i-\hat{y}_i\) (hours) |
|---|---|---|
| 10 | 12 | \(12-10=2\) |
| 12 | 13 | \(13-12=1\) |
| 20 | 19 | \(19-20=-1\) |
| 22 | 20 | \(20-22=-2\) |
| 30 | 34 | \(34-30=4\) |
| 32 | 35 | \(35-32=3\) |
The low-prediction errors average \((2+1)/2=1.5\) hours, so predictions in that region are too low on average. The middle errors average \((-1-2)/2=-1.5\) hours, indicating overprediction there. The high-prediction errors average \((4+3)/2=3.5\) hours, indicating underprediction at the high end.
The overall mean error is \((2+1-1-2+4+3)/6=7/6\), or about \(1.17\) hours. That number indicates overall underprediction, but it does not reveal the change in direction in the middle region. Plotting the six errors against their predictions exposes that local pattern.
Why Grouped Mean Errors Reveal Local Bias
A scatterplot contains individual errors, which can be noisy. To describe its center in a prediction region, one may calculate the mean error for cases in that region. If \(G\) is a group of \(m\) cases, its mean error is
The equality follows by distributing the sum across the subtraction: \(\sum_{i\in G}(y_i-\hat{y}_i)=\sum_{i\in G}y_i-\sum_{i\in G}\hat{y}_i\), then dividing each sum by \(m\). Thus, a positive mean error in a region means that its observed outcomes average higher than its predictions; a negative mean error means they average lower.
This is a descriptive identity, not a test of whether the difference is statistically significant. Group boundaries can affect the summary, so do not treat a particular binning as uniquely correct. The scatterplot retains the individual points; grouped summaries can help describe a pattern, but should not replace inspection of the data.
Worked Example: Recognizing Curvature Despite Zero Mean Error
Consider hypothetical validation outcomes, measured in score points. The predictions are \(10,20,30,40,50\), and the observed scores are \(14,18,26,38,54\). The prediction errors are:
From low to high predictions, the plotted errors are \(4,-2,-4,-2,4\) points. They are positive at both ends and negative in the middle, making a U-shaped pattern. At the low and high ends, predictions tend to be too low; in the middle, predictions tend to be too high.
The overall mean error is \((4-2-4-2+4)/5=0\) points. A zero mean therefore does not mean that the errors are patternless: positive and negative errors cancel in this calculation. The shape suggests that the average error changes across prediction levels. It is a reason to investigate the prediction procedure and its validation performance, not by itself proof of a particular cause or a guarantee that changing the model will improve predictions.
Changing Error Spread Is a Different Pattern
Errors can have similar average directions across predicted values but very different degrees of scatter. A tight cluster indicates that individual outcomes tend to stay close to their predictions in that region. A wide cluster indicates less consistent accuracy there. Neither statement alone explains why the variability differs.
Worked Example: Finding Greater Error Spread at High Predictions
In a hypothetical validation set, an outcome and its predictions are measured in the same units. The following cases have predictions of 10, 20, or 30 units.
| Prediction \(\hat{y}_i\) | Observed outcomes \(y_i\) | Errors \(y_i-\hat{y}_i\) |
|---|---|---|
| 10 | 9, 11, 12 | \(-1, 1, 2\) |
| 20 | 18, 20, 22 | \(-2, 0, 2\) |
| 30 | 22, 31, 37 | \(-8, 1, 7\) |
The mean errors at predictions 10, 20, and 30 are, respectively, \(({-1+1+2})/3=0.667\), \(({-2+0+2})/3=0\), and \(({-8+1+7})/3=0\) units. There is little evidence of a change in average signed error across these three prediction levels in this small example.
But the mean absolute errors are \((1+1+2)/3=1.33\), \((2+0+2)/3=1.33\), and \((8+1+7)/3=5.33\) units. The high-prediction cases have much more dispersed errors, from \(-8\) to \(7\) units, compared with ranges of \(-1\) to \(2\) and \(-2\) to \(2\) in the other groups. A plot would show a wider vertical cloud at prediction 30. The average signed error alone would miss this change in prediction accuracy.
A Practical Reading Process
Confirm that every point pairs one observed outcome with its corresponding prediction, and that the error is calculated as observed minus predicted. Record the outcome units.
Use \(e=0\) as the reference. Points above it indicate underprediction; points below it indicate overprediction.
Look across low, middle, and high predicted values for regions where errors tend to be positive or negative, or for a bend or other systematic shape.
Separately assess whether the vertical scatter is fairly similar or becomes wider or narrower as predictions change.
Describe the visible pattern in context, then consider whether it matters for the intended use. A plot suggests what to investigate; it does not establish a cause or replace validation on relevant cases.
Why the Plot Matters—and a Common Mistake
The plot matters because prediction errors are not always interchangeable across the prediction range. Underpredicting smaller outcomes and overpredicting larger ones can yield a modest overall mean error while still being systematically wrong in both regions. Likewise, predictions may be close on average in every region but much less reliable for some predictions than others. Those distinctions can matter to decisions that depend on the size or direction of an error.
A common mistake is to see an overall mean error near zero and conclude that the predictions are unbiased everywhere. That reasoning confuses an overall average with the pattern of local errors. The correction is to inspect the signed errors against predicted values: check both whether the center shifts relative to zero and whether the vertical spread changes. Also avoid treating every visible wiggle as a stable feature; with few observations or substantial random variation, apparent structure may not persist in new data.
The plot is descriptive, not a formal hypothesis test. A pattern may motivate further investigation, but visual evidence alone does not show that it is statistically reliable, identify its cause, or establish that a proposed change will improve performance. Make conclusions in the context of the validation cases and the prediction task.
Check Your Understanding
Use the sign convention \(e_i=y_i-\hat{y}_i\) in your answers.
- A point has a predicted value of 8 units and an observed value of 11 units. What is its error, and was the prediction too high or too low?
- In a region of predicted values, the errors are mostly negative. Describe the direction of prediction bias in that region.
- Why can an overall mean error of zero coexist with a curved error pattern?
- What does a visibly wider vertical spread at high predicted values suggest, and what does it not establish?
- Why is it important that the prediction and observed outcome for each plotted case refer to the same case and use the same outcome scale?