Why Examine Prediction Errors in Time Order?
An overall error summary can describe average performance while concealing when that performance occurred. For example, a model may underpredict in one part of a validation period and overpredict in another. Those errors can offset in a signed average, even though the pattern could matter for the decisions supported by the predictions. A chronological plot makes the sequence visible.
This tutorial focuses on plotting numeric prediction errors in their time order. The goal is diagnostic: identify sustained bias, gradual changes, repeated patterns, or changes in error magnitude, and distinguish them from an isolated unusual observation. A visible pattern can prompt investigation, but it does not by itself establish why the pattern occurred.
The sign convention must be stated because some applications define error in the opposite direction. Here, positive always means underprediction. If errors are expressed in dollars, the vertical axis is in dollars; if the outcome is a count, the errors are in count units. A horizontal reference line at zero marks exact predictions. Points above and below that line indicate the direction of error.
The plot is meaningful only if the time labels match the order in which the outcomes occurred. Use calendar dates or periods when the intervals matter; a simple sequence index is appropriate when the observations are equally spaced and the actual dates add no useful information. Do not sort errors by size or by the predictions themselves: doing so destroys the temporal sequence the plot is intended to inspect.
Constructing and Reading the Plot
For a validation set, compute one signed error for each observed outcome using the prediction made for that observation. Plot those errors in chronological order and include the zero reference line. In R, if `time` contains dates or ordered period labels and `error` contains \(y_t-\hat{y}_t\), a basic plot is:
The line connects adjacent observations to make runs and changes easier to see; it does not imply that error takes a meaningful value between observed times. With a large number of observations, points may overlap, so a point-only plot or a less crowded time scale may be clearer. Keep the chronological order intact.
Read the plot by asking several distinct questions, rather than trying to label every fluctuation as a problem:
- Is there sustained signed bias? A long run mostly above zero suggests underprediction during that interval; a run below zero suggests overprediction.
- Does the center appear to drift? Errors that move gradually from one side of zero to the other may indicate a changing relationship between predictions and outcomes.
- Does the pattern recur? Similar rises and falls at repeated intervals may point to a recurring calendar or operational pattern worth checking.
- Does the spread change? Errors may become more variable or more extreme in a later interval even if their signed average remains near zero.
- Is a feature driven by one point? An isolated large error differs from a run or a sustained shift, even if it has a large effect on an overall error metric.
These are visual diagnostic descriptions, not statistical tests. A short run may occur by chance, and a plot alone cannot determine whether a trend is statistically meaningful. Nor does a pattern prove a cause: a change in the data, exposure, recording process, prediction procedure, or operating environment could be relevant. The context and construction of the validation data must be checked before drawing conclusions.
Summarizing a Pattern Without Hiding It
A plot should remain the primary display because its ordering reveals runs and isolated points. Simple summaries can help quantify what the plot suggests. For a set of \(n\) errors in one interval, the mean signed error is \(\bar{e}=\sum_{t=1}^{n}e_t/n\). A positive value indicates net underprediction in that interval; a negative value indicates net overprediction. But opposite-signed errors can cancel, so a mean near zero does not imply that individual errors are small.
To examine error magnitude, one may also calculate the mean absolute error, \(\sum_{t=1}^{n}|e_t|/n\), or the root mean squared error, \(\sqrt{\sum_{t=1}^{n}e_t^2/n}\), for a clearly defined interval. These summaries ignore the order within that interval. Therefore, calculate them as a supplement to the time plot, not as a substitute for it. Comparisons are most interpretable when the intervals and the outcome scale are clearly stated.
The following justification explains why signed averages are useful for describing direction, but also why they can conceal offsetting errors. For each observation, \(e_t=y_t-\hat{y}_t\). Summing both sides over the same \(n\) observations and dividing by \(n\) gives:
Thus the mean signed error is exactly the difference between the average observed outcome and the average prediction for those same observations. If positive and negative errors cancel, that difference can be zero even when many individual errors are nonzero. The identity is arithmetic and requires no distributional assumption; interpreting it as evidence beyond the observed period would require additional assumptions and analysis.
Worked Examples
Worked Example: A Shift From Overprediction to Underprediction
A hypothetical portfolio model produces monthly claim-cost predictions in thousands of dollars. The observed costs and predictions for six consecutive months are shown below. Calculate the signed errors, compare the first three months with the last three, and interpret the sequence.
| Month | Observed \(y_t\) | Predicted \(\hat{y}_t\) | Error \(e_t=y_t-\hat{y}_t\) |
|---|---|---|---|
| 1 | 48 | 50 | -2 |
| 2 | 51 | 52 | -1 |
| 3 | 53 | 53 | 0 |
| 4 | 56 | 53 | 3 |
| 5 | 57 | 53 | 4 |
| 6 | 59 | 54 | 5 |
Step 1: Check each error. For month 1, \(48-50=-2\). For month 2, \(51-52=-1\). For month 3, \(53-53=0\). For months 4, 5, and 6, the calculations are \(56-53=3\), \(57-53=4\), and \(59-54=5\), respectively. The errors are therefore \((-2,-1,0,3,4,5)\), in thousands of dollars.
Step 2: Compare signed averages by interval. The first-three-month mean is \((-2-1+0)/3=-3/3=-1\) thousand dollars. The last-three-month mean is \((3+4+5)/3=12/3=4\) thousand dollars. The later interval has a mean error \(4-(-1)=5\) thousand dollars higher.
Interpretation and check. The sequence moves from negative errors to positive errors, so the model shifts from overprediction toward underprediction. The two interval means agree with that visual change. This is evidence of a time pattern to investigate, not proof of its cause. The common mistake would be to report only the six-month mean: the total is \(-2-1+0+3+4+5=9\), so the overall mean is \(9/6=1.5\) thousand dollars, which hides the change in direction.
Worked Example: One Isolated Error or a Persistent Pattern?
Two hypothetical validation sequences each contain ten daily errors, in dollars. Sequence A is \((0,1,-1,0,8,0,-1,1,0,1)\). Sequence B is \((0,1,3,4,5,4,3,1,0,1)\). Compare the overall signed means and the shape of each sequence to decide whether a positive mean reflects one unusual point or a sustained run.
Step 1: Calculate Sequence A's signed mean. Its sum is \(0+1-1+0+8+0-1+1+0+1=9\), so its mean is \(9/10=0.9\) dollars. The absolute errors sum to \(0+1+1+0+8+0+1+1+0+1=13\) dollars. If the fifth-day error of 8 dollars is excluded, the remaining sum is \(9-8=1\), and the remaining nine-error mean is \(1/9\approx0.1111\) dollars.
Step 2: Calculate Sequence B's signed mean. Its sum is \(0+1+3+4+5+4+3+1+0+1=22\), so its mean is \(22/10=2.2\) dollars. All ten errors are nonnegative, and eight are positive. The values rise to 5 and then fall, with several consecutive positive errors.
Interpretation and check. Sequence A's mean is strongly influenced by one isolated error: removing that observation reduces the mean from 0.9 to about 0.1111 dollars. Sequence B instead shows a sustained positive run and a larger mean of 2.2 dollars. A mean by itself would not show this distinction. The appropriate check is to inspect the time-ordered points and ask whether the pattern remains when the isolated point is set aside for diagnostic comparison; do not silently delete a genuine observation from reported results.
Worked Example: Greater Error Magnitude With No Signed Bias
A hypothetical monthly predictor has four early validation errors of \((1,-1,1,-1)\) dollars and four later errors of \((3,-3,4,-4)\) dollars. Compare the signed mean and root mean squared error (RMSE) for each interval. Use the definition \(\mathrm{RMSE}=\sqrt{\sum e_t^2/n}\).
Step 1: Early interval. The signed sum is \(1-1+1-1=0\), so the mean signed error is \(0/4=0\). The squared errors are \(1,1,1,1\); their sum is 4. Thus \(\mathrm{RMSE}=\sqrt{4/4}=\sqrt{1}=1\) dollar.
Step 2: Later interval. The signed sum is \(3-3+4-4=0\), so the mean signed error is also \(0/4=0\). The squared errors are \(9,9,16,16\); their sum is 50. Thus \(\mathrm{RMSE}=\sqrt{50/4}=\sqrt{12.5}\approx3.536\) dollars, rounded to three decimal places.
Interpretation and check. Both intervals have zero mean signed error, but the later errors are larger in magnitude: the RMSE rises from 1 dollar to about 3.536 dollars. The time plot would show wider swings around zero in the later interval. The common mistake is to conclude that performance is unchanged because both signed means are zero; checking the squared errors confirms a clear increase in error magnitude.
Common Traps and Practical Interpretation
- Reversing the error sign. Under this tutorial's definition, \(y_t-\hat{y}_t\), a positive error means underprediction. State the convention before interpreting a plot.
- Reporting only a whole-period average. Positive and negative runs can cancel. Compare meaningful time intervals and retain the chronological plot.
- Equating zero mean with small errors. As the third example shows, errors can average to zero while their magnitudes are substantial.
- Calling every spike a persistent failure. A single unusual observation deserves investigation, but it is not the same evidence as a run of errors on one side of zero.
- Assuming the plot identifies a cause. A temporal pattern can be associated with a change in data, exposure, process, or conditions. The plot alone cannot distinguish these explanations.
- Ignoring the observation window. A chronological validation plot describes performance on the period shown. It does not guarantee performance in a different future period.
When a pattern appears, record what it is in concrete terms: for example, “errors are mostly negative in the first interval and positive in the second,” or “the later interval has wider swings around zero.” Then check that the time ordering, units, and prediction-to-outcome matching are correct. These statements are more defensible than claiming that the model has drifted for a particular reason based on the graph alone.
Check Your Understanding
For each problem, use the signed-error convention \(e_t=y_t-\hat{y}_t\). Interpret the sequence as well as calculating the requested summary.
- A hypothetical five-period forecast has errors \((2,1,0,-1,-2)\) dollars. Calculate the overall mean signed error and describe the chronological direction of the sequence. (Answer: mean error \(0\) dollars; the sequence moves from underprediction through zero to overprediction.)
- A six-day sequence of errors is \((-1,0,1,6,0,-1)\) dollars. Calculate its mean, then calculate the mean of the other five errors after setting aside the 6-dollar point for diagnostic comparison. What does the comparison suggest? (Answer: overall mean \(5/6\approx0.8333\) dollars; other-five mean \(-1/5=-0.2\) dollars; the positive overall mean is strongly influenced by one isolated error.)
- Four early errors are \((1,-1,1,-1)\) dollars and four later errors are \((2,-2,2,-2)\) dollars. For each interval calculate the mean signed error and RMSE. State what changes over time. (Answer: both mean errors are \(0\); early RMSE is \(1\) dollar and later RMSE is \(2\) dollars, so error magnitude increases without a change in signed mean.)
- A hypothetical quarterly sequence is \((0,1,2,3,0,1,2,3)\) dollars. Compare the mean of quarters 1–4 with the mean of quarters 5–8. What feature should the time plot prompt you to investigate? (Answer: both means are \(1.5\) dollars; the repeated within-block rise suggests a recurring pattern rather than a change in the two block averages.)
- Eight consecutive errors are \((0,0,0,0,4,4,4,4)\) dollars. Calculate the mean error in the first four periods, the last four periods, and all eight periods. Interpret why the overall mean is incomplete. (Answer: first-four mean \(0\), last-four mean \(4\), overall mean \(2\) dollars; the overall mean conceals a sustained shift from zero errors to persistent underprediction.)