Why Break Forecast Errors Into Groups?
A forecasting model can have a reasonable average error and still perform poorly in a way that matters. It might be accurate one step ahead but deteriorate at longer leads, or it might miss the same seasonal peak year after year. A single validation score averages across these situations and can hide the pattern. Grouping errors by forecast horizon and seasonal position makes the pattern visible.
This evaluation uses forecasts produced at validation origins: time points at which the model is fitted or updated using only information available up to that point. For an origin \(o\), let \(\hat{y}_{o+h|o}\) be the forecast for the observation \(h\) periods ahead, and let \(y_{o+h}\) be the subsequently observed value. The forecast error is the observed value minus the forecast. This sign convention makes a positive error an underforecast and a negative error an overforecast.
For a fixed horizon, calculate an error summary over all validation forecasts at that lead. For a fixed seasonal position, summarize errors whose targets fall at that point in the seasonal cycle. A two-way breakdown examines horizon and target season together. For example, it can show whether a model is especially inaccurate when forecasting a particular month several steps ahead.
Summaries Within Each Group
Common summaries include mean absolute error (MAE), root mean squared error (RMSE), and mean error. For a group \(G\) containing \(n_G\) errors, these are:
MAE summarizes typical absolute error in the original measurement units. RMSE gives relatively more weight to large errors because errors are squared before averaging. Mean error retains direction: a positive value indicates average underforecasting, and a negative value indicates average overforecasting. A mean error near zero does not establish accuracy; positive and negative errors can cancel.
The set \(G\) might contain all forecasts at horizon \(h\), all forecasts targeting seasonal position \(s\), or all forecasts with both a particular \(h\) and \(s\). Report the number of errors in each group as well as its score. A group based on few validation targets can have an unstable summary, so a seemingly striking result may reflect limited evidence.
Worked Example: Detecting Deterioration Across Horizons
A hypothetical monthly claims series is evaluated at four rolling origins. At each origin, errors are recorded for horizons 1, 2, and 3. The signed errors, in thousands of dollars, are as follows:
| Horizon | Errors (thousands of dollars) |
|---|---|
| 1 | 1, -2, 3, -2 |
| 2 | 4, -4, 2, -2 |
| 3 | 6, -6, 4, -4 |
Method. Calculate the mean error, MAE, and RMSE separately for each horizon. This distinguishes signed bias from error magnitude and avoids assuming that a pooled score describes every lead equally well.
At horizon 1, the signed-error sum is \(1-2+3-2=0\), so the mean error is \(0/4=0\). The absolute errors are \(1,2,3,2\), whose sum is \(8\), giving \(\operatorname{MAE}_1=8/4=2\). The squared errors sum to \(1+4+9+4=18\), so \(\operatorname{RMSE}_1=\sqrt{18/4}=\sqrt{4.5}\approx2.121\).
At horizon 2, the signed-error sum is \(4-4+2-2=0\). The absolute errors sum to \(4+4+2+2=12\), giving \(\operatorname{MAE}_2=12/4=3\). The squared errors sum to \(16+16+4+4=40\), so \(\operatorname{RMSE}_2=\sqrt{40/4}=\sqrt{10}\approx3.162\).
At horizon 3, the signed-error sum is \(6-6+4-4=0\). The absolute errors sum to \(6+6+4+4=20\), giving \(\operatorname{MAE}_3=20/4=5\). The squared errors sum to \(36+36+16+16=104\), so \(\operatorname{RMSE}_3=\sqrt{104/4}=\sqrt{26}\approx5.099\).
Check. Each horizon has four errors and zero mean error, while both MAE and RMSE rise with horizon. The zero mean errors arise because positive and negative values cancel; they do not indicate zero forecast error. MAEs and RMSEs are rounded to three decimals. All errors and summaries are in thousands of dollars.
Seasonal Position and the Horizon-by-Season View
For a series with seasonal period \(m\), seasonal position can be assigned by numbering the observations in the cycle \(1,\ldots,m\). If time \(t\) has position \(s(t)\), then \(s(t)\) repeats every \(m\) periods. A forecast's seasonal position is determined by its target time. Thus, with monthly data, a three-month-ahead forecast made in January targets April, not January.
A seasonal summary can reveal recurring errors even if error size does not increase with lead. A two-way table of horizon and target seasonal position is more specific: it can distinguish a general difficulty forecasting winter months from a difficulty forecasting winter months only at longer leads. Use this detail when each cell has enough validation observations to support interpretation.
Worked Example: Finding a Recurring Seasonal Miss
A hypothetical quarterly benefit-payment forecast is evaluated at several origins. The errors below are grouped by the target quarter, with two validation errors in each group. Amounts are in thousands of dollars:
| Target quarter | Errors |
|---|---|
| Q1 | 2, -2 |
| Q2 | 1, 3 |
| Q3 | -4, 0 |
| Q4 | -1, -5 |
Method. For each target quarter, find the mean error and MAE. The mean error shows direction, while MAE shows magnitude without cancellation.
For Q1, the mean error is \((2-2)/2=0\). Its MAE is \((|2|+|-2|)/2=(2+2)/2=2\).
For Q2, the mean error is \((1+3)/2=2\). Its MAE is \((1+3)/2=2\).
For Q3, the mean error is \((-4+0)/2=-2\). Its MAE is \((4+0)/2=2\).
For Q4, the mean error is \((-1-5)/2=-3\). Its MAE is \((1+5)/2=3\).
Check. Q4 has the largest MAE, \(3\) thousand dollars, and its negative mean error, \(-3\) thousand dollars, indicates overforecasting on average. Q1's mean error is zero despite a nonzero MAE, illustrating why signed error alone is insufficient. These values are exact; no rounding is needed.
A seasonal pattern should be interpreted in context rather than treated as proof of a fixed mechanism. Validation errors may vary because of a small number of observations, unusual events, or changes in the data-generating process. The grouped summaries locate a pattern for investigation; they do not, by themselves, establish its cause.
How Grouped and Overall Scores Relate
If groups partition the same set of forecast errors, the pooled MAE is a weighted average of group MAEs. The weights are the group sample sizes divided by the total sample size. This matters when some horizons or seasonal positions occur more often in validation data than others.
An equally weighted average of group MAEs answers a different question: it gives each group the same influence regardless of how often it occurs in the validation set. That may be useful when groups represent equally important decisions, but it is not the pooled MAE. State which summary is being used and why. For RMSE, the pooled value is not the weighted average of group RMSEs; pool the squared errors first, then take the square root.
Worked Example: Comparing Models When Group Sizes Differ
A hypothetical monthly reserve forecast is evaluated in four horizon-by-season cells. Each cell is a distinct combination of short or long horizon and quiet or peak season. The table gives each cell's number of errors and MAE, in thousands of dollars, for two candidate models.
| Cell | Number of errors | Model A MAE | Model B MAE |
|---|---|---|---|
| Short horizon, quiet season | 8 | 1 | 2 |
| Short horizon, peak season | 2 | 5 | 3 |
| Long horizon, quiet season | 2 | 2 | 3 |
| Long horizon, peak season | 8 | 8 | 4 |
Method. First compute pooled MAE using cell counts as weights. Then compute the equally weighted mean of the four cell MAEs to compare performance when each cell is assigned equal importance.
There are \(8+2+2+8=20\) errors. For Model A, the pooled absolute-error total represented by the cell summaries is \(8(1)+2(5)+2(2)+8(8)=8+10+4+64=86\). Its pooled MAE is \(86/20=4.3\) thousand dollars. For Model B, the total is \(8(2)+2(3)+2(3)+8(4)=16+6+6+32=60\), giving pooled MAE \(60/20=3.0\) thousand dollars.
With equal weight for each cell, Model A's average is \((1+5+2+8)/4=16/4=4\). Model B's is \((2+3+3+4)/4=12/4=3\). Both summaries favor Model B overall, but the cell scores show Model A is better in the short-horizon quiet-season cell and the long-horizon quiet-season cell. Model B is better in both peak-season cells.
Check. The overall comparison does not mean Model B wins in every condition. If peak-season forecasts carry greater decision importance, its advantage in those cells may be especially relevant; if not, the decision requires an explicitly justified weighting scheme. The calculation uses cell MAEs and counts, not a simple average of the four cells as though their sizes were equal.
Common Traps and a Practical Evaluation Sequence
- Grouping by the origin's season rather than the target's season. An \(h\)-step forecast should be assigned to the seasonal position of \(o+h\). Otherwise, errors can be attributed to the wrong period.
- Using mean error as an accuracy score. Opposite-signed errors cancel. Report an absolute or squared-error measure alongside signed error.
- Comparing averages without checking group counts. A seasonal or horizon score based on very few observations may be unreliable. Show the count and interpret it cautiously.
- Confusing a pooled average with an equal-cell average. Pooled MAE weights each error equally; equal-cell averaging weights each chosen group equally. They answer different questions.
- Allowing validation information into model construction. For a meaningful evaluation, each forecast must be generated from information available at its origin. Model fitting, preprocessing, or tuning that uses future validation observations can make errors look too small.
A useful evaluation sequence is to verify the error sign and units, tabulate observations by horizon and target season, calculate an appropriate magnitude measure and signed mean in each group, and then compare those results with the pooled score. Rolling-origin validation and leakage-safe preprocessing are essential to trustworthy forecasts; grouped summaries cannot repair a validation design that used future information.
Check Your Understanding
For each problem, identify the relevant grouping before calculating. Error is observed minus forecast, so positive values indicate underforecasting.
- A quarterly model has errors \(3,-1,2\) at horizon 1 and errors \(5,-3,4\) at horizon 4, all in thousands of dollars. Find the MAE at each horizon and identify which horizon has greater average absolute error. (Answer: Horizon 1 MAE \(=2\) thousand dollars; horizon 4 MAE \(=4\) thousand dollars, so horizon 4 is larger.)
- A monthly forecast has two errors for a particular target month: \(-6\) and \(2\), in thousands of dollars. Find the mean error and MAE, and interpret the sign of the mean error. (Answer: Mean error \(=(-6+2)/2=-2\) thousand dollars, indicating average overforecasting; MAE \(=(6+2)/2=4\) thousand dollars.)
- A seasonal series has period \(m=4\), and observations are numbered so that time \(t=1\) is position 1. A forecast is made at \(o=10\) for horizon \(h=3\). What is the target time and seasonal position? (Answer: Target time \(t=13\); since \(13\) is position 1 in a four-period cycle, the seasonal position is 1.)
- Two disjoint validation groups have sizes 6 and 4 and MAEs of 2 and 5, respectively, in the same units. Find the pooled MAE and the equally weighted average of group MAEs. (Answer: Pooled MAE \(=[6(2)+4(5)]/10=3.2\); equally weighted average \(=(2+5)/2=3.5\).)
- For a target season, a model's four signed errors are \(-2,-2,2,2\). A second model's errors are \(1,1,1,1\). Compare their mean errors and MAEs, and explain why mean error alone is insufficient. (Answer: Both mean errors are \(0\), but the first model's MAE is \(2\) and the second's is \(1\); the signed errors cancel for the first model, so equal mean error does not imply equal accuracy.)