Why Compare With a Benchmark?
A forecasting model can have errors that look small in isolation but still fail to improve on a simple, credible alternative. A benchmark makes the comparison interpretable: it answers whether a candidate forecast reduces error relative to a baseline that could have been used instead. This is useful when assessing whether a more elaborate time-series model adds practical value on held-out observations.
The benchmark must suit the series and the forecasting task. For a series without a repeating seasonal pattern, a naïve forecast is a common reference. For a series with a stable seasonal cycle, a seasonal-naïve forecast may be more demanding and more credible. Neither baseline is automatically appropriate in every setting; the choice should reflect the structure and information available when the forecasts would have been made.
For example, with monthly data and \(m=12\), a seasonal-naïve forecast for next January uses the most recently observed January value. For a horizon longer than one seasonal cycle, the formula continues to use the latest available observation from the target month. This method repeats a seasonal pattern; it does not extrapolate a trend.
Relative Loss and Skill
Let \(\mathcal I\) be the set of forecast cases being evaluated. Each case specifies a forecast origin and a horizon, and has an observed value \(y_j\), a candidate forecast \(\hat y_j\), and a benchmark forecast \(\hat y_{B,j}\). Compare the forecasts on the same cases. Otherwise differences in missing forecasts, origins, or horizons can be mistaken for differences in model quality.
Two common nonnegative loss summaries are mean absolute error (MAE) and root mean squared error (RMSE). With \(n\) evaluation cases, define the absolute forecast error as \(|y_j-\hat y_j|\) and the squared forecast error as \((y_j-\hat y_j)^2\). Compute the chosen loss for the candidate and the benchmark separately:
Use the same definitions with the benchmark forecasts to obtain \(\operatorname{MAE}_{B}\) or \(\operatorname{RMSE}_{B}\). MAE and RMSE are measured in the units of the series. RMSE gives relatively greater influence to large errors because it squares them before averaging.
A ratio of \(0.75\) means that the candidate has 75% of the benchmark's loss. The corresponding skill is \(1-0.75=0.25\), or a 25% reduction in loss. State which loss measure and which benchmark you used: a model may have positive skill under MAE but lower or negative skill under RMSE if its occasional large errors matter.
Why the Skill Formula Measures Improvement
The skill score is the reduction in loss divided by the benchmark loss. If the candidate loss is \(L_M\) and the benchmark loss is \(L_B>0\), the reduction is \(L_B-L_M\). Dividing this reduction by \(L_B\) gives the fraction of benchmark loss removed:
This also establishes the sign interpretation. If \(L_M<L_B\), then \(L_B-L_M>0\), so \(S>0\). If \(L_M=L_B\), then \(S=0\). If \(L_M>L_B\), then \(S<0\). The derivation applies to a nonnegative loss summary with a positive benchmark value; it does not make a zero benchmark loss usable as a denominator.
Worked Example: Comparing a Model With a Naïve Forecast
A hypothetical analyst evaluates monthly claim-count forecasts at four one-month-ahead origins. The actual counts are \(104, 108, 111,\) and \(115\). The naïve benchmark forecasts are \(99, 110, 106,\) and \(119\); a candidate model forecasts \(102, 112, 108,\) and \(116\). Calculate MAE and RMSE for each method, then calculate the candidate's relative loss ratio and skill under both measures.
Method. Use the same four actual values and forecast origins for both methods. Define each signed error as actual minus forecast; the losses use absolute errors or squared errors, so the sign does not affect the result.
For the candidate, the signed errors are \(104-102=2\), \(108-112=-4\), \(111-108=3\), and \(115-116=-1\). The absolute errors are \(2,4,3,1\), whose sum is \(10\). Thus \(\operatorname{MAE}_M=10/4=2.5\). The squared errors are \(4,16,9,1\), whose sum is \(30\), so \(\operatorname{RMSE}_M=\sqrt{30/4}=\sqrt{7.5}\approx2.7386\).
For the naïve benchmark, the signed errors are \(104-99=5\), \(108-110=-2\), \(111-106=5\), and \(115-119=-4\). The absolute errors sum to \(5+2+5+4=16\), so \(\operatorname{MAE}_B=16/4=4\). The squared errors sum to \(25+4+25+16=70\), so \(\operatorname{RMSE}_B=\sqrt{70/4}=\sqrt{17.5}\approx4.1833\).
The MAE ratio is \(2.5/4=0.6250\), and the MAE skill is \(1-0.6250=0.3750\). The RMSE ratio is \(\sqrt{7.5}/\sqrt{17.5}=\sqrt{3/7}\approx0.6547\), and the RMSE skill is \(1-0.6547=0.3453\). Rounded to four decimals, the model reduces MAE by 37.50% and RMSE by about 34.53% relative to this benchmark.
Check. Both ratios are below \(1\) and both skill scores are positive, consistent with the candidate's lower losses. The different scores also reflect that MAE and RMSE summarize errors differently.
Seasonal Benchmarks and Forecast Horizon
A seasonal-naïve forecast is evaluated just like any other forecast, but its construction depends on the target season and horizon. In the next example, the seasonal period is four observations. For horizons one through four from the same origin, the benchmark uses the four most recent observations from the corresponding seasons. The candidate and benchmark must still be scored against the same actual outcomes.
Worked Example: Scoring Against a Seasonal-Naïve Forecast
A hypothetical quarterly portfolio has a seasonal period \(m=4\). At origin \(t=8\), its observed values are \(y_1,\ldots,y_8=(10,15,12,18,12,17,14,20)\). For horizons \(h=1,2,3,4\), the actual values are \(13,18,15,22\), and a candidate model forecasts \(14,17,16,21\). Find the seasonal-naïve forecasts, then compare the candidate and benchmark using MAE and RMSE.
Seasonal benchmark. For \(m=4\), the benchmark at horizon \(h\) is \(y_{8+h-4\lceil h/4\rceil}\). For \(h=1\), the index is \(8+1-4=5\), giving \(y_5=12\). For \(h=2\), the index is \(6\), giving \(y_6=17\). For \(h=3\), it is \(7\), giving \(y_7=14\). For \(h=4\), it is \(8\), giving \(y_8=20\). The benchmark forecasts are therefore \(12,17,14,20\).
Candidate loss. Actual minus candidate forecast gives errors \(-1,1,-1,1\). The absolute errors sum to \(4\), so \(\operatorname{MAE}_M=4/4=1\). The squared errors are \(1,1,1,1\), so \(\operatorname{RMSE}_M=\sqrt{4/4}=1\).
Benchmark loss. Actual minus seasonal-naïve forecast gives errors \(1,1,1,2\). The absolute errors sum to \(5\), so \(\operatorname{MAE}_B=5/4=1.25\). The squared errors sum to \(1+1+1+4=7\), so \(\operatorname{RMSE}_B=\sqrt{7/4}=\sqrt{1.75}\approx1.3229\).
The MAE ratio is \(1/1.25=0.8000\), giving skill \(1-0.8000=0.2000\). The RMSE ratio is \(1/\sqrt{1.75}\approx0.7559\), giving skill \(1-0.7559=0.2441\). Rounded to four decimals, the candidate's skill is \(0.2000\) under MAE and \(0.2441\) under RMSE.
Check. Both scores indicate improvement on these four cases. The last benchmark error is larger than the others, so RMSE gives that error extra weight, which is why its skill differs from the MAE skill.
Interpreting Scores When Models Differ in Complexity
A relative score is evidence about performance on the evaluation cases, not proof that a more complex model will perform better in the future. For a fair comparison, models and baseline should use the same forecast origins, horizons, and information available at each origin. For multi-horizon forecasts, calculate scores separately by horizon when horizon-specific performance matters. A pooled score is also possible, but the number of cases at each horizon then determines how much that horizon contributes unless explicit weights are used.
The following example compares two candidate models with a naïve benchmark on six one-step-ahead validation cases. The forecasts are given so that every method is scored on the same observations.
Worked Example: Does Added Complexity Improve on the Baseline?
A hypothetical analyst compares a simple model and a more complex model on six one-step-ahead validation cases. The actual values are \(50,55,52,60,58,65\). The respective benchmark forecasts are \(48,51,49,55,54,59\); simple-model forecasts are \(49,52,50,56,55,60\); and complex-model forecasts are \(48,54,50,57,54,62\). Compare their MAEs and benchmark-relative skill scores.
Method. Compute absolute errors case by case, add them, and divide by six. Since the benchmark is common, both skill scores use its MAE as the denominator.
| Case | Actual | Benchmark forecast | Simple forecast | Complex forecast | Benchmark absolute error | Simple absolute error | Complex absolute error |
|---|---|---|---|---|---|---|---|
| 1 | 50 | 48 | 49 | 48 | 2 | 1 | 2 |
| 2 | 55 | 51 | 52 | 54 | 4 | 3 | 1 |
| 3 | 52 | 49 | 50 | 50 | 3 | 2 | 2 |
| 4 | 60 | 55 | 56 | 57 | 5 | 4 | 3 |
| 5 | 58 | 54 | 55 | 54 | 4 | 3 | 4 |
| 6 | 65 | 59 | 60 | 62 | 6 | 5 | 3 |
The benchmark absolute errors sum to \(2+4+3+5+4+6=24\), so its MAE is \(24/6=4\). The simple-model errors sum to \(1+3+2+4+3+5=18\), giving MAE \(18/6=3\). Its relative loss ratio is \(3/4=0.7500\), and its skill is \(1-0.7500=0.2500\).
The complex-model errors sum to \(2+1+2+3+4+3=15\), giving MAE \(15/6=2.5\). Its ratio is \(2.5/4=0.6250\), and its skill is \(1-0.6250=0.3750\). On these validation cases, the complex model has lower MAE than both the benchmark and the simple model.
Check. The complex model has MAE \(2.5<3<4\), matching the ordering of its positive skill, the simple model's positive skill, and the benchmark's zero skill by definition. This comparison supports the complex model on this evaluation set; it does not establish that added complexity will always help.
Common Mistakes and Practical Checks
- Reversing the score interpretation. The relative loss ratio is lower when the candidate is better; the skill score is higher when the candidate is better. They are related but are not the same quantity.
- Choosing a weak or mismatched baseline. A model can beat a naïve forecast on a strongly seasonal series and still fail to beat a seasonal-naïve forecast. State why the selected benchmark is credible for the series.
- Comparing different cases. A candidate evaluated on easy origins and a benchmark evaluated on difficult origins do not provide a fair relative score. Match origins and horizons before calculating either loss.
- Pooling horizons without noticing the weighting. A pooled MAE over many short-horizon cases and a few long-horizon cases gives more weight to the short horizon. Report horizon-specific scores or explain the pooling and weights.
- Dividing by a zero benchmark loss. If \(L_B=0\), \(L_M/L_B\) and \(1-L_M/L_B\) are undefined. Report the losses directly and explain that the benchmark made no errors on the evaluation set.
A particularly common exam trap is to call the relative loss ratio itself a percentage improvement. In the first worked example, \(2.5/4=0.6250\) means the candidate's MAE is 62.50% of the benchmark MAE; it does not mean a 62.50% improvement. The improvement is \(1-0.6250=0.3750\), or 37.50%. Keep the ratio and skill definitions distinct, and name the loss measure and benchmark whenever reporting a score.
Check Your Understanding
For each problem, compute the requested score using the stated benchmark and loss measure.
- A model has MAE \(3.5\) and a naïve benchmark has MAE \(5\) on the same one-step-ahead cases. Find the relative loss ratio and skill. (Answer: ratio \(0.7000\); skill \(0.3000\))
- A quarterly series has seasonal period \(m=4\) and is observed through \(t=30\). Using the seasonal-naïve definition, which observations supply the forecasts for horizons \(h=2\) and \(h=5\)? (Answer: \(y_{28}\) for \(h=2\) and \(y_{27}\) for \(h=5\))
- A candidate has RMSE \(4\), while its benchmark has RMSE \(5\), computed on the same cases. Find the RMSE ratio and skill. (Answer: ratio \(0.8000\); skill \(0.2000\))
- Across the same number of cases at each horizon, a benchmark has MAE \(2\) at horizon one and \(8\) at horizon two. A candidate has MAE \(3\) at horizon one and \(6\) at horizon two. Find the pooled MAE for each method and the pooled skill against the benchmark. (Answer: benchmark MAE \(5\); candidate MAE \(4.5\); skill \(0.1000\))
- A seasonal-naïve benchmark has zero MAE on a validation set, while a candidate has positive MAE. Can the relative MAE ratio and skill be calculated, and how should the result be reported? (Answer: no; the ratio and skill are undefined because the benchmark MAE is zero, so report both MAEs directly)