Why Forecast Accuracy Depends on the Horizon
A forecast made one period ahead answers a different question from a forecast made twelve periods ahead. A model may be useful for adjusting next week’s staffing while being less useful for setting next year’s budget. A single score that combines both forecasts can conceal this difference. For model comparison, accuracy should first be examined separately at each lead time.
A forecast origin is the time at which a forecast is made. The horizon, or lead time, is the number of periods between that origin and the target observation. For example, a forecast made at the end of month \(t\) for month \(t+3\) has horizon \(h=3\). The observed value is compared with a forecast made using information available at the origin.
It is important to match each forecast with the observation it was intended to predict. If the same origin produces forecasts for horizons one, two, and three, those are three different forecast-and-observation pairs. They should not be treated as interchangeable repetitions of a single forecast.
Scoring Each Horizon Separately
For a fixed horizon \(h\), the mean absolute error and root mean squared error summarize the errors over the evaluation origins. If there are \(n_h\) evaluated forecasts at that horizon, these measures are
Both measures are in the units of the outcome. MAE gives each absolute error a cost proportional to its size, whereas RMSE gives relatively more influence to large errors because it squares them before averaging. The same choice of metric should be used when comparing models at a given horizon, and the models should be assessed on comparable origins and target observations.
Worked Example: Calculating Scores at Two Horizons
A hypothetical insurer forecasts the monthly number of roadside-assistance claims from several forecast origins. At three origins, the actual counts and forecasts from two models are as follows. Each entry is a count of claims; the rows at each horizon compare forecasts for the same target months.
| Horizon | Actuals | Model A forecasts | Model B forecasts |
|---|---|---|---|
| 1 month | 100, 110, 90 | 98, 112, 91 | 99, 107, 94 |
| 2 months | 105, 95, 102 | 101, 98, 100 | 104, 97, 105 |
Method. For each horizon, subtract each forecast from its actual value, then calculate MAE and RMSE using only that horizon’s errors.
At one month, Model A’s errors are \(100-98=2\), \(110-112=-2\), and \(90-91=-1\). Its absolute errors sum to \(2+2+1=5\), and its squared errors sum to \(2^2+(-2)^2+(-1)^2=9\). Therefore,
At the same horizon, Model B’s errors are \(100-99=1\), \(110-107=3\), and \(90-94=-4\). Its absolute errors sum to \(1+3+4=8\), and its squared errors sum to \(1+9+16=26\). Thus,
At two months, Model A’s errors are \(105-101=4\), \(95-98=-3\), and \(102-100=2\). The absolute-error sum is \(4+3+2=9\); the squared-error sum is \(16+9+4=29\). Its scores are \(\operatorname{MAE}_{2,A}=9/3=3\) and \(\operatorname{RMSE}_{2,A}=\sqrt{29/3}=3.109\).
Model B’s two-month errors are \(105-104=1\), \(95-97=-2\), and \(102-105=-3\). The absolute-error sum is \(1+2+3=6\); the squared-error sum is \(1+4+9=14\). Its scores are \(\operatorname{MAE}_{2,B}=6/3=2\) and \(\operatorname{RMSE}_{2,B}=\sqrt{14/3}=2.160\).
To three decimal places, Model A has lower error at one month under both metrics, while Model B has lower error at two months. The comparison changes with the horizon. Common mistake: selecting Model A from the one-month results and claiming it is more accurate overall. The calculation supports that conclusion only for the one-month horizon.
Why Forecast Errors Can Change With Lead Time
Even when a time-series model is appropriate, uncertainty in its forecast often increases with the horizon. A simple justification comes from a stationary AR(1) model with known parameters:
where the innovations \(\varepsilon_t\) are uncorrelated, have mean zero, and have common variance \(\sigma^2\). Assume the information available at the forecast origin includes the current value \(y_t\), and the model parameters are treated as known. Iterating the model forward gives an \(h\)-step forecast error of
Because the innovations at these future times are uncorrelated, the variance of their weighted sum is the sum of their variances multiplied by the squared weights. Therefore,
The geometric-sum identity applies because \(|\phi|<1\), so \(1-\phi^2\neq 0\). Increasing the horizon from \(h\) to \(h+1\) adds the nonnegative term \(\sigma^2\phi^{2h}\) to the variance. This explains why a model’s forecast errors can become more dispersed at longer leads, even when its structure is unchanged. The result depends on the stated AR(1) assumptions and known parameters; it is not a universal formula for every time series.
Different models can also have different error patterns across horizons. For example, one model may track short-lived changes well but fail to represent a persistent trend. Another may be less responsive to immediate fluctuations but produce more stable longer-lead forecasts. The variance result above explains one source of horizon dependence; it does not establish which model will rank better in a particular evaluation.
Worked Example: A Ranking Reversal Hidden by a Single Score
A hypothetical utility compares two models for forecasting daily water demand at horizons of one day and four days. Each model is evaluated at four origins, with the same number of forecasts at each horizon. The signed errors are listed below.
| Model | One-day errors | Four-day errors |
|---|---|---|
| A | 1, -1, 1, -1 | 8, -8, 8, -8 |
| B | 2, -2, 2, -2 | 5, -5, 5, -5 |
At one day, Model A has absolute errors \(1,1,1,1\). Its absolute-error sum is \(4\), so \(\operatorname{MAE}_{1,A}=4/4=1\). Model B’s absolute-error sum is \(2+2+2+2=8\), so \(\operatorname{MAE}_{1,B}=8/4=2\). Model A performs better at this horizon.
At four days, Model A has absolute errors \(8,8,8,8\), giving \(\operatorname{MAE}_{4,A}=32/4=8\). Model B has absolute errors \(5,5,5,5\), giving \(\operatorname{MAE}_{4,B}=20/4=5\). Model B performs better at this horizon.
If all eight errors are pooled with equal weight, Model A’s MAE is \((4+32)/8=4.5\), and Model B’s is \((8+20)/8=3.5\). The pooled result favors B, but it does not describe A’s advantage at the one-day lead. Common mistake: treating the pooled score as if it answered every horizon-specific question. The correction is to report each horizon’s score first and use a combined score only for a decision that genuinely combines those horizons.
When and How to Aggregate Across Horizons
Aggregation is appropriate when the decision being supported assigns value to forecasts at more than one horizon. The weights should represent that decision, not merely make the calculation convenient. For example, if the operational objective is to minimize average absolute error across selected lead times, a weighted score can be defined as
where \(\mathcal{H}\) is the set of horizons in scope. This score retains the outcome’s units. It represents the stated objective only if the weights are meaningful for that objective. A weighted average of horizon-specific RMSEs is likewise possible, but it is not the same calculation as pooling all squared errors and taking one square root.
Worked Example: Choosing Weights From an Operational Objective
A hypothetical maintenance team evaluates two models at one-month and three-month horizons. It calculates the following horizon-specific MAEs, in thousands of dollars of monthly repair cost:
| Model | One-month MAE | Three-month MAE |
|---|---|---|
| A | 1 | 7 |
| B | 3 | 4 |
Method. Calculate a weighted MAE using weights that match the stated operational priority. Suppose the team’s first plan gives one-month forecasts 60% of the total importance and three-month forecasts 40%. The weights are \(w_1=0.6\) and \(w_3=0.4\); they are nonnegative and sum to \(1\).
For Model A, the weighted score is \(0.6(1)+0.4(7)=0.6+2.8=3.4\) thousand dollars. For Model B, it is \(0.6(3)+0.4(4)=1.8+1.6=3.4\) thousand dollars. The models tie under this objective.
Now suppose the team’s decision changes: longer-lead forecasts receive 75% of the importance, so \(w_1=0.25\) and \(w_3=0.75\). Model A’s score is \(0.25(1)+0.75(7)=0.25+5.25=5.5\) thousand dollars. Model B’s score is \(0.25(3)+0.75(4)=0.75+3=3.75\) thousand dollars. Model B now has the lower weighted MAE.
The ranking changed because the operational priorities changed, not because the underlying errors did. These calculations assume the stated weights accurately represent the decision. Common mistake: choosing weights after seeing which model they favor. Specify the relevant horizons and their importance based on the use of the forecasts, then apply the same weights to every candidate model.
Interpreting Horizon-Specific Comparisons
A useful comparison reports the scores by horizon before offering an overall recommendation. A table is often clearer than one headline number, particularly when the rankings differ. Report the number of evaluated forecasts at each horizon as well: the available set \(\mathcal{O}_h\) can vary with \(h\), especially near the end of a time series where longer-lead outcomes may not yet be observed.
When horizons have different numbers of evaluated forecasts, pooling every error gives more influence to horizons with more available observations. That may be intentional, but it is not equivalent to giving each horizon equal weight. Similarly, averaging horizon scores equally gives each horizon equal importance regardless of its number of forecasts. Neither choice is automatically correct; the appropriate calculation follows from the objective.
- Compare like with like. At a given horizon, use forecasts for the same target observations when comparing models.
- Keep the metric explicit. MAE and RMSE can rank models differently because RMSE responds more strongly to large errors.
- Show the horizon profile. A single aggregate can hide a model’s weakness at the lead time that matters most.
- Explain the weights. If results are aggregated, state how horizons are weighted and why those weights fit the decision.
This approach is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus: calculate forecast accuracy at the relevant leads, interpret model comparisons in context, and avoid claiming that one aggregate ranking answers every forecasting question.
Check Your Understanding
For each problem, calculate or interpret the scores at the stated horizons. Round probabilities only when applicable; these problems use forecast errors and MAE.
- A model has one-month errors \(3,-1,2,-2\) for four evaluated origins. Calculate its one-month MAE and RMSE. (Answer: MAE \(=2\); RMSE \(=\sqrt{4.5}\approx2.121\).)
- At a two-month horizon, Model C has errors \(1,-3,2\), while Model D has errors \(2,-2,1\), on the same three targets. Which model has the lower MAE at this horizon? (Answer: Model D; its MAE is \(5/3\approx1.667\), compared with Model C’s MAE of \(2\).)
- A team gives a one-month horizon weight of \(0.6\) and a three-month horizon weight of \(0.4\). Model E has MAEs \(1\) and \(7\); Model F has MAEs \(3\) and \(4\). Calculate each weighted MAE and state the ranking. (Answer: Model E \(=3.4\), Model F \(=3.4\); the models tie.)
- At horizon one, a model has four absolute errors summing to \(4\). At horizon three, it has three absolute errors summing to \(18\). Calculate the MAE at each horizon, the pooled MAE across all seven errors, and the equally weighted average of the two horizon MAEs. (Answer: Horizon-one MAE \(=1\); horizon-three MAE \(=6\); pooled MAE \(=22/7\approx3.143\); equally weighted average \(=3.5\).)
- A planning group uses one-month and six-month forecasts. The one-month MAEs are 2 for Model G and 3 for Model H; the six-month MAEs are 8 for G and 5 for H. If the six-month horizon receives weight \(0.75\), with the remaining weight assigned to one month, calculate the weighted MAEs and select the lower-scoring model. (Answer: Model G \(=0.25(2)+0.75(8)=6.5\); Model H \(=0.25(3)+0.75(5)=4.5\); Model H has the lower weighted MAE.)