From Observations to a Defensible Forecast
A forecast is not justified merely because a model produces a number. A defensible analysis preserves chronology, checks whether the series is suitable for the proposed model, evaluates competing specifications, and tests predictive performance without using future observations to make earlier decisions. These steps matter when a forecasting error affects a reserve estimate, staffing plan, or hypothetical insurance product decision.
The case below uses a hypothetical monthly index of average claim handling costs. It is a short, nonseasonal series intended to illustrate the reasoning, not represent actual company or industry data. The central question is the next few monthly index values. The earlier material on ARIMA models, residual diagnostics, forecast evaluation, and leakage provides the underlying tools; here they are applied as one connected workflow.
Worked Case: Establishing the Time-Series Structure
The observed index values are \(y_1,\ldots,y_{20}\). Each value is recorded once per month, in chronological order. The observations are hypothetical.
| Month \(t\) | 1–5 | 6–10 | 11–15 | 16–20 |
|---|---|---|---|---|
| Index values | 50, 52, 51, 54, 55 | 57, 56, 59, 60, 59 | 62, 64, 63, 66, 68 | 67, 70, 72, 71, 74 |
Worked Example: Decide Whether to Difference Before Fitting
Method. First inspect the ordered levels for trend or changing behavior, then inspect first differences for a more stable pattern. The candidate ARIMA models below use first differences, so the relevant transformation is \(\Delta y_t=y_t-y_{t-1}\).
Step 1: Preserve and inspect chronology. The index rises from 50 at month 1 to 74 at month 20, with some reversals but no sustained return to a fixed level. A plot of the ordered levels would show a clear upward movement. Treating the 20 values as an unordered sample would erase the information needed to identify that behavior.
Step 2: Calculate the changes. Subtract each value from the next. The 19 first differences are \(2,-1,3,1,2,-1,3,1,-1,3,2,-1,3,2,-1,3,2,-1,3\). Their sum must equal the last level minus the first: \(24=74-50\), which checks the subtraction. Their mean is therefore \(24/19=1.263\) index points per month, to three decimals.
Step 3: Assess the transformation. The differences fluctuate around a positive average without the sustained level trend visible in \(y_t\). For this hypothetical case, suppose the level plot and autocorrelation pattern show slow decay, while the differenced-series plot and unit-root test provide evidence consistent with stationarity after one difference. No stable annual repetition is evident, so there is no case here for seasonal differencing. With only 20 observations, these diagnostics are evidence rather than certainty.
Conclusion and trap avoided. One difference is a reasonable working choice, so compare nonseasonal ARIMA models with \(d=1\). The incorrect shortcut is to difference repeatedly simply because the original series trends. Extra differencing can remove useful structure and create artificial dependence; use the smallest differencing order that makes the series reasonably stable.
Specify Candidates, Then Check Their Residuals
An ARIMA\((p,d,q)\) model applies an autoregressive moving-average model to a series differenced \(d\) times. For this case, compare three candidates with one difference. A simple drift model has constant expected change. An ARIMA\((1,1,0)\) model allows the current change to depend on the previous change, and an ARIMA\((0,1,1)\) model allows dependence through the previous innovation. These are plausible alternatives, not an exhaustive search.
For any candidate, a useful fitted-model comparison combines fit with a penalty for complexity. AICc is a small-sample corrected information criterion, with \(k\) counting estimated parameters and \(n\) the number of observations used in fitting. It is defined when \(n>k+1\):
Lower AICc is preferred among the candidates being compared, but it is not a forecast-accuracy score and does not prove that the model assumptions hold. After fitting, examine residuals: their mean should be near zero, their variance should be reasonably stable, and their autocorrelations should not show a material remaining pattern. A Ljung–Box test can summarize whether a group of residual autocorrelations differs from zero; a small \(p\)-value is a warning, not a diagnosis of the precise model defect.
Worked Example: Select a Candidate Without Treating AICc as Proof
Method. Compare AICc and residual checks for the three \(d=1\) candidates. Assume the following are hypothetical fitting and diagnostic outputs from the same 20 observations, using the same estimation conventions. The Ljung–Box \(p\)-values are reported at lags 6 and 10.
| Candidate | AICc | Ljung–Box \(p\), lag 6 | Ljung–Box \(p\), lag 10 |
|---|---|---|---|
| ARIMA(0,1,0) with drift | 64.8 | 0.03 | 0.01 |
| ARIMA(1,1,0) with drift | 61.9 | 0.42 | 0.31 |
| ARIMA(0,1,1) with drift | 62.4 | 0.18 | 0.08 |
Step 1: Compare information criteria. The minimum is 61.9, for ARIMA\((1,1,0)\) with drift. Its AICc advantage over ARIMA\((0,1,1)\) is \(62.4-61.9=0.5\), which is small. Thus AICc alone does not make the choice overwhelming.
Step 2: Check residual dependence. The drift-only candidate has small \(p\)-values at both lags, indicating evidence of remaining serial dependence. The ARIMA\((1,1,0)\) candidate has larger \(p\)-values at both lags, so these tests do not identify the same warning. The ARIMA\((0,1,1)\) results are less clearly satisfactory, especially at lag 10. A \(p\)-value above a conventional threshold does not establish that residuals are independent; inspect residual plots and autocorrelations as well.
Step 3: Make a provisional choice. ARIMA\((1,1,0)\) with drift is the leading candidate because it has the lowest AICc and no flagged residual autocorrelation in these checks. It still needs out-of-sample evaluation. The common mistake is to choose the model with the lowest AICc and stop: residual diagnostics and predictive evaluation answer different questions.
Rolling-Origin Evaluation and the Final Refit
Rolling-origin evaluation imitates repeated forecasting in real time. At each origin, fit using only observations available through that date, forecast the next observation (or a specified horizon), and compare that forecast with the subsequently observed value. All candidates must be evaluated at the same origins and horizons. This prevents a favorable score from being caused by a model receiving more information or being tested on different periods.
For one-step forecasts \(y_{t_j}\) and corresponding forecasts \(\hat y_{t_j|t_j-1}\), define forecast errors \(e_j=y_{t_j}-\hat y_{t_j|t_j-1}\). The mean absolute error and root mean squared error are:
Both measures are smaller when forecasts are closer to the observations; RMSE gives relatively more weight to large errors. The choice of metric should reflect the forecasting decision, and these scores should not be interpreted without knowing the evaluation period and horizon.
Worked Example: Compare Models at Four Rolling Origins
Method. Suppose each model is refitted at origins \(t=14,15,16,17\), using only data available through that origin, to forecast months 15, 16, 17, and 18 respectively. The actual values are 68, 67, 70, and 72. The forecasts below are hypothetical model outputs from those chronological refits.
| Model | Forecasts for months 15–18 | Errors \(actual-forecast\) |
|---|---|---|
| ARIMA(0,1,0) with drift | 67, 65, 70, 70 | 1, 2, 0, 2 |
| ARIMA(1,1,0) with drift | 68.5, 66, 70.5, 71 | -0.5, 1, -0.5, 1 |
| ARIMA(0,1,1) with drift | 66, 66, 71, 69 | 2, 1, -1, 3 |
Step 1: Score the drift-only model. Its absolute errors sum to \(1+2+0+2=5\), so \(\operatorname{MAE}=5/4=1.25\). Its squared errors sum to \(1+4+0+4=9\), so \(\operatorname{RMSE}=\sqrt{9/4}=1.50\).
Step 2: Score ARIMA\((1,1,0)\). Its absolute errors sum to \(0.5+1+0.5+1=3\), giving \(\operatorname{MAE}=3/4=0.75\). Its squared errors sum to \(0.25+1+0.25+1=2.5\), giving \(\operatorname{RMSE}=\sqrt{2.5/4}=\sqrt{0.625}=0.7906\), rounded to four decimals.
Step 3: Score ARIMA\((0,1,1)\). Its absolute errors sum to \(2+1+1+3=7\), so \(\operatorname{MAE}=7/4=1.75\). Its squared errors sum to \(4+1+1+9=15\), so \(\operatorname{RMSE}=\sqrt{15/4}=\sqrt{3.75}=1.9365\), rounded to four decimals.
Conclusion. The ARIMA\((1,1,0)\) candidate performs best on both metrics at these four one-step origins, supporting the evidence from AICc and residual checks. The common mistake is to randomly split these monthly observations into training and test sets. That can train on later months and test on earlier ones, no longer representing forecasting; use time-ordered origins instead. Four origins provide limited evidence, so the result supports rather than guarantees the model choice.
Calculate and Qualify the Forecast
After selecting a specification, refit it using all observations available at the actual forecast date. Suppose the refitted model is ARIMA\((1,1,0)\) with drift and is written as \(\Delta y_t=c+\phi\Delta y_{t-1}+\varepsilon_t\), where \(\varepsilon_t\) has conditional mean zero and is uncorrelated over time with constant variance. For this fitted model, let \(c=0.8\), \(\phi=0.4\), and \(\Delta y_{20}=74-71=3\). Since \(|\phi|<1\), the differenced process has a stable autoregressive component. These fitted parameter values are hypothetical.
The recurrence follows by taking the conditional expectation of the model equation given the observations through month 20. At horizon one, \(E[\varepsilon_{21}\mid y_1,\ldots,y_{20}]=0\), so the forecast difference is \(c+\phi\Delta y_{20}\). At later horizons the same equation applies with the prior forecast difference in place of the unobserved difference:
Worked Example: Produce Three Forecasts After Re-estimation
Method. Use the refitted ARIMA\((1,1,0)\) recurrence above. Start from the observed \(y_{20}=74\) and \(\Delta y_{20}=3\). No interest rate, mortality assumption, or loss distribution is involved; this is a time-series forecast.
Month 21. The forecast change is \(0.8+0.4(3)=0.8+1.2=2.0\). Therefore \(\hat y_{21|20}=74+2.0=76.0\).
Month 22. The forecast change is \(0.8+0.4(2.0)=1.6\). Therefore \(\hat y_{22|20}=76.0+1.6=77.6\).
Month 23. The forecast change is \(0.8+0.4(1.6)=0.8+0.64=1.44\). Therefore \(\hat y_{23|20}=77.6+1.44=79.04\).
Check. The forecast differences decline toward the long-run mean change \(c/(1-\phi)=0.8/0.6=1.333\ldots\), consistent with the stable coefficient \(\phi=0.4\). Adding the three changes gives \(2.0+1.6+1.44=5.04\), and \(74+5.04=79.04\), matching the final level forecast. This is a conditional-mean forecast, not a guarantee; prediction intervals should accompany it when the decision requires uncertainty quantification.
Common Traps in an Integrated Forecast Case
- Ignoring time order. Randomly shuffling observations can leak future information into model fitting and invalidate a forecast evaluation.
- Differencing automatically. A trend in levels motivates investigation, not unlimited differencing. Choose the transformation that makes the series reasonably stable without introducing avoidable structure.
- Confusing residual fit with forecast skill. Residual checks assess whether model structure remains in fitted errors; rolling-origin scores assess predictive accuracy on later observations. Neither replaces the other.
- Comparing unequal evaluations. Models should use identical forecast origins, target observations, and horizons. Otherwise, their error scores are not a fair comparison.
- Refitting inconsistently. For an operational forecast, refit the selected specification using all observations available at the forecast date, not only the earlier training segment.
A final choice is strongest when the chronology and transformation are defensible, residuals show no important unmodeled dependence, and the model performs well under a time-ordered evaluation matched to the decision. If these signals disagree, revisit the specification or collect more evidence rather than selecting a model on one score alone.
Check Your Understanding
Work through each original problem, keeping the time order and the forecast horizon explicit.
- A hypothetical quarterly index is 31, 33, 32, 35, 36. Calculate all first differences and their mean. State one reason the mean change alone cannot establish stationarity. (Answer: Differences are 2, -1, 3, 1; mean change is 1.25; a constant average does not show that variance and dependence are stable.)
- Two candidates have AICc values 48.2 and 47.6. The lower-AICc candidate has Ljung–Box \(p=0.01\), while the other has \(p=0.24\). What is the appropriate conclusion? (Answer: Do not automatically select the 47.6 model; investigate its residual autocorrelation and compare predictive performance, because lower AICc does not establish adequate residual behavior.)
- At four matched rolling origins, a model has errors \(1,-2,0,1\). Calculate MAE and RMSE. (Answer: MAE \(=(1+2+0+1)/4=1.00\); RMSE \(=\sqrt{(1+4+0+1)/4}=\sqrt{1.5}=1.2247\).)
- A fitted differenced model is \(\Delta y_t=0.5+0.25\Delta y_{t-1}+\varepsilon_t\), with last observed level 40 and last observed difference 4. Calculate two forecasts. Assume future innovations have conditional mean zero. (Answer: Forecast differences are \(0.5+0.25(4)=1.5\) and \(0.5+0.25(1.5)=0.875\); level forecasts are 41.5 and 42.375.)
- A team uses months 1–18 for fitting, randomly assigns month 19 to training and month 12 to testing, then reports the test error. Identify the issue and the correction. (Answer: The split permits future-to-past information leakage and does not mimic forecasting; evaluate with a chronological origin, fitting only through the month before the test target.)