Why Model-Selection Evidence Can Disagree
A time-series model can look attractive by one measure and problematic by another. An information criterion may favor a model whose residuals still show autocorrelation. A model with satisfactory residuals may forecast worse at the horizon that matters. And two candidates may have different average forecast scores even when the comparison is too uncertain to establish a dependable winner.
The evidence answers different questions. Residual diagnostics ask whether important patterns remain unexplained by a fitted model. Information criteria compare fit and complexity under a likelihood-based penalty. Rolling-origin forecast scores assess predictions on observations not used to fit each forecast, for a chosen horizon and loss measure. Reconciling them means deciding what each result says, checking that the comparison is valid, and selecting in light of the intended use.
For a fitted model, residuals are the observed values minus their fitted values. A basic adequacy check is whether residuals have approximately zero mean and no consequential remaining autocorrelation; distributional checks may also matter when the model or the use of prediction intervals relies on a distributional assumption. In a Ljung–Box check, the statistic at lag \(m\) is
where \(n\) is the residual-series length and \(\hat{\rho}_j\) is its sample autocorrelation at lag \(j\). Under the null hypothesis of no autocorrelation through the tested lags, \(Q\) is compared approximately with a chi-square distribution, with degrees of freedom adjusted for fitted model parameters. A small p-value is evidence against that null, not a measure of how much better another model is. A large p-value is not proof that the residuals are independent or that every model assumption holds.
An information criterion is a relative fit-complexity score. For Gaussian likelihood with \(k\) estimated parameters and sample size \(n\), one common criterion is
where \(L\) is the maximized likelihood and \(n>k+1\). Lower AICc is preferred among the candidates being compared, but only when their likelihoods and data basis make the comparison meaningful. The criterion does not directly measure forecast accuracy for a particular horizon or business loss. In particular, comparisons can be misleading if candidates use different response data or incompatible likelihood calculations.
A Practical Order for Reconciling Evidence
Check that the candidates were assessed on the same response and comparable estimation samples, that the information-criterion calculation counts parameters consistently, and that forecast origins and horizons are aligned.
Look for remaining patterns that matter, such as persistent residual autocorrelation. Treat a concerning diagnostic as a reason to investigate or revise a candidate, not as a mechanical rule that a particular p-value selects a model.
Name the horizon and loss measure. For instance, squared-error loss places more weight on large errors than absolute-error loss, and a one-step score need not rank models like a longer-horizon score.
Compare criterion differences, forecast-score differences, and their stability across origins. If rankings conflict or differences are small relative to their uncertainty, report the trade-off rather than claiming an unambiguous winner.
A short derivation explains why forecast objectives matter. Let \(Y\) be the future value, \(f\) a forecast fixed using the training data, and suppose \(Y\) has finite variance. Add and subtract \(E[Y]\) inside the squared error:
The cross-term vanishes because \(E[Y-E[Y]]=0\). Thus expected squared error penalizes both forecast bias and the irreducible variation in \(Y\). In practice, a rolling-origin score estimates predictive performance from the available forecast errors; it does not reveal the exact future expected loss. Root mean squared error (RMSE) is the square root of mean squared error, so minimizing one gives the same ranking as minimizing the other on the same errors. Mean absolute error (MAE) can rank models differently because it does not square errors.
Worked Comparisons
Worked Example: A Better Criterion but a Diagnostic Failure
A hypothetical analyst forecasts monthly log claim frequency using three Gaussian-innovation time-series candidates. There is no interest-rate or mortality assumption in this statistical comparison. Each candidate is fitted to the same \(n=24\) observations; \(k\) counts all estimated parameters, including the innovation variance. The reported maximized likelihood summaries are \(-2\log L\) values. At residual lag \(m=6\), the Ljung–Box p-values are also reported. Rolling-origin one-step forecast errors, on the same log scale, are listed below.
| Candidate | \(k\) | \(-2\log L\) | Residual p-value | Four forecast errors |
|---|---|---|---|---|
| A | 2 | 100 | 0.0100 | 0, 1, -1, 4 |
| B | 3 | 99 | 0.1800 | 1, -1, 1, -1 |
| C | 4 | 98 | 0.4200 | 0, 2, -2, 2 |
Method. Compute AICc on the common sample, inspect the stated residual diagnostic, and calculate one-step RMSE from the given errors. The assumed innovations are Gaussian; the forecast errors are observed validation errors, not additional assumed model parameters.
For A, \(2k=4\), and the correction is \(2(2)(3)/(24-2-1)=12/21=0.5714\). Therefore, \(\mathrm{AICc}_A=100+4+0.5714=104.5714\).
For B, \(2k=6\), and the correction is \(2(3)(4)/(24-3-1)=24/20=1.2000\). Thus, \(\mathrm{AICc}_B=99+6+1.2000=106.2000\).
For C, \(2k=8\), and the correction is \(2(4)(5)/(24-4-1)=40/19=2.1053\). Thus, \(\mathrm{AICc}_C=98+8+2.1053=108.1053\). A has the lowest AICc.
Now calculate the forecast scores. A has squared errors \(0,1,1,16\), whose sum is \(18\), so its RMSE is \(\sqrt{18/4}=\sqrt{4.5}=2.1213\). B has squared errors \(1,1,1,1\), so its RMSE is \(\sqrt{4/4}=1.0000\). C has squared errors \(0,4,4,4\), so its RMSE is \(\sqrt{12/4}=\sqrt{3}=1.7321\).
A has the best criterion, but its residual p-value of 0.0100 raises a concern about remaining autocorrelation at the tested lags. B has a less favorable AICc, no such flag at the stated test level, and the lowest observed one-step RMSE. On this evidence, B is the more defensible provisional choice for one-step forecasting; the analyst should also investigate whether A's residual pattern points to an omitted time-series feature.
Common mistake avoided: Selecting A solely because its AICc is smallest ignores both the diagnostic failure and the forecast objective. Conversely, a p-value of 0.1800 does not prove B is adequate in every respect. The conclusion is limited to the checks and scores supplied.
Check: the AICc values increase from 104.5714 to 106.2000 to 108.1053, while the calculated forecast RMSE is smallest for B. These rankings are consistent with the stated conflict.
Worked Example: The Best Model Depends on the Forecast Horizon
A hypothetical service center forecasts monthly repair demand. Two Gaussian-innovation candidates, D and E, are fitted to the same \(n=30\) observations; \(k\) includes all estimated parameters, including the innovation variance. Both pass the stated residual checks, and there is no interest-rate or mortality assumption. The analyst cares about forecasts three months ahead, although one-step errors are also available.
Candidate D has \(k=2\) and \(-2\log L=120\). Its AICc is \(120+4+12/(30-2-1)=124+12/27=124.4444\). Candidate E has \(k=3\) and \(-2\log L=119\). Its AICc is \(119+6+24/(30-3-1)=125+24/26=125.9231\). The criterion favors D.
At one step, D's four errors are \(0,0,2,2\). The squared errors sum to \(0+0+4+4=8\), giving \(\mathrm{RMSE}_{D,1}=\sqrt{8/4}=\sqrt{2}=1.4142\). E's one-step errors are \(1,1,1,1\); its squared errors sum to \(4\), giving \(\mathrm{RMSE}_{E,1}=\sqrt{4/4}=1.0000\). E is better at one step.
At three steps, D's errors are \(2,2,2,2\). Their squared errors sum to \(16\), so \(\mathrm{RMSE}_{D,3}=\sqrt{16/4}=2.0000\). E's errors are \(0,4,0,4\); their squared errors sum to \(0+16+0+16=32\), so \(\mathrm{RMSE}_{E,3}=\sqrt{32/4}=\sqrt{8}=2.8284\). D is better at three steps.
Because the stated objective is three-month-ahead forecasting, D is supported by both AICc and the horizon-specific RMSE. The one-step result favoring E should still be reported, but it does not answer the primary forecasting question.
Common mistake avoided: Using the one-step ranking for a three-step decision would select E against the stated objective. The error scale, forecast origins, and horizon must match the intended comparison.
Check: D has lower AICc and three-step RMSE, whereas E has lower one-step RMSE. The recommendation of D follows from the specified three-step objective, not from an assumption that D is best at every horizon.
Worked Example: A Near Tie Is Not a Reliable Forecast Winner
A hypothetical analyst forecasts a continuous monthly workload measure using Gaussian-innovation candidates P and Q. Both pass the residual checks. Each is fitted to the same \(n=20\) observations, \(k\) includes the innovation variance, and no interest-rate or mortality assumption applies. The analyst uses four matched rolling-origin one-step forecasts and MAE as the primary score.
P has \(k=2\) and \(-2\log L=80\). Its AICc is \(80+4+12/(20-2-1)=84+12/17=84.7059\). Q has \(k=3\) and \(-2\log L=77.2\). Its AICc is \(77.2+6+24/(20-3-1)=83.2+1.5=84.7000\). Q's AICc is lower by \(84.7059-84.7000=0.0059\), a negligible difference in this comparison.
P's forecast errors are \(2,1,3,2\). Their absolute values sum to \(2+1+3+2=8\), so \(\mathrm{MAE}_P=8/4=2\). Q's errors are \(1,2,2,3\), with absolute values summing to \(8\), so \(\mathrm{MAE}_Q=8/4=2\). Each model's squared errors sum to \(4+1+9+4=18\), giving RMSE \(\sqrt{18/4}=2.1213\) for both.
To inspect the matched absolute-error differences, subtract Q's absolute error from P's at each origin. The differences are \(1,-1,1,-1\), with mean \(0/4=0\). Their sample variance is \(\{1+1+1+1\}/(4-1)=4/3\), so their sample standard deviation is \(\sqrt{4/3}=1.1547\), and the standard error of the mean difference is \(1.1547/\sqrt{4}=0.5774\). The estimated difference is zero and small relative to its standard error; four origins do not establish a dependable forecast-score advantage.
The evidence does not justify declaring Q a meaningful winner because its AICc is lower by 0.0059. If a choice is required and the forecast performance is treated as tied, P is a reasonable practical choice because it has fewer estimated parameters. The choice should be described as a parsimonious resolution of essentially tied evidence, not proof that P forecasts better.
Common mistake avoided: Announcing Q as superior based on a tiny criterion difference overstates the evidence, while claiming P has lower forecast error contradicts the calculated equal scores.
Check: both MAEs equal 2, both RMSEs equal 2.1213, and the mean paired absolute-error difference is 0. The calculations support a tie in observed forecast performance.
Common Traps and a Defensible Conclusion
When evidence conflicts, the goal is not to make every measure agree. It is to avoid asking one measure to answer a different measure's question. Keep the following distinctions explicit:
- Diagnostic evidence is not a ranking. A residual test flags a pattern relative to its null hypothesis. It does not say which alternative model has the best forecasting performance.
- A criterion is not a forecast score. AICc balances likelihood fit and parameter count on the fitted sample. It does not specify a forecast horizon or a cost for errors.
- A forecast score is objective-dependent. Name the evaluation period, horizon, and loss measure. Do not treat RMSE, MAE, and a horizon-specific score as interchangeable.
- Small differences deserve caution. Scores can vary across forecast origins. A modest ranking difference, especially over few origins, may not be practically or statistically persuasive.
- Search history matters. If many candidates or scoring choices were examined, the apparent winner may reflect repeated search rather than stable predictive advantage. Keep the model search and validation procedure honest.
A careful conclusion states the evidence and its limits: for example, “Candidate B has no flagged autocorrelation in the tested residual lags and the lowest one-step RMSE, while candidate A has the lowest AICc; B is preferred for the stated one-step objective, subject to further residual review.” This is more informative than declaring a universally best model.
This approach to model selection is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus: distinguish what diagnostics, criteria, and forecast validation establish, and make a recommendation that follows from the stated objective rather than from a single winning number.
Check Your Understanding
For each situation, distinguish the evidence and make a conclusion that respects the stated objective.
- Two Gaussian-innovation candidates are fitted to \(n=16\) observations. Candidate R has \(k=2\), \(-2\log L=40\), residual Ljung–Box p-value 0.0200, and one-step RMSE 0.9000. Candidate S has \(k=3\), \(-2\log L=39\), residual p-value 0.2700, and one-step RMSE 1.2000. Calculate both AICc values and state a defensible provisional choice if remaining autocorrelation is material to the use. No interest-rate or mortality assumption applies. (Answer: \(\mathrm{AICc}_R=40+4+12/13=44.9231\); \(\mathrm{AICc}_S=39+6+24/12=47.0000\). R has lower AICc and RMSE, but its diagnostic flag warrants investigation; if the residual autocorrelation is material, S is the more defensible provisional choice, not a proven winner.)
- For a three-step objective, candidate U has errors \(1,1,1,1\) and candidate V has errors \(0,0,2,2\). Compute the RMSE for each and choose by this objective. (Answer: \(\mathrm{RMSE}_U=\sqrt{4/4}=1.0000\); \(\mathrm{RMSE}_V=\sqrt{8/4}=1.4142\); choose U by three-step RMSE.)
- Candidate W has forecast errors \(0,0,0,6\), while candidate Z has errors \(2,2,2,2\), all at the same horizon and origins. Compute MAE and RMSE for each, then explain why the choice depends on the loss measure. (Answer: W has MAE \(6/4=1.5000\) and RMSE \(\sqrt{36/4}=3.0000\); Z has MAE \(8/4=2.0000\) and RMSE \(\sqrt{16/4}=2.0000\). Choose W by MAE and Z by RMSE.)
- Two candidates pass the same residual checks and have identical MAE on eight matched forecast origins. Their AICc values are 62.40 and 62.43, with the first candidate having fewer parameters. What conclusion is justified? (Answer: Forecast MAE is tied on the observed origins and the 0.03 AICc difference is negligible; the simpler first candidate is a reasonable practical choice, but the evidence does not establish better forecast performance.)