Why Different Criteria Can Choose Different Models
Two time-series models can both describe the observed data reasonably well and still receive different rankings from AIC, AICc, and BIC. The reason is that each criterion balances fit against model complexity differently. A more complex model can improve its likelihood, but it must improve it enough to offset the criterion’s penalty for its additional estimated parameters.
No criterion is a universal winner. AIC is motivated by predictive performance, AICc adds a small-sample correction, and BIC imposes a penalty that grows with the sample size. Their rankings are useful evidence for model comparison, not proof that one model is correct. In time-series work, the candidate models must also have comparable likelihoods and must adequately describe the data.
Here, \(k\) counts all parameters estimated in the likelihood, including any estimated mean or intercept and, when included in the likelihood, an innovation variance. The exact count depends on the model and likelihood convention. The AICc formula requires \(n>k+1\); if that condition fails, the displayed correction is undefined and AICc should not be computed using this formula.
For time-series comparisons, “comparable likelihoods” matters. A criterion comparison should use the same observations and a consistent likelihood treatment across candidates. If likelihoods are computed over different sets of observations or under incompatible conventions, differences in the criterion may not represent a meaningful model comparison. The criteria also do not replace adequacy checks: a low value does not establish that residuals behave like white noise.
How the Penalties Differ
All three criteria include the fit term \(-2\ell\). A larger maximized likelihood means a smaller value of this term, so better fit lowers the criterion. The remaining terms penalize the number of estimated parameters.
| Criterion | Complexity penalty | Selection emphasis |
|---|---|---|
| AIC | \(2k\) | Predictive performance; its penalty does not grow with \(n\) |
| AICc | \(2k+2k(k+1)/(n-k-1)\) | Predictive performance with an additional correction, especially relevant for small samples |
| BIC | \(k\log(n)\) | A stronger complexity penalty as \(n\) grows; under suitable conditions, favors the true model when it is among the candidates |
The AIC penalty \(2k\) is associated with correcting for the optimism that arises because the same data are used to fit a model and assess its fit. AIC is therefore commonly used when predictive performance is the main goal. The extra AICc term accounts for additional small-sample bias in the usual setting; for fixed \(k\), that correction shrinks as \(n\) increases. BIC’s penalty increases with \(\log(n)\). Under regularity conditions and when the true model is among a suitable finite set of candidates, BIC is consistent in selecting that model as sample size grows. Those conditions are substantive; BIC is not guaranteed to identify truth in every modeling exercise.
A Useful Comparison by Algebra
The trade-off is especially clear when comparing a larger model with a smaller one. Let the larger model add \(q\) parameters, and let its maximized log-likelihood exceed the smaller model’s by \(\Delta\ell\). The change in AIC is the larger model’s AIC minus the smaller model’s AIC.
Thus, AIC favors the larger model exactly when \(\Delta\ell>q\), or equivalently when the reduction in \(-2\ell\) exceeds \(2q\). For BIC, the same subtraction gives \(\Delta\mathrm{BIC}=-2\Delta\ell+q\log(n)\), so BIC favors the larger model only when the reduction in \(-2\ell\) exceeds \(q\log(n)\). This derivation explains why identical likelihood improvements can lead to different rankings: each criterion sets a different threshold for the added complexity.
For AICc, the threshold includes the change in the small-sample correction as well as the \(2q\) AIC penalty. Because that correction depends on \(k\) and \(n\), it should be calculated for each candidate rather than treated as a fixed surcharge.
Worked Example: AIC and BIC Disagree on a Longer Series
A hypothetical sensor produces \(n=80\) consecutive measurements. Two comparable candidate time-series models have the following maximized likelihood summaries: Model S has \(k=2\) parameters and \(-2\ell=210\); Model L has \(k=4\) parameters and \(-2\ell=204\). Calculate all three criteria and identify their rankings. Use natural logarithms and round criterion values to three decimals.
Method. Substitute each model’s \(n\), \(k\), and \(-2\ell\) into each formula. The likelihood improvement for Model L is \(210-204=6\), while it adds \(4-2=2\) parameters.
AIC. For Model S, \(\mathrm{AIC}=210+2(2)=214\). For Model L, \(\mathrm{AIC}=204+2(4)=212\). AIC selects Model L because \(212<214\).
AICc. For Model S, the correction is \(2(2)(3)/(80-2-1)=12/77=0.155844\), so \(\mathrm{AICc}=214+0.155844=214.156\). For Model L, it is \(2(4)(5)/(80-4-1)=40/75=0.533333\), so \(\mathrm{AICc}=212+0.533333=212.533\). AICc selects Model L because \(212.533<214.156\).
BIC. Since \(\log(80)=4.382027\), Model S has \(\mathrm{BIC}=210+2(4.382027)=218.764\). Model L has \(\mathrm{BIC}=204+4(4.382027)=221.528\). BIC selects Model S because \(218.764<221.528\).
Check and trap. Model L improves \(-2\ell\) by 6. AIC requires an improvement greater than \(2(2)=4\), so it accepts the added parameters. BIC requires more than \(2\log(80)=8.764\), so it does not. A common mistake is to say that BIC must choose the simpler model: it does so here because of the calculated threshold, not merely because it is BIC.
AICc Can Change a Short-Series Ranking
AICc’s additional correction can be substantial when the sample is small relative to the parameter count. It is not a separate likelihood calculation: it adds a model-specific correction to AIC. This means that even if AIC prefers a larger candidate, AICc may prefer a simpler one.
Worked Example: Applying the Small-Sample Correction
A hypothetical quarterly claims index has \(n=18\) usable observations. Candidate P has \(k=2\) parameters and \(-2\ell=50\); candidate Q has \(k=5\) parameters and \(-2\ell=43\). Compare AIC, AICc, and BIC. Use natural logarithms and round criterion values to three decimals.
Method. First calculate AIC for each model, then add each model’s own AICc correction. Both candidates satisfy the AICc condition: \(18>2+1\) and \(18>5+1\).
AIC. Candidate P has \(50+2(2)=54\). Candidate Q has \(43+2(5)=53\). AIC selects Q.
AICc. For P, the correction is \(2(2)(3)/(18-2-1)=12/15=0.800\), giving \(54+0.800=54.800\). For Q, it is \(2(5)(6)/(18-5-1)=60/12=5.000\), giving \(53+5.000=58.000\). AICc selects P.
BIC. Since \(\log(18)=2.890372\), P has \(50+2(2.890372)=55.781\), and Q has \(43+5(2.890372)=57.452\). BIC selects P.
Check and trap. AIC’s fit improvement for Q is \(50-43=7\), larger than its extra-parameter AIC penalty of \(2(5-2)=6\). But AICc adds a much larger correction to Q than to P. A common mistake is to apply the correction only to the larger model or to assume AICc must agree with AIC. Here the corrections are \(0.800\) and \(5.000\), so AICc reverses the AIC ranking.
Use the Ranking in Context
When criteria disagree, first verify that the candidates were compared fairly and that the arithmetic and parameter counts are correct. Then consider the modeling objective. For a forecasting purpose, AIC or AICc may be informative because of their predictive motivation. For a goal that emphasizes identifying a parsimonious data-generating model under appropriate assumptions, BIC may be informative. Neither motivation makes a criterion automatically decisive for a particular application.
Next, consider adequacy. If the apparent winner leaves important residual autocorrelation or another systematic pattern, its low criterion value does not make it an adequate model. The sensible response is to revisit the candidate set or the model assumptions, not to select a model solely because one score is smallest. Conversely, a small numerical difference should not be described as decisive evidence when the criteria are ranking candidates rather than testing a hypothesis.
Worked Example: Resolve a Three-Criterion Disagreement
A hypothetical monthly river-flow series has \(n=60\) observations. Three comparable candidate models have these summaries: Model A has \(k=3\) and \(-2\ell=130\); Model B has \(k=4\) and \(-2\ell=126\); Model C has \(k=6\) and \(-2\ell=121\). Residual checks indicate that A and B have no identified autocorrelation pattern, while C has clear residual autocorrelation. Calculate the criteria, then state which model is eligible among the candidates that pass the stated adequacy check. Use natural logarithms and round to three decimals.
Method. Calculate each criterion before applying the adequacy information. Since \(\log(60)=4.094345\), the BIC penalty per parameter is \(4.094345\).
AIC. A has \(130+2(3)=136\); B has \(126+2(4)=134\); C has \(121+2(6)=133\). AIC ranks C first, then B, then A.
AICc. A’s correction is \(2(3)(4)/(60-3-1)=24/56=0.428571\), giving \(136.429\). B’s is \(2(4)(5)/(60-4-1)=40/55=0.727273\), giving \(134.727\). C’s is \(2(6)(7)/(60-6-1)=84/53=1.584906\), giving \(134.585\). AICc ranks C first, then B, then A.
BIC. A has \(130+3(4.094345)=142.283\); B has \(126+4(4.094345)=142.377\); C has \(121+6(4.094345)=145.566\). BIC ranks A first, then B, then C.
Decision and check. C has the smallest AIC and AICc, but it fails the stated residual-adequacy check. Among A and B, AIC and AICc select B, while BIC selects A. If predictive performance is the stated priority, B is a defensible choice among the adequate candidates; if greater parsimony is prioritized under a BIC-style goal, A is defensible. The criteria do not settle that goal for the analyst. The trap is to declare C the winner from its lowest score while ignoring its residual autocorrelation.
Common Mistakes and Practical Rules
- Choosing the largest criterion. For all three criteria, lower is preferred within a comparable candidate set.
- Comparing incompatible likelihoods. Different observations or inconsistent likelihood treatments can undermine the comparison.
- Forgetting the parameter count. Omitting an estimated parameter changes the penalty and can change the ranking.
- Computing AICc outside its condition. The stated formula requires \(n>k+1\); do not interpret a zero or negative denominator as a usable correction.
- Treating a criterion as an adequacy test. Information criteria rank candidates; they do not establish that residuals are white noise.
- Calling one criterion universally best. State the selection goal, report the disagreement, and explain how adequacy and complexity affect the decision.
Check Your Understanding
For each problem, use natural logarithms where needed and compare only the criteria requested.
- A hypothetical vibration series has \(n=40\). Model R has \(k=2\) and \(-2\ell=72\); Model S has \(k=3\) and \(-2\ell=70\). Calculate AIC and BIC for both and identify the preferred model under each. (Answer: AIC ties at 76.000; BIC selects R, with BIC values 79.378 for R and 81.067 for S.)
- For a hypothetical series with \(n=25\), Model U has \(k=2\), \(-2\ell=90\), and Model V has \(k=4\), \(-2\ell=83\). Which model has the smaller AICc? (Answer: Model V; AICc values are 94.545 for U and 93.000 for V.)
- A candidate model has \(n=12\) and \(k=11\). Can AICc be calculated using the formula in this tutorial? (Answer: No; \(n-k-1=0\), so the correction is undefined and the required condition \(n>k+1\) fails.)
- Two comparable models use the same observations and each has \(k=3\). Their \(-2\ell\) values are 55 and 53. Which model has the smaller AIC, AICc, and BIC? (Answer: The model with \(-2\ell=53\) under all three criteria, because the penalties are equal for equal \(k\) and \(n\).)
- With \(n=100\), a larger model adds \(q=3\) parameters and reduces \(-2\ell\) by 12. Does AIC or BIC favor the larger model? (Answer: AIC favors it because \(12>2q=6\); BIC does not because \(12<q\log(100)\approx13.816\).)