Why a Close Ranking Needs Careful Interpretation
Information criteria provide a disciplined way to compare candidate models, but they do not turn a close ranking into a decisive result. When the smallest AIC or BIC is only slightly below the next score, it is usually more informative to describe the candidates as having similar support than to announce a clear winner. This matters in actuarial work because two severity models with similar scores can imply different tail losses, premiums, or capital estimates.
AIC and BIC are each calculated from a fitted model’s likelihood and a penalty for model complexity. For a candidate model with maximized likelihood \(L\), \(k\) estimated parameters, and \(n\) observations, the usual forms are:
For either criterion, a lower value is preferred within the set of models being compared. That is a ranking rule, not a test that establishes the selected model as true. In a close comparison, the size of the difference matters alongside the ranking.
What “Nearly Tied” Means
A small difference means the criterion scores are close; it does not mean the fitted models are identical. As a practical interpretation, differences below about 2 are often treated as weak separation: the higher-scoring model remains a credible competitor. This is a rule of thumb, not a universal threshold, and it does not replace actuarial judgment about the modeling purpose, the data, or the consequences of different estimates.
Akaike weights provide one way to express relative support for a set of candidates. They use the AIC differences, not the raw AIC values. A weight is larger for a candidate with stronger support under this calculation, but it is not the probability that the model is true. It depends on which candidates were included and on the AIC comparison assumptions.
The factor of one-half is important. The weights compare candidates on a relative scale; the exponential terms are rescaled so that the weights sum to one. A high weight does not certify adequacy, and a low weight does not prove that a candidate is impossible. The comparison is only among the listed models.
Why the Weight Formula Uses Criterion Differences
The formula follows directly from comparing each model’s AIC to the minimum AIC. If model \(j\) has AIC \(A_j\), then \(\Delta_j=A_j-A_{\min}\). The unnormalized relative support associated with that difference is \(\exp(-\Delta_j/2)\). Dividing by the sum across the candidates makes the resulting quantities add to one:
Since subtracting the same minimum from every AIC cancels when the terms are normalized, the same weights would result from using \(\exp(-A_j/2)\). In practice, differences are preferable: the best model contributes exactly 1, and calculations are more manageable. The interpretation remains relative support in the candidate set, not a posterior probability that a model is correct.
Worked Example: Three Severity Models With a Close AIC Ranking
A hypothetical insurer compares three models for \(n=100\) independent, complete, positive claim severities measured in thousands of dollars. No interest or mortality assumptions are involved. The candidates are an exponential model with fitted scale \(\hat{\theta}=4.8\), a gamma model with fitted shape \(\hat{\alpha}=1.1\) and scale \(\hat{\theta}=4.2\), and a lognormal model with fitted parameters \(\hat{\mu}=1.30\) and \(\hat{\sigma}=0.95\), where \(\mu\) and \(\sigma\) are the mean and standard deviation of log severity. The models have respectively 1, 2, and 2 estimated parameters. Their reported maximized log-likelihood quantities are \(-2\log L=408.0,\ 406.6,\ 407.4\). Calculate the AIC values, differences, and weights, then interpret the result.
Step 1: Calculate AIC. For the exponential model, \(\mathrm{AIC}=408.0+2(1)=410.0\). For the gamma model, \(\mathrm{AIC}=406.6+2(2)=410.6\). For the lognormal model, \(\mathrm{AIC}=407.4+2(2)=411.4\).
Step 2: Calculate differences. The minimum AIC is 410.0, so the differences are 0.0 for the exponential model, 0.6 for the gamma model, and 1.4 for the lognormal model.
Step 3: Calculate the exponential terms and their sum. The terms are \(\exp(0)=1.0000\), \(\exp(-0.6/2)=\exp(-0.3)=0.7408\), and \(\exp(-1.4/2)=\exp(-0.7)=0.4966\). Their sum is \(1.0000+0.7408+0.4966=2.2374\).
Step 4: Normalize to obtain weights. The exponential-model weight is \(1.0000/2.2374=0.4470\). The gamma-model weight is \(0.7408/2.2374=0.3312\). The lognormal-model weight is \(0.4966/2.2374=0.2219\). The rounded weights sum to 1.0001 because of rounding; before rounding, they sum to 1.
Conclusion and trap avoided. The exponential model ranks first, but the gamma and lognormal models are not eliminated: their differences are both below 2, and the top candidate has less than half the total Akaike weight. Calling the exponential model a uniquely established winner would overstate the result. The method avoids confusing the lowest score with decisive separation.
When AIC and BIC Are Both Close but Disagree
AIC and BIC can rank candidates differently because their complexity penalties differ. AIC uses \(2k\), while BIC uses \(k\log n\). For a given sample size, the BIC penalty per parameter may be greater or smaller than the AIC penalty; the comparison depends on \(n\). When rankings reverse but the criterion differences are small, the appropriate conclusion is not that one criterion has settled the question. Instead, report the disagreement and the limited separation each criterion indicates.
Worked Example: AIC and BIC Give Opposite Near-Tie Rankings
For a hypothetical sample of 80 independent, uncensored positive losses measured in thousands of dollars, compare two fitted models. Model 1 is lognormal with estimates \(\hat{\mu}=8.10\) and \(\hat{\sigma}=0.85\), so \(k_1=2\). Model 2 is a three-parameter shifted Weibull with fitted shape \(1.35\), scale \(9.0\), and threshold \(0.4\), so \(k_2=3\). Suppose their maximized likelihood quantities are \(-2\log L_1=300.0\) and \(-2\log L_2=296.8\). Calculate both criteria and determine what can reasonably be concluded.
Step 1: Calculate AIC. Model 1 has \(\mathrm{AIC}_1=300.0+2(2)=304.0\). Model 2 has \(\mathrm{AIC}_2=296.8+2(3)=302.8\). AIC ranks Model 2 first, with a difference of \(304.0-302.8=1.2\).
Step 2: Calculate the AIC weights. The differences are 1.2 for Model 1 and 0 for Model 2. The unnormalized terms are \(\exp(-1.2/2)=\exp(-0.6)=0.5488\) and 1.0000, with sum 1.5488. The weights are \(0.5488/1.5488=0.3543\) for Model 1 and \(1/1.5488=0.6457\) for Model 2.
Step 3: Calculate BIC. Since \(\log(80)=4.3820\), Model 1 has \(\mathrm{BIC}_1=300.0+2(4.3820)=308.7640\). Model 2 has \(\mathrm{BIC}_2=296.8+3(4.3820)=309.9460\). BIC ranks Model 1 first, by \(309.9460-308.7640=1.1820\).
Conclusion and trap avoided. The AIC preference for Model 2 and the BIC preference for Model 1 are both modest. The results do not support a confident claim that either model is uniquely best. The weights above are AIC weights; they should not be presented as BIC weights. The method avoids treating a small rank reversal as a decisive contradiction.
Close Scores Do Not Guarantee Similar Actuarial Results
Information criteria summarize a balance between fit and complexity; they do not directly measure whether two models produce similar estimates for every quantity of interest. A near tie can coexist with meaningful differences in predicted tail losses or other decision-relevant quantities. Conversely, two candidates may have different mathematical forms but produce similar estimates for the particular quantity being priced or reserved.
The consequence is practical: when scores are close, retain the competing candidates in the interpretation rather than silently treating the minimum as certainty. State which model has the smallest criterion, how close the alternatives are, and whether the fitted candidates imply materially different results for the actuarial quantity being assessed. That is a description of model uncertainty, not a claim that every candidate is equally suitable.
Worked Example: A Three-Model Comparison With No Dominant Candidate
An actuary fits three distributions to a hypothetical set of 120 complete, independent positive property-loss severities, in thousands of dollars. There are no interest or mortality assumptions. The candidates are exponential with fitted scale \(5.0\) and \(k=1\); gamma with fitted shape \(1.2\), scale \(4.5\), and \(k=2\); and lognormal with fitted \(\hat{\mu}=1.35\), \(\hat{\sigma}=0.90\), and \(k=2\). Their reported values of \(-2\log L\) are 408.0, 406.6, and 407.4, respectively. Calculate their AIC values and weights, then assess whether the top-ranked model is dominant.
Step 1: Apply the AIC penalty. The exponential AIC is \(408.0+2(1)=410.0\). The gamma AIC is \(406.6+2(2)=410.6\). The lognormal AIC is \(407.4+2(2)=411.4\). The AIC differences are therefore 0.0, 0.6, and 1.4.
Step 2: Obtain the relative-support terms. The terms are \(1.0000\), \(\exp(-0.3)=0.7408\), and \(\exp(-0.7)=0.4966\), giving a total of \(2.2374\).
Step 3: Normalize and interpret. The weights are \(0.4470\) for exponential, \(0.3312\) for gamma, and \(0.2219\) for lognormal, to four decimal places. The exponential model is the AIC leader, but its weight is below 0.50, and both alternatives have small differences from the minimum. There is no dominant candidate by this comparison alone.
Conclusion and trap avoided. A correct report gives the ranking and the close differences, rather than presenting “exponential is best” as a settled conclusion. The calculation avoids ignoring the parameter penalty: the gamma model has a better likelihood quantity than the exponential model, but its extra parameter raises its AIC.
Common Exam Traps
- Using the wrong exponent. For a difference of 0.8, the correct unnormalized AIC support term is \(\exp(-0.8/2)=\exp(-0.4)=0.6703\), not \(\exp(-0.8)=0.4493\). The latter incorrectly omits the factor of one-half.
- Calling a small difference decisive. A minimum score identifies the ranking leader, not necessarily a uniquely supported model. State the difference and qualify the strength of separation.
- Calling a weight the probability that a model is true. Akaike weights are normalized relative support measures for the candidate set. They are not probabilities of truth and do not establish that any candidate adequately represents the data.
- Mixing criteria or candidate sets. Calculate differences and weights within a clearly specified comparison using comparable data and consistent likelihood conventions. Do not combine AIC and BIC values in one weight calculation.
Check Your Understanding
For each problem, state what the close or separated scores support; do not interpret a weight as the probability that a model is true.
- Two models have AIC values 75.4 and 76.4. Calculate their AIC differences and weights to four decimals. (Answer: Differences 0.0 and 1.0; weights 0.6225 and 0.3775. The separation is weak.)
- Three models have AIC values 75.4, 76.4, and 79.4. Calculate their differences and weights to four decimals. (Answer: Differences 0.0, 1.0, and 4.0; weights 0.5741, 0.3482, and 0.0777.)
- For \(n=50\), Model A has \(-2\log L=210.0\) and \(k=2\); Model B has \(-2\log L=206.5\) and \(k=3\). Calculate the BIC values and their difference. (Answer: \(\log(50)=3.9120\); \(\mathrm{BIC}_A=217.8240\), \(\mathrm{BIC}_B=218.2360\); Model A is lower by 0.4120, a near tie.)
- Two models have AIC differences 0.0 and 3.8. What is the second model’s unnormalized relative-support term, and does that alone rule it out? (Answer: \(\exp(-3.8/2)=\exp(-1.9)=0.1496\); no, it does not rule the model out.)
- A candidate has the lowest AIC, but its Akaike weight is 0.42 and two alternatives have differences below 2. Give a defensible conclusion. (Answer: It is the ranking leader, but support is shared among close alternatives; the evidence does not establish a uniquely best model.)