Why Small Calculation Errors Change Model Selection
AIC and BIC are straightforward to calculate, but a small error in a sign, parameter count, or sample size can reverse a model ranking. A further trap is to treat the resulting values as probabilities or as test statistics with a universal cutoff. Reliable use begins by checking that the likelihoods being compared are genuinely comparable, then applying the formulas consistently.
This tutorial focuses on those calculation and interpretation checks. The formulas are useful only when each candidate model has been fitted to the same observed data under a compatible sampling setup. The guidance here is aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.
Use the maximized log-likelihood, not the likelihood itself. A log-likelihood is often negative, so \(-2\hat{\ell}\) is often positive. The penalty is added: it is \(+2k\) for AIC and \(+k\ln(n)\) for BIC. Neither criterion is automatically the same as the model's negative log-likelihood.
A Sign Check and a Direct Comparison
A useful check is to calculate \(-2\hat{\ell}\) on its own before adding the penalty. When comparing two models, it is also possible to work directly with the criterion difference. For model 2 minus model 1,
and
A negative difference means model 2 has the lower criterion. These equations also show why a better fit does not always win: its improvement in log-likelihood must offset the added complexity penalty.
Worked Example: Avoiding the Log-Likelihood Sign Error
A hypothetical insurer fits two severity models to the same \(n=40\) loss observations. Model A has \(\hat{\ell}_A=-120\) and \(k_A=3\). Model B has \(\hat{\ell}_B=-117\) and \(k_B=5\). Calculate AIC and BIC, then state which model each criterion prefers. The method is to compute the fit term and penalty separately; this avoids subtracting the penalty or treating the negative log-likelihood as the log-likelihood.
AIC for A: The fit term is \(-2(-120)=240\), and the penalty is \(2(3)=6\). Therefore, \(\operatorname{AIC}_A=240+6=246\).
AIC for B: The fit term is \(-2(-117)=234\), and the penalty is \(2(5)=10\). Therefore, \(\operatorname{AIC}_B=234+10=244\). AIC prefers B because \(244<246\).
BIC for A: The penalty is \(3\ln(40)=3(3.688879)=11.066637\). Thus, \(\operatorname{BIC}_A=240+11.066637=251.066637\).
BIC for B: The penalty is \(5\ln(40)=5(3.688879)=18.444395\). Thus, \(\operatorname{BIC}_B=234+18.444395=252.444395\). BIC prefers A because \(251.066637<252.444395\). Rounded to three decimal places, the BIC values are 251.067 and 252.444.
Check: Model B improves the log-likelihood by \(3\), so its \(-2\hat{\ell}\) term decreases by \(6\). Its additional two parameters add \(4\) to AIC, making B's AIC lower by \(2\). For BIC, those parameters add \(2\ln(40)=7.377758\), which exceeds the fit improvement of \(6\); hence BIC favors A. The criteria need not agree.
Count Every Estimated Parameter Consistently
The parameter count \(k\) includes every freely estimated parameter in the likelihood for that model. Depending on the model, this may include location or scale parameters, variance parameters, mixture weights subject to their constraints, and any other estimated distributional parameters. Do not count a parameter merely because it is named, and do not omit one because it is not the main quantity of interest. Use the same counting convention for every candidate.
A common error is to count only the parameters that differ between models, or to include a scale parameter for one model but omit it for another. The penalty is based on each model's full parameter count, not just the number of extra parameters relative to a reference model.
Worked Example: Correcting an Inconsistent Parameter Count
Two severity models are fitted by maximum likelihood to the same \(n=100\) losses. Model C estimates a location and scale parameter, so \(k_C=2\), and has \(\hat{\ell}_C=-100\). Model D estimates those two parameters plus two shape parameters, so \(k_D=4\), and has \(\hat{\ell}_D=-96.5\). Compare them by BIC. The method is to use each model's full parameter count; the common error is to count only two parameters for D because only two were added beyond C.
For C, \(-2\hat{\ell}_C=200\), and its penalty is \(2\ln(100)=2(4.605170)=9.210340\). Therefore, \(\operatorname{BIC}_C=209.210340\).
For D, \(-2\hat{\ell}_D=193\), and its penalty is \(4\ln(100)=4(4.605170)=18.420680\). Therefore, \(\operatorname{BIC}_D=211.420680\). Rounded to three decimal places, the values are 209.210 and 211.421, so BIC prefers C.
If an analyst incorrectly entered \(k_D=2\), the reported value would instead be \(193+2\ln(100)=202.210340\), falsely making D appear preferred. The check is to list the actual estimated parameters for each fitted likelihood before substituting into the formula.
Use the Sample Size That Matches the Likelihood
The \(n\) in BIC is not automatically the number of rows in a printed table, the number of fitted parameters, or the total exposure measured in some convenient unit. It must reflect the independent sampling units appropriate to the likelihood and the model's sampling design. If observations are grouped, clustered, or recorded as aggregate counts, identify what constitutes the data contributing independent information before using \(n\). Do not change \(n\) just because the same data are displayed in a different number of bins.
A wrong \(n\) changes the BIC penalty while leaving the fitted log-likelihood unchanged. Because that penalty can affect the ranking, record the definition of \(n\) alongside the calculation.
Worked Example: Using the Observation Count Rather Than the Number of Display Bins
A hypothetical portfolio has 30 independent individual losses. They are displayed in five histogram bins for a report, but the candidate severity models were fitted to the 30 individual losses. Model E has \(\hat{\ell}_E=-52.5\), \(k_E=2\); model F has \(\hat{\ell}_F=-50\), \(k_F=4\). Compare BIC using the observation count in the likelihood. The method is to use \(n=30\), not the five display bins; the trap is to confuse a reporting format with the sampling units.
For E, the fit term is \(-2(-52.5)=105\), and the penalty is \(2\ln(30)=2(3.401197)=6.802394\). Thus, \(\operatorname{BIC}_E=111.802394\).
For F, the fit term is \(-2(-50)=100\), and the penalty is \(4\ln(30)=13.604788\). Thus, \(\operatorname{BIC}_F=113.604788\). Rounded to three decimal places, the values are 111.802 and 113.605, so BIC prefers E.
If someone used the five histogram bins as \(n\), the computed values would be \(105+2\ln(5)=108.219\) for E and \(100+4\ln(5)=106.438\) for F, incorrectly reversing the ranking. Those values do not use the sample size corresponding to the individual-loss likelihood. The comparison is checked by confirming both that the likelihood uses the individual observations and that \(n=30\) counts those observations.
Make Sure the Likelihoods Are Comparable
AIC and BIC compare maximized likelihoods for candidate models fitted to the same observed response under compatible sampling assumptions. A likelihood for complete observations is not directly comparable to one for censored or truncated observations if the latter does not account for how those observations entered the data. Likewise, log-likelihoods calculated on different response scales require careful treatment.
For a transformation \(Z=\ln(X)\), the density transformation is \(f_X(x)=f_Z(\ln x)/x\) for \(x>0\). Therefore, the log-likelihood on the original \(X\)-scale includes the Jacobian adjustment \(-\sum_i\ln(x_i)\). Comparing a log-scale log-likelihood without that adjustment against a likelihood on the original scale can reverse the ranking. Likelihood constants may be omitted only when the omitted terms are common to all candidate models in the comparison.
Worked Example: Correcting a Log-Scale Likelihood
Two models are compared for losses \(x_1=1\) and \(x_2=2\), and each estimates \(k=2\) parameters. Model G has an original-scale maximized log-likelihood of \(-10.0\). Model H was evaluated for \(Z=\ln(X)\) and has a log-scale log-likelihood of \(-9.6\). Convert H's value to the original scale before comparing. The method is to apply the Jacobian; the mistake avoided is ranking the raw log-scale likelihood against an original-scale likelihood.
For H, the Jacobian adjustment is \(-\ln(1)-\ln(2)=-0.693147\). Its original-scale log-likelihood is therefore \(-9.6-0.693147=-10.293147\).
For G, \(\operatorname{AIC}_G=-2(-10.0)+2(2)=20+4=24\). For H, \(\operatorname{AIC}_H=-2(-10.293147)+4=20.586294+4=24.586294\). Rounded to three decimal places, the values are 24.000 and 24.586, so AIC prefers G.
As a check, the original-scale log-likelihood for H is lower than G's, and the models have equal parameter counts. Their AIC ranking must therefore match their likelihood ranking. Comparing the unadjusted value \(-9.6\) would wrongly suggest H fits better.
Interpretation: Ranking Is Not a Probability or a Test
AIC and BIC are relative model-selection criteria: among the candidates supplied, each ranks models by combining fit and a complexity penalty. A difference of, for example, 2 does not mean a model has a 2% probability of being correct. Nor does an AIC or BIC value have a universal threshold at which a model is accepted or rejected. These criteria are not, by themselves, formal hypothesis tests with a p-value.
The ranking also does not establish that the preferred model is an adequate description of the data. It is conditional on the candidates and on the likelihoods being appropriately specified and comparable. Goodness-of-fit evidence and actuarial consequences remain relevant when assessing a selected severity model.
A Practical Error-Check
Check that each candidate uses the same observed response and a compatible likelihood and sampling setup.
Use the maximized log-likelihood, retain any necessary transformation or likelihood terms, and calculate \(-2\hat{\ell}\) with its sign intact.
Count all estimated parameters consistently. For BIC, identify the sample size that matches the independent observations represented by the likelihood.
Add the penalty, select the smaller criterion among comparable candidates, and do not report the result as a probability or hypothesis test.
Check Your Understanding
For each problem, calculate the requested criterion and state the preferred candidate where applicable.
- Two severity models are fitted to the same 60 losses. Model J has \(\hat{\ell}=-90\), \(k=2\), and model K has \(\hat{\ell}=-86\), \(k=4\). Calculate both AIC values and identify the preferred model. (Answer: \(\operatorname{AIC}_J=184\), \(\operatorname{AIC}_K=180\); K.)
- For \(n=50\), model L has \(\hat{\ell}=-40\), \(k=2\), and model M has \(\hat{\ell}=-37\), \(k=3\). Calculate both BIC values, rounded to three decimals, and select the lower one. (Answer: \(\operatorname{BIC}_L=87.824\), \(\operatorname{BIC}_M=85.736\); M.)
- For \(n=25\), model N has \(\hat{\ell}=-30\), \(k=2\). Model O has \(\hat{\ell}=-27.5\) and actually estimates four parameters, although an analyst counts only two. Calculate the correct BIC values, rounded to three decimals, and state how the incorrect count changes the choice. (Answer: \(\operatorname{BIC}_N=66.438\), \(\operatorname{BIC}_O=67.876\), so N is preferred; using \(k_O=2\) gives 61.438 and falsely favors O.)
- Thirty independent losses are fitted individually, then displayed in five bins. Model P has \(\hat{\ell}=-52.5\), \(k=2\), and model Q has \(\hat{\ell}=-50\), \(k=4\). Calculate BIC using the correct \(n\), rounded to three decimals. (Answer: use \(n=30\), not 5; \(\operatorname{BIC}_P=111.802\), \(\operatorname{BIC}_Q=113.605\); P.)
- For data \(x=(1,2)\), model R has original-scale \(\hat{\ell}=-10.0\) and model S has log-scale \(\hat{\ell}=-9.6\); both have \(k=2\). Convert S's likelihood to the original scale and compare AIC, rounded to three decimals. (Answer: \(\hat{\ell}_S=-9.6-\ln(1)-\ln(2)=-10.293\); \(\operatorname{AIC}_R=24.000\), \(\operatorname{AIC}_S=24.586\); R.)