Choosing A Criterion For Prediction Or Parsimony

Introduction: Let the Intended Use Guide the Criterion

A selected model may be used to predict future claims, summarize a portfolio, or support a decision that favors a compact specification. Those goals are related, but they are not identical. A model that is useful for prediction need not be the simplest model, and the model with the fewest parameters need not predict best. AIC and BIC express different trade-offs between fit and complexity, so the intended use should guide how their results are interpreted.

Neither criterion is a universal rule for choosing a model. Both compare candidate models by balancing a measure of fit against a parameter-count penalty. The fit must be evaluated on comparable data using consistent likelihood calculations; otherwise the comparison is not meaningful. The smaller criterion value is preferred under that criterion, not necessarily for every actuarial purpose.

Definition: For a candidate model with maximized log-likelihood $\ell(\hat{\theta})$, $k$ estimated free parameters, and $n$ observations, the Akaike information criterion and Bayesian information criterion are $$ \mathrm{AIC}=-2\ell(\hat{\theta})+2k, \qquad \mathrm{BIC}=-2\ell(\hat{\theta})+k\ln(n). $$ The model with the smaller value is preferred by the corresponding criterion. The likelihood and observation count must be based on the same data and a consistent likelihood convention across candidates.

Here $k$ counts estimated free parameters, not merely the number of distribution names or model terms. The BIC expression uses the natural logarithm. If two models have the same fitted likelihood and one has more parameters, both criteria favor the model with fewer parameters. If the more complex model fits better, the decision depends on whether the improvement offsets the criterion’s extra penalty.

Why the Penalties Differ

AIC is motivated by estimating relative expected information loss when a fitted model is used to represent data generated from an underlying process. Fitting parameters to a sample tends to make the model appear better on that sample than it will perform on new observations. Under regularity conditions, the AIC penalty $2k$ corrects for this fitting optimism, to first order. This motivates AIC when the central goal is predictive performance, including when the candidate models are approximations and none is exactly true.

BIC is motivated by approximating the evidence for a model through its marginal likelihood. Its penalty grows with sample size: $k\ln(n)$. This increasingly penalizes additional parameters as more observations become available. Under suitable regularity conditions, including a finite candidate set and a correctly specified model among the candidates, BIC has a model-identification consistency property: as $n$ grows, it selects the true candidate with probability approaching one. These conditions matter. BIC does not guarantee that a selected model is true, nor does its name make it automatically preferable for every Bayesian or actuarial application.

\mathrm{BIC}-\mathrm{AIC}=k\bigl(\ln(n)-2\bigr).

For a given model with $k>0$, the BIC penalty exceeds the AIC penalty when $n>e^2$, approximately 7.39. In comparing two models, however, the relevant quantity is the difference in their parameter counts. For a more complex model with $r$ additional parameters and an improvement of $G$ in $-2\ell(\hat{\theta})$, the differences are

\Delta\mathrm{AIC}=-G+2r, \qquad \Delta\mathrm{BIC}=-G+r\ln(n),

where each difference is “complex model minus simpler model.” A negative difference favors the complex model; a positive difference favors the simpler one. The equations show directly why the sample size and intended objective can affect the decision.

A Short Justification for the BIC Penalty

The BIC penalty can be understood through a large-sample approximation to a model’s marginal likelihood. Suppose a model has $k$ regular, identifiable parameters and a smooth prior density that is positive near the maximum-likelihood estimate. The marginal likelihood integrates the likelihood over parameter values rather than evaluating it only at its maximum. Near a well-behaved maximum, the log-likelihood is approximately quadratic in the $k$ parameter directions. As $n$ increases, the region around the maximum that contributes materially to the integral has width on the order of $n^{-1/2}$ in each direction. Its volume is therefore on the order of $n^{-k/2}$.

Consequently, up to factors whose logarithms remain bounded as $n$ grows, the marginal likelihood is approximately the maximized likelihood multiplied by $n^{-k/2}$. Taking negative twice the logarithm gives the leading terms

-2\ln(\text{marginal likelihood}) \approx -2\ell(\hat{\theta})+k\ln(n)+O(1).

Dropping the bounded terms that do not grow with $n$ yields the BIC formula. This argument explains both the parameter penalty and why regularity matters. It is an asymptotic approximation, not a claim that every model’s exact marginal likelihood equals BIC.

Worked Comparisons

Worked Example: Predictive Fit Versus a Parsimony Preference

A hypothetical insurer fits two severity models to the same 40 positive claim amounts, with no censoring or truncation. Model D is a two-parameter lognormal distribution. Model E is a three-parameter generalized gamma distribution. Suppose the maximized log-likelihoods are $-25.5$ and $-23.7$, respectively. Compare AIC and BIC, and explain what a prediction-oriented analyst and a parsimony-oriented analyst should conclude.

Method and inputs. Use the same 40 observations and the stated maximized likelihood for each model. The parameter counts are $k_D=2$ and $k_E=3$. First calculate the deviance terms:

-2\ell_D=-2(-25.5)=51.0, \qquad -2\ell_E=-2(-23.7)=47.4.

For AIC, the two-parameter model gives

\mathrm{AIC}_D=51.0+2(2)=55.0.

The three-parameter model gives

\mathrm{AIC}_E=47.4+2(3)=53.4.

Thus AIC favors Model E by $55.0-53.4=1.6$ AIC units. Its improvement in fit is $51.0-47.4=3.6$, which exceeds the additional AIC penalty of $2(3-2)=2$.

For BIC, $\ln(40)\approx3.688879454$. Therefore,

\begin{aligned} \mathrm{BIC}_D &=51.0+2(3.688879454)\\ &=51.0+7.377758908\\ &=58.377758908\approx58.378,\\[4pt] \mathrm{BIC}_E &=47.4+3(3.688879454)\\ &=47.4+11.066638362\\ &=58.466638362\approx58.467. \end{aligned}

BIC favors Model D, but only slightly: its value is approximately $0.089$ lower. The calculation checks because both models use the same observations and likelihood convention, and Model E’s fit improvement of $3.6$ is just below BIC’s extra penalty $\ln(40)\approx3.689$. For a prediction-focused decision, AIC supports considering Model E; for an objective that places greater weight on a compact model and the conditions for BIC’s interpretation are plausible, BIC supports Model D. Neither comparison alone establishes that either severity distribution is adequate.

Common mistake avoided: Do not say that BIC always chooses the simplest model. It chooses the simpler model here because the improvement in fit does not offset the additional BIC penalty.

Worked Example: How Sample Size Can Change the BIC Decision

A hypothetical analyst compares an exponential severity model with one free parameter against a gamma severity model with two free parameters. Both are fitted to the same 100 uncensored positive losses. The exponential model has $-2\ell=120$, and the gamma model has $-2\ell=116$. Determine each criterion’s preference and explain the role of sample size.

Method. The gamma model improves the deviance by $G=120-116=4$, at the cost of one additional parameter, so $r=1$. The AIC difference, gamma minus exponential, is

\Delta\mathrm{AIC}=-4+2(1)=-2.

A negative difference means AIC favors the gamma model. Direct calculation confirms this:

\mathrm{AIC}_{\text{exp}}=120+2(1)=122, \qquad \mathrm{AIC}_{\text{gamma}}=116+2(2)=120.

For BIC, $\ln(100)\approx4.605170186$, so

\Delta\mathrm{BIC}=-4+4.605170186=0.605170186.

This positive difference favors the exponential model. The values are

\mathrm{BIC}_{\text{exp}}=120+\ln(100)\approx124.605, \qquad \mathrm{BIC}_{\text{gamma}}=116+2\ln(100)\approx125.210.

The gamma model’s fit gain is four deviance units. That exceeds AIC’s additional penalty of two, but not BIC’s additional penalty of approximately $4.605$ at $n=100$. If the observation count were different, the BIC penalty for this extra parameter would be $\ln(n)$, so the conclusion could change. The check is $\Delta\mathrm{BIC}\approx125.210-124.605=0.605$, agreeing with the difference calculation.

Common mistake avoided: Do not use the total number of parameters in the difference formula as if both models gained parameters. Only the difference $r=2-1=1$ receives the incremental penalty.

Worked Example: Matching the Criterion to an Actuarial Objective

A hypothetical mutual insurer models annual claim counts for 80 policy-years using two candidate distributions. Model P is a two-parameter negative binomial model with $-2\ell=310$. Model Q is a three-parameter alternative count model with $-2\ell=304$. The insurer is considering a model for prospective pricing forecasts but also needs a compact model for a management summary. Compare the criteria and make a purpose-specific recommendation.

Step 1: Calculate AIC. For Model P,

\mathrm{AIC}_P=310+2(2)=314.

For Model Q,

\mathrm{AIC}_Q=304+2(3)=310.

AIC favors Model Q by $314-310=4$. The fit improvement is $310-304=6$ deviance units, while its extra parameter costs two AIC units.

Step 2: Calculate BIC. Since $\ln(80)\approx4.382026635$,

\begin{aligned} \mathrm{BIC}_P &=310+2(4.382026635)=318.764053270\approx318.764,\\ \mathrm{BIC}_Q &=304+3(4.382026635)=317.146079905\approx317.146. \end{aligned}

BIC also favors Model Q, by approximately $318.764-317.146=1.618$. Equivalently, its difference is $-6+\ln(80)\approx-1.618$, which is negative. The calculations agree.

For prospective forecasting, AIC’s predictive-risk motivation makes it a natural primary comparison, and it favors Q. For the management summary, parsimony may still be valuable, but it does not justify claiming that BIC favors P: BIC also favors Q in these data. If management requires a compact presentation, the simpler model could be reported as a deliberate communication choice, with the predictive fit trade-off disclosed. A criterion’s name or a general preference for simplicity should not override its calculated result.

Common mistake avoided: Do not choose a criterion after seeing which one supports a preferred answer. State the intended use first, calculate the criteria consistently, and explain what each result does and does not establish.

Making a Defensible Choice

The choice between AIC and BIC is best framed as a choice about the objective and assumptions, not as a contest in which one criterion is always superior. In an actuarial setting, a model intended to represent future claims may justify emphasizing predictive performance. AIC is aligned with that aim under its regularity assumptions. If the task emphasizes identifying a compact candidate from a well-specified finite set as the sample grows, BIC’s stronger sample-size-dependent penalty may be relevant.

These are motivations, not guarantees about a particular portfolio. AIC does not ensure accurate predictions for every tail or subpopulation, and BIC does not ensure that a selected model is simple enough for every operational purpose. A criterion summarizes fit and complexity in one number; it does not replace checking whether the model assumptions and data treatment are appropriate. For example, candidates fitted to different observations or using inconsistent likelihood terms cannot be fairly ranked by their AIC or BIC values.

When the criteria disagree, report the conflict and connect it to the use case. Ask how much fit improvement the more complex model supplies, how many extra parameters it requires, and whether the sample size makes the BIC penalty materially larger than AIC’s. A close difference should be described as close rather than converted into a categorical claim of decisive support. Earlier material on interpreting information-criterion differences and resolving conflicts with diagnostics is relevant when the numerical comparison alone is inconclusive.

Key takeaway: Use AIC when the primary objective is expected predictive performance and its assumptions are reasonable; consider BIC when a stronger, sample-size-dependent penalty and model-identification objective are appropriate. Always state the objective, compare models on the same data, and interpret the result as criterion-specific evidence rather than a universal verdict.

Check Your Understanding

For each numerical question, use differences defined as complex model minus simpler model. A negative difference favors the complex model; a positive difference favors the simpler model.

A model with one extra parameter reduces $-2\ell$ by $2.5$ units in a sample of $n=50$. Find the AIC and BIC differences. (Answer: $\Delta\mathrm{AIC}=-0.5$, so AIC favors the complex model; $\Delta\mathrm{BIC}=-2.5+\ln(50)\approx1.412$, so BIC favors the simpler model.)
A model with two extra parameters improves $-2\ell$ by $4$ units in a sample of $n=20$. Find both differences and identify any tie. (Answer: $\Delta\mathrm{AIC}=-4+2(2)=0$, an AIC tie; $\Delta\mathrm{BIC}=-4+2\ln(20)\approx1.991$, so BIC favors the simpler model.)
In a sample of $n=120$, a model with one extra parameter improves $-2\ell$ by $5$ units. Which model does each criterion favor? (Answer: $\Delta\mathrm{AIC}=-5+2=-3$ and $\Delta\mathrm{BIC}=-5+\ln(120)\approx-0.213$; both favor the complex model.)
A team is selecting a model primarily to forecast future claim counts, and a second team is selecting a model from a finite candidate set with a strong preference for identifying a compact, correctly specified model as the sample grows. Which criterion’s usual motivation better matches each objective? (Answer: AIC’s predictive-risk motivation better matches the forecasting objective; BIC’s model-identification and stronger parsimony motivation better matches the second objective, subject to its assumptions.)
For $n=30$, a model with one additional parameter improves $-2\ell$ by $3.2$. Compute the criterion differences to three decimals. (Answer: $\Delta\mathrm{AIC}=-3.2+2=-1.200$, favoring the complex model; $\Delta\mathrm{BIC}=-3.2+\ln(30)\approx0.201$, favoring the simpler model.)

What You'll Learn