Comparing AIC And BIC Penalties As Sample Size Grows

Why Compare the Penalties?

A more flexible model can often achieve a higher maximized likelihood because it has more parameters to fit. A model-selection criterion balances that fit improvement against a penalty for complexity. AIC and BIC use the same fit term but different penalties: AIC charges a fixed amount per parameter, while BIC’s charge grows with the sample size. Consequently, the two criteria can prefer different models even when calculated from the same observations and likelihoods.

The comparison is meaningful when both candidate models are fitted to the same observations using comparable likelihoods. Let $n$ be the number of observations contributing to the likelihood, let $k_j$ be the number of free parameters in model $j$, and let $\ell_j$ be that model’s maximized log likelihood. Smaller criterion values are preferred.

Definition: For a model with maximized log likelihood $\ell$, $k$ free parameters, and sample size $n$, the Akaike information criterion and Bayesian information criterion are $$ \mathrm{AIC}=-2\ell+2k, \qquad \mathrm{BIC}=-2\ell+k\ln n. $$ The fit term is $-2\ell$. The AIC penalty is $2k$, and the BIC penalty is $k\ln n$.

The parameter count refers to free parameters estimated from the data, not every quantity appearing in a model formula. The logarithm in BIC is the natural logarithm. In standard applications, $n$ is the number of independent observations used in the likelihood; it is not the number of parameters or the number of candidate models.

Deriving the Ranking Thresholds

Consider a simpler model $M_0$ and a more complex model $M_1$, fitted to the same $n$ observations. Suppose $M_1$ has $q$ additional free parameters, so $k_1=k_0+q$, where $q>0$. Define the improvement in maximized log likelihood as $\Delta\ell=\ell_1-\ell_0$. When the models are nested and fitted by maximum likelihood, adding parameters cannot reduce the maximized likelihood, so $\Delta\ell\geq 0$.

Subtracting the simpler model’s score from the more complex model’s score gives the change in each criterion. A negative difference means the more complex model has the smaller criterion and is preferred.

\begin{aligned} \mathrm{AIC}_1-\mathrm{AIC}_0 &=-2(\ell_1-\ell_0)+2(k_1-k_0)\\ &=-2\Delta\ell+2q,\\[4pt] \mathrm{BIC}_1-\mathrm{BIC}_0 &=-2(\ell_1-\ell_0)+(k_1-k_0)\ln n\\ &=-2\Delta\ell+q\ln n. \end{aligned}

These differences give the decision rules directly. AIC prefers the more complex model when $\Delta\ell>q$. BIC prefers it when $\Delta\ell>q\ln(n)/2$. Equality gives a tie under that criterion. The comparison depends on the likelihood improvement per added parameter, not merely on whether the more complex model fits better.

Result: For a model with $q>0$ additional parameters, AIC’s log-likelihood improvement threshold is $q$, while BIC’s is $q\ln(n)/2$. The two rankings can diverge when $\Delta\ell$ lies between these thresholds. For $n>e^2$, BIC has the higher threshold; for $n<e^2$, it has the lower threshold. At $n=e^2$, the thresholds are equal.

The reason is visible by comparing the penalties themselves. BIC’s penalty per parameter is $\ln n$; AIC’s is $2$. They are equal when $\ln n=2$, or $n=e^2\approx 7.389$. For integer sample sizes greater than 7, BIC’s penalty per parameter exceeds AIC’s; for integer sample sizes less than 7, it is smaller. This is a comparison of the formulas, not a claim that BIC is generally preferable at small sample sizes.

Worked Comparisons

Worked Example: AIC Accepts an Extra Severity Parameter but BIC Does Not

A hypothetical insurer fits two severity models to the same $n=100$ independent losses. The simpler model has $k_0=2$ parameters and maximized log likelihood $\ell_0=-120$. The more complex model has $k_1=3$ parameters and maximized log likelihood $\ell_1=-118.5$. Calculate both criteria and compare the rankings. Use the maximized log likelihoods as given; round criterion values to three decimal places.

Method. Calculate each model’s AIC and BIC, or equivalently calculate the criterion differences. Here $q=3-2=1$, and $\Delta\ell=-118.5-(-120)=1.5$.

For AIC, the improvement threshold is $q=1$. Since $1.5>1$, AIC should prefer the more complex model. Direct calculation confirms this:

\begin{aligned} \mathrm{AIC}_0&=-2(-120)+2(2)=240+4=244,\\ \mathrm{AIC}_1&=-2(-118.5)+2(3)=237+6=243. \end{aligned}

The AIC difference is $243-244=-1$, so the more complex model has the lower AIC.

For BIC, the improvement threshold is $q\ln(100)/2=\ln(100)/2\approx 2.303$. Since $1.5<2.303$, BIC should prefer the simpler model. Direct calculation gives:

\begin{aligned} \mathrm{BIC}_0&=240+2\ln(100)\approx249.210,\\ \mathrm{BIC}_1&=237+3\ln(100)\approx250.816. \end{aligned}

The BIC difference is approximately $250.816-249.210=1.605$, favoring the simpler model. The result is checked both by the score calculations and by the thresholds: $1<1.5<2.303$. The common mistake is to conclude that a higher likelihood automatically wins under both criteria; each criterion charges for the extra parameter.

Worked Example: BIC’s Penalty Can Be Smaller for a Small Sample

A hypothetical analyst compares two fitted models for $n=4$ observed claim amounts. The simpler model has $k_0=2$ and $\ell_0=-20$; the more complex model has $k_1=3$ and $\ell_1=-19.2$. Calculate the scores and state which model each criterion prefers. Round scores to three decimal places.

Method. The complex model adds $q=1$ parameter and improves the maximized log likelihood by $\Delta\ell=-19.2-(-20)=0.8$. AIC’s threshold is $1$. BIC’s threshold is $\ln(4)/2\approx0.693$. Thus AIC should prefer the simpler model, while BIC should prefer the more complex model.

The AIC scores are

\begin{aligned} \mathrm{AIC}_0&=-2(-20)+2(2)=40+4=44,\\ \mathrm{AIC}_1&=-2(-19.2)+2(3)=38.4+6=44.4. \end{aligned}

AIC prefers the simpler model because $44<44.4$. For BIC, using $\ln(4)\approx1.386294$,

\begin{aligned} \mathrm{BIC}_0&=40+2\ln(4)\approx42.773,\\ \mathrm{BIC}_1&=38.4+3\ln(4)\approx42.559. \end{aligned}

BIC prefers the more complex model because $42.559<42.773$. The BIC difference is approximately $-0.214$, consistent with its threshold: $0.8>0.693$. This example illustrates the formula’s small-$n$ behavior. It does not establish that BIC is a better choice for a small sample.

Worked Example: Finding the Sample-Size Crossover

A hypothetical analyst compares a base model with $k_0=4$, $\ell_0=-510$, and an expanded model with $k_1=6$, $\ell_1=-505$. Both use the same independent observations. Determine the sample sizes at which AIC and BIC prefer the expanded model, and compare their decisions at $n=100$ and $n=200$.

Method. The expanded model adds $q=6-4=2$ parameters, and its log-likelihood improvement is $\Delta\ell=-505-(-510)=5$. AIC prefers it when $5>2$, which holds regardless of $n$. BIC prefers it when

5>\frac{2\ln n}{2}=\ln n, \qquad\text{or equivalently}\qquad n<e^5\approx148.413.

At $n=100$, $\ln(100)\approx4.605$, so BIC prefers the expanded model as well as AIC. At $n=200$, $\ln(200)\approx5.298$, so BIC prefers the base model, while AIC still prefers the expanded model. At $n=e^5$, BIC gives a tie. The sample size changes BIC’s penalty threshold, not the observed log-likelihood improvement of 5.

The common mistake is to say that BIC eventually favors the more complex model because more observations provide more information. In this comparison, the likelihood improvement is held fixed while $n$ increases; the BIC threshold rises, so a fixed improvement can eventually fail to justify the extra parameters.

General Comparison and Interpretation

The same subtraction works for two candidate models that are not described as a simple-versus-expanded pair. Label them $M_1$ and $M_0$, and define $\Delta k=k_1-k_0$ and $\Delta\ell=\ell_1-\ell_0$. Then

\mathrm{AIC}_1-\mathrm{AIC}_0=-2\Delta\ell+2\Delta k, \qquad \mathrm{BIC}_1-\mathrm{BIC}_0=-2\Delta\ell+\Delta k\ln n.

These equations remain valid whether $\Delta k$ is positive, zero, or negative. If the models have the same parameter count, both criteria rank them using the same maximized likelihood, because their penalty differences are zero. If one model has fewer parameters, relabeling the models so the simpler one is $M_0$ makes the earlier thresholds easier to apply.

More generally, subtracting the two score differences yields

(\mathrm{BIC}_1-\mathrm{BIC}_0)-(\mathrm{AIC}_1-\mathrm{AIC}_0) =\Delta k(\ln n-2).

For a comparison in which $M_1$ has extra parameters, this expression shows exactly how the BIC score difference shifts relative to the AIC score difference as the sample size changes. When $n>e^2$, the BIC difference is larger: relative to AIC, BIC is less favorable to the added complexity. When $n<e^2$, the BIC difference is smaller. This algebra explains how the criteria can disagree without either calculation being inconsistent.

AIC and BIC reflect different model-selection aims and approximations. For this comparison, the operational lesson is to calculate each score from the same likelihood and parameter counts, then state the ranking under each criterion separately. Agreement is possible, but it should not be assumed.

Common Traps

Reversing the preferred direction. Both criteria are minimized. If $\mathrm{AIC}_1-\mathrm{AIC}_0<0$, model 1 has the lower AIC and is preferred by AIC.
Using $\log_{10} n$ instead of $\ln n$. BIC uses the natural logarithm. Changing the logarithm changes the penalty and can change the ranking.
Counting the wrong $n$. Use the sample size associated with the likelihood being compared, not the number of model parameters or the number of groups of candidate models.
Assuming a larger likelihood guarantees a smaller criterion. An added parameter can improve the likelihood but still increase AIC or BIC after its penalty is included.
Comparing scores from noncomparable likelihoods. The subtraction argument assumes both models use the same observations and compatible likelihood definitions. A numerical score difference is not meaningful if the underlying likelihoods are not comparable.

Key takeaway: For $q$ added parameters, compare the log-likelihood improvement with $q$ for AIC and $q\ln(n)/2$ for BIC. Their thresholds are equal at $n=e^2$; above that sample size BIC demands a greater improvement than AIC to favor the more complex model.

Check Your Understanding

For each problem, assume the candidate models use the same observations and comparable maximized log likelihoods. A negative score difference for the more complex model means that criterion prefers it.

A more complex model adds one parameter and improves the maximized log likelihood by $2.5$. The sample size is $64$. Which model does each criterion prefer? (Answer: Both AIC and BIC prefer the more complex model.)
A model with two additional parameters improves the maximized log likelihood by $4$, using $n=1000$ observations. Which model does each criterion prefer? (Answer: AIC prefers the more complex model; BIC prefers the simpler model.)
A model adds one parameter and improves the maximized log likelihood by $1.05$, with $n=9$. Which model does each criterion prefer? (Answer: AIC prefers the more complex model; BIC prefers the simpler model.)
A model adds two parameters and improves the maximized log likelihood by $1.2$, with $n=3$. Which model does each criterion prefer? (Answer: AIC prefers the simpler model; BIC prefers the more complex model.)
A model adds one parameter and improves the maximized log likelihood by $1.4$. For what integer sample sizes does AIC prefer the more complex model while BIC prefers the simpler model? (Answer: Every integer $n\geq17$. AIC prefers the complex model because $1.4>1$; BIC prefers the simpler model when $\ln(n)/2>1.4$, or $n>e^{2.8}\approx16.445$.)

What You'll Learn