Why Compare the AIC and BIC Penalties?
AIC and BIC both combine a measure of model fit with a penalty for model complexity. Their fit contributions use the same scale: each includes \(-2\ell\), where \(\ell\) is the maximized log-likelihood. Their penalties differ. AIC charges a fixed amount per parameter, while BIC’s per-parameter charge depends on the sample size. Comparing these penalties shows exactly when BIC applies the larger complexity charge.
The comparison is about the penalties, not automatically about which model is preferable. When comparing two models, their fit contributions may differ as well. The penalty comparison isolates one part of that decision and is useful when assessing how the criteria treat model complexity.
The comparison assumes both criteria are computed for the same model, using the same observations, maximized likelihood, and parameter-count convention. In Gaussian linear regression, the counted parameters must be treated consistently—for example, whether the error variance is included in \(k\). A parameter counted in one criterion should also be counted in the other. AIC and BIC conventions can include terms that are constant across the models being compared; such common terms do not change the penalty comparison.
Deriving the Crossover Sample Size
For one parameter, AIC’s penalty is \(2\) and BIC’s is \(\ln n\). The crossover occurs where those charges are equal. The natural logarithm is increasing, so solving this equality also tells us which criterion has the larger per-parameter penalty on either side of the crossover.
The per-parameter penalties match when \(\ln n=2\).
Exponentiating both sides gives \(n=e^2\).
Since \(e^2\approx7.3891\), BIC’s per-parameter penalty is larger when \(n>e^2\), and AIC’s is larger when \(n<e^2\).
Sample size is ordinarily a positive integer. Therefore, at \(n=7\), BIC’s per-parameter penalty is still slightly smaller than AIC’s; at \(n=8\), it is larger. For integer sample sizes, BIC’s per-parameter penalty exceeds AIC’s beginning at \(n=8\). At the noninteger value \(e^2\), the two per-parameter penalties are exactly equal.
Calculating the Penalty Difference
For a model with \(k\) parameters, define the penalty difference as BIC’s penalty minus AIC’s penalty. This sign convention makes a positive result mean that BIC imposes the larger penalty.
For \(r\) additional parameters, replace \(k\) with \(r\) to compare the penalties associated with those parameters. The per-parameter gap depends only on \(n\); the total gap is multiplied by the number of parameters. Use the natural logarithm, not the base-10 logarithm.
| Sample size \(n\) | AIC penalty per parameter | BIC penalty per parameter | Larger per-parameter penalty |
|---|---|---|---|
| 5 | 2 | \(\ln 5\approx1.6094\) | AIC |
| 7 | 2 | \(\ln 7\approx1.9459\) | AIC |
| 8 | 2 | \(\ln 8\approx2.0794\) | BIC |
| 50 | 2 | \(\ln 50\approx3.9120\) | BIC |
Worked Example: A Model With Six Parameters at a Small Sample Size
A hypothetical model is fitted to \(n=6\) observations and has \(k=5\) estimated parameters under the convention used for both criteria. Calculate the two complexity penalties and their difference, then identify the larger penalty. Use four decimal places for the difference.
Method. Calculate \(2k\) for AIC and \(k\ln n\) for BIC, then subtract AIC’s penalty from BIC’s. This avoids comparing the formulas’ per-parameter amounts while forgetting to multiply by the parameter count.
AIC’s penalty is \(2k=2(5)=10\). For BIC, \(\ln 6=1.7917594692\ldots\), so its penalty is \(5\ln 6=5(1.7917594692\ldots)=8.958797346\ldots\).
The difference is \(8.958797346\ldots-10=-1.041202654\ldots\), which rounds to \(-1.0412\). The negative sign means BIC’s complexity penalty is smaller here. This agrees with \(6<e^2\). As a check, the per-parameter penalties are approximately \(1.7918\) for BIC and \(2\) for AIC; multiplying their difference by five gives the same total difference.
Worked Example: Three Parameters at the First Integer Sample Size Above the Crossover
A hypothetical model extension introduces \(r=3\) additional estimated parameters. The data set contains \(n=8\) observations. Compare the AIC and BIC penalty increases for those parameters, using four decimal places for the difference.
Method. Apply each criterion’s per-parameter penalty to all three added parameters. The crossover derivation says BIC should be larger at \(n=8\), but the calculation verifies the size of the difference.
The AIC penalty increase is \(2r=2(3)=6\). Since \(\ln 8=2.0794415417\ldots\), the BIC penalty increase is \(r\ln 8=3(2.0794415417\ldots)=6.2383246251\ldots\).
Subtracting gives \(6.2383246251\ldots-6=0.2383246251\ldots\), or \(0.2383\) to four decimal places. BIC’s penalty increase is larger by \(0.2383\). This is a penalty comparison only: a full score comparison would also require the change in \(-2\ell\) between the models.
Worked Example: Four Added Parameters With Fifty Observations
A hypothetical regression extension adds \(r=4\) parameters to a candidate model fitted using the same \(n=50\) observations. Calculate the increase in each criterion’s complexity penalty and their difference. Report the difference to four decimal places.
Method. Calculate \(2r\) for AIC and \(r\ln n\) for BIC. This calculation concerns the extra penalties only; it does not assert that the extension is favored or rejected.
AIC’s penalty increase is \(2r=2(4)=8\). For BIC, \(\ln 50=3.9120230054\ldots\), so \(4\ln 50=4(3.9120230054\ldots)=15.6480920216\ldots\).
The difference is \(15.6480920216\ldots-8=7.6480920216\ldots\), which rounds to \(7.6481\). Thus BIC charges \(7.6481\) more than AIC for these four parameters. Equivalently, \(4(\ln 50-2)=4(1.9120230054\ldots)=7.6480920216\ldots\), confirming the result.
What the Penalty Comparison Does—and Does Not—Tell You
For the same fitted model, the \(-2\ell\) term is identical in AIC and BIC. Consequently, BIC’s full score minus AIC’s full score for that model equals the penalty difference \(k(\ln n-2)\). This does not mean that AIC and BIC always rank different models in the same way or that a larger penalty alone decides which model has the lower full score.
When comparing a smaller model with an extension, both the fit change and the penalty change matter. The extension may improve the maximized log-likelihood, changing \(-2\ell\), while also incurring additional complexity penalties. The calculations here isolate the latter; deciding between full scores requires including both components consistently.
The sample-size effect is direct: as \(n\) increases, \(\ln n\) increases, so BIC’s charge per parameter grows, while AIC’s remains fixed at \(2\). This is why the crossover is useful as a quick diagnostic. It tells you which criterion charges more for complexity at a given \(n\), but it does not by itself establish the overall model ranking.
Common Mistake: Using the Wrong Logarithm or Threshold
A frequent error is to use \(\log_{10} n\) in BIC’s penalty, or to round the crossover \(e^2\approx7.3891\) to \(7\) and conclude that BIC’s penalty is larger at \(n=7\). The BIC formula uses the natural logarithm, and the inequality must be checked at the actual sample size.
For example, the incorrect calculation \(\log_{10}8=0.9031<2\) would suggest AIC has the larger per-parameter penalty at \(n=8\). The correction is to use \(\ln 8=2.0794415417\ldots>2\), so BIC’s penalty is larger. At \(n=7\), \(\ln 7=1.9459101491\ldots<2\), so AIC’s per-parameter penalty remains slightly larger. The integer boundary is therefore \(n=8\), not \(n=7\).
Check Your Understanding
For each problem, calculate or interpret the penalty difference as BIC’s penalty minus AIC’s penalty. Use the natural logarithm.
- A model has \(k=2\) counted parameters and is fitted to \(n=5\) observations. Calculate BIC’s total penalty minus AIC’s total penalty to four decimal places, and state which penalty is larger. (Answer: \(2(\ln 5-2)=-0.7811\); AIC’s penalty is larger.)
- A study has \(n=10\) observations. Compare the per-parameter penalties and calculate BIC’s excess per parameter to four decimal places. (Answer: \(\ln 10-2=0.3026\); BIC’s per-parameter penalty is larger.)
- An extension adds \(r=3\) parameters to a model fitted to \(n=7\) observations. Find the difference between the penalty increases to four decimal places. (Answer: \(3(\ln 7-2)=-0.1623\); AIC’s penalty increase is larger.)
- A model with \(k=2\) parameters is fitted using \(n=25\) observations. Calculate the BIC-minus-AIC penalty difference to four decimal places. (Answer: \(2(\ln 25-2)=2.4378\); BIC’s penalty is larger.)
- A model has \(k=6\) parameters and is fitted to \(n=100\) observations. Calculate BIC’s total penalty minus AIC’s total penalty to four decimal places, and explain whether the result agrees with the crossover rule. (Answer: \(6(\ln 100-2)=15.6310\); BIC’s penalty is larger, as \(100>e^2\).)