AIC Versus BIC

Why Two Information Criteria?

A regression model can fit the observed data well by including many terms, but added complexity has a cost. AIC and BIC both compare fit with complexity, yet they charge different penalties and are motivated by different aims. Consequently, the same pair of models can receive opposite rankings without either calculation being wrong.

AIC is associated with estimating expected predictive information loss: a model with a smaller AIC is preferred for that criterion’s predictive aim. BIC uses a larger, sample-size-dependent penalty and is an approximation to Bayesian model evidence under regularity conditions. A smaller BIC therefore supports a different kind of preference. Neither criterion is a universal measure of whether a model is “true,” and neither replaces checking whether the candidate models and their assumptions are appropriate.

Models and the Two Penalties

For the comparisons below, suppose each candidate is a Gaussian linear regression fitted to the same response measurements on the same \(n\) cases. For candidate model \(j\), write \(Y_i=\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}_j+\varepsilon_i\), where the errors are independent and each has distribution \(N(0,\sigma^2)\). Within a candidate, the error variance is common across cases. The regression coefficients and variance are estimated by maximum likelihood. We compare models using their maximized likelihoods, so the response, cases, and likelihood basis must be common.

The AIC penalty is two units for each estimated parameter. The BIC penalty is \(\ln(n)\) units per parameter. Thus, when \(n\) is greater than \(e^2\), BIC charges more per parameter than AIC. The penalty difference becomes more pronounced as sample size increases. For a comparison in which the larger model adds \(q\) regression coefficients, the additional penalties are \(2q\) for AIC and \(q\ln(n)\) for BIC.

For Gaussian least-squares regression, the likelihood can be written in terms of the residual sum of squares, or RSS. With the maximum-likelihood estimate \(\hat{\sigma}^2=\mathrm{RSS}/n\), the criteria, apart from the same constant for all models in the comparison, are:

The shared constant does not affect rankings. In the examples, \(\Delta\mathrm{AIC}\) or \(\Delta\mathrm{BIC}\) means the larger model’s criterion minus the simpler model’s criterion. A negative difference favors the larger model; a positive difference favors the simpler one.

Comparing the Criteria by Their Differences

Suppose the larger model reduces RSS from \(\mathrm{RSS}_0\) to \(\mathrm{RSS}_1\), where \(\mathrm{RSS}_1<\mathrm{RSS}_0\), and adds \(q\) regression coefficients. Define \(G\) as the improvement in the fit component of the criteria. The same fit improvement enters AIC and BIC, but their complexity penalties differ.

These equations give a direct decision rule. AIC favors the larger model if \(G>2q\); BIC favors it if \(G>q\ln(n)\). When \(n>e^2\), the BIC threshold is higher. A fit improvement between these two thresholds makes the criteria disagree.

A Short Derivation of the Difference Formula

For a Gaussian model with independent errors, the maximized log-likelihood is \(\ell(\hat{\theta})=-\frac{n}{2}\{\ln(2\pi)+1+\ln(\mathrm{RSS}/n)\}\). Therefore, the difference in \(-2\ell(\hat{\theta})\) between the larger and smaller models is:

The difference in the AIC penalty is \(2q\), and the difference in the BIC penalty is \(q\ln(n)\). Adding each penalty difference to the shared fit difference \(-G\) gives the displayed formulas. This also explains why one must calculate from the original RSS values rather than round \(G\) early: a small rounding change can alter a reported criterion difference near zero.

What Each Ranking Supports

AIC’s predictive motivation makes it useful when the aim is to choose among candidate models for prediction, subject to the model assumptions and the suitability of the candidate set. In broad terms, its penalty aims to account for the optimism that can arise when a model is evaluated on the data used to fit it. The ranking does not guarantee that predictions will be accurate in a new setting; that depends on the data and on whether the fitted relationship generalizes.

BIC’s penalty grows with sample size. Under regularity conditions, it approximates a comparison of Bayesian model evidence, and it can favor a simpler model more strongly as the sample size increases. In a fixed candidate set with suitable regularity conditions and a true model included, BIC has a model-selection consistency property: as sample size grows, it selects the true candidate with probability approaching one. These conditions matter. A low BIC does not establish that a model is literally true, and the ranking can be misleading if the candidate set or likelihood assumptions are unsuitable.

Both criteria are relative to the models being compared. A lower value says that a model ranks better than its competitors under that criterion; it does not show that the model is adequate in an absolute sense. Residual patterns, scientific context, and the purpose of the analysis remain important.

Common Mistake: Treating the Rankings as a Hypothesis Test

Incorrect reasoning: “AIC prefers the larger regression, so its added coefficient is statistically significant and the larger model is true.”

Correction: AIC and BIC are model-ranking criteria, not coefficient tests. Their differences do not provide a p-value, test a null hypothesis, or quantify the probability that a model is true. They support a preference among the candidate models for their respective aims and penalties. State the criterion, the compared candidates, and which has the smaller value; do not turn that ranking into a causal claim or a significance conclusion.