When AIC And BIC Disagree

Why Two Familiar Criteria Can Choose Different Models

AIC and BIC both reward a model for fitting the observed data well and penalize it for using more parameters. Their rankings can nevertheless disagree because the penalties are designed around different goals. AIC is motivated by prediction: it aims to compare models by their expected loss when predicting new observations. BIC is motivated by model identification: it approximates a Bayesian comparison of model evidence and, under suitable conditions, favors the true model as the sample grows.

The disagreement is not a calculation error or a contradiction. It signals that the criteria value fit and complexity differently. To interpret it, compare the improvement in log-likelihood with the additional penalty, then ask which goal is relevant to the analysis.

Definitions and the Two Goals

Definition: For a fitted model with maximized log-likelihood $\ell(\hat{\theta})$, $k$ estimated free parameters, and $n$ independent sampling units, define $$ \operatorname{AIC}=-2\ell(\hat{\theta})+2k, \qquad \operatorname{BIC}=-2\ell(\hat{\theta})+k\log n. $$ Lower values are preferred. The likelihoods being compared must be calculated for the same response data on a comparable likelihood scale; constants common to every candidate model do not affect the ranking.

Here $k$ counts the model’s estimated parameters, including any estimated nuisance or dispersion parameters that differ across candidates. The value of $n$ is the number of independent sampling units represented by the likelihood, not automatically the number of rows in an arbitrary summary table. A model comparison must use a consistent definition of both quantities.

AIC’s $2k$ penalty comes from a correction for the optimism of the fitted log-likelihood: a model is evaluated on data used to fit it, so its apparent fit is usually better than its expected fit on new data. In regular, correctly specified likelihood models of fixed dimension, AIC is an asymptotic estimate of relative expected Kullback–Leibler predictive discrepancy, up to terms common across models. This is a predictive target, not a guarantee that AIC selects the model closest to a particular scientific truth.

BIC’s penalty grows with $n$. Under regularity conditions, a Laplace approximation to a model’s marginal likelihood gives a leading term of $-\tfrac{k}{2}\log n$ on the log scale, in addition to the maximized log-likelihood. Multiplying the negative log marginal likelihood by two yields the BIC form, apart from terms that are smaller or common across models. With a fixed finite candidate set, a true model in that set, identifiable regular models, and suitable priors with positive density near the true parameter, BIC can consistently identify the true model. Those conditions matter: BIC is not a universal measure of predictive accuracy, and irregular or misspecified models need not have this interpretation.

Thus, “BIC penalizes complexity more” is usually true for practical sample sizes, but is not itself the full explanation. AIC and BIC answer different questions, and the size of the likelihood gain determines whether those different priorities change the ranking.

The Exact Disagreement Threshold

Consider a simpler model $S$ and a richer model $R$, fitted to the same observations. Suppose the richer model has $q=k_R-k_S>0$ additional parameters, and write $\Delta\ell=\ell_R-\ell_S$ for its increase in maximized log-likelihood.

Theorem: The differences in criteria, richer minus simpler, are $$ \operatorname{AIC}_R-\operatorname{AIC}_S=-2\Delta\ell+2q, \qquad \operatorname{BIC}_R-\operatorname{BIC}_S=-2\Delta\ell+q\log n. $$ If $n>e^2$, AIC selects the richer model while BIC selects the simpler model exactly when $$ q<\Delta\ell<\frac{q}{2}\log n. $$ At either endpoint, the corresponding criterion ties the two models.

Proof. Subtract the simpler model’s AIC from the richer model’s AIC. The likelihood contribution is $-2(\ell_R-\ell_S)=-2\Delta\ell$, and the penalty difference is $2(k_R-k_S)=2q$. This gives the first expression. Subtracting BIC in the same way gives a penalty difference $q\log n$, hence the second expression. AIC prefers $R$ exactly when $-2\Delta\ell+2q<0$, or $\Delta\ell>q$. BIC prefers $S$ exactly when $-2\Delta\ell+q\log n>0$, or $\Delta\ell<q\log n/2$. When $n>e^2$, the latter upper bound exceeds $q$, so both inequalities hold precisely in the stated interval. $\square$

For the usual case $n>e^2$, a modest gain can clear AIC’s fixed per-parameter threshold but not BIC’s sample-size-dependent threshold. If $n<e^2$, the penalty ordering reverses: BIC’s penalty per added parameter is smaller than AIC’s. At $n=e^2$, the penalties coincide. These small-sample cases are a reminder to calculate rather than rely on a slogan.

Worked Example: A Binary GLM Where AIC and BIC Disagree

There are 100 independent binary outcomes, split into two groups of 50 by a binary predictor $x$. The simpler logistic GLM has $\operatorname{logit}(p_i)=\beta_0$; the richer model has $\operatorname{logit}(p_i)=\beta_0+\beta_1x_i$. Suppose the groups have 8 and 17 events, respectively. The simpler model estimates the common event probability as $25/100=0.25$; the richer model estimates probabilities $8/50=0.16$ and $17/50=0.34$. The models have $k_S=1$ and $k_R=2$.

Ignoring binomial coefficients, which are identical across these models, the maximized log-likelihoods are

\begin{aligned} \ell_S &=25\log(0.25)+75\log(0.75) \approx -56.2335,\\ \ell_R &=8\log(0.16)+42\log(0.84)+17\log(0.34)+33\log(0.66) \approx -54.0353. \end{aligned}

Thus $\Delta\ell\approx2.1982$. There is one added parameter, so $q=1$. The richer-minus-simpler AIC difference is $-2(2.1982)+2\approx-2.3964$, and AIC selects the richer model. The BIC difference is $-2(2.1982)+\log(100)\approx0.2088$, and BIC selects the simpler model. The disagreement threshold is $1<\Delta\ell<\log(100)/2$, or approximately $1<2.1982<2.3026$, which confirms both rankings.

What the Growing BIC Penalty Does

The asymptotic contrast is especially clear when the simpler model is true and the richer model adds parameters that have true values zero. For regular nested models, with the true parameter in the interior and the usual identifiability and nonsingular-information conditions, Wilks’ theorem states that the likelihood-ratio statistic $T_n=2(\ell_R-\ell_S)$ converges in distribution to a chi-squared random variable with $q$ degrees of freedom.

Result: Under these conditions, the probability that AIC selects the redundant richer model does not tend to zero, whereas the probability that BIC selects it does tend to zero.

Derivation. AIC selects $R$ when $T_n>2q$. Since $T_n\xrightarrow{d}\chi^2_q$, and the chi-squared distribution is continuous, $P(T_n>2q)$ converges to $P(\chi^2_q>2q)$, which is strictly positive. In contrast, BIC selects $R$ only when $T_n>q\log n$. Convergence in distribution implies that the sequence $T_n$ is bounded in probability, while $q\log n$ tends to infinity. Therefore $P(T_n>q\log n)\to0$. This result describes selection of redundant terms in this nested setting; it is not, by itself, a proof of BIC’s overall consistency across every possible candidate set. $\square$

This difference helps explain why AIC is not generally consistent for identifying a finite true model: even when an added term is truly unnecessary, a fixed penalty can leave a nonzero asymptotic chance of retaining it. BIC’s increasing penalty suppresses that chance under the stated assumptions. Whether that is desirable depends on the purpose. If small predictive improvements are valuable, rejecting a term merely because the sample is large may not be the goal. If identifying a parsimonious true model is the goal and the assumptions are credible, BIC’s behavior may be attractive.

Worked Example: A Smaller Gain Does Not Always Produce Disagreement

Take the same kind of comparison, now with $n=20$, one extra parameter, and a fitted log-likelihood gain $\Delta\ell=1.2658$. For a concrete binomial GLM, suppose two groups each contain 10 observations, with 1 event in the first and 4 in the second. Compare an intercept-only logistic model, with fitted probability $5/20=0.25$, to an intercept-plus-group model, with fitted probabilities $0.1$ and $0.4$.

Ignoring the common binomial coefficients, the log-likelihoods are

\begin{aligned} \ell_S&=5\log(0.25)+15\log(0.75)\approx-11.2467,\\ \ell_R&=\log(0.1)+9\log(0.9)+4\log(0.4)+6\log(0.6)\approx-9.9809. \end{aligned}

The gain is about $1.2658$. With $q=1$, the AIC difference is $-2(1.2658)+2\approx-0.5316$, so AIC selects the richer model. The BIC difference is $-2(1.2658)+\log(20)\approx0.4642$, so BIC selects the simpler model. Here the disagreement interval is $1<\Delta\ell<\log(20)/2$, or approximately $1<1.2658<1.4979$.

Worked Example: A Strong Fit Gain Can Make the Criteria Agree

Suppose independent counts follow a Poisson GLM with a log link and an exposure offset. The simpler model includes an intercept and one covariate; the richer model adds two prespecified covariates, so $q=2$. For a fit with $n=1000$, suppose the maximized log-likelihoods are $-410$ and $-390$, respectively, giving $\Delta\ell=20$. These are the fitted likelihood values for the comparison; only their difference is needed for the calculation.

The AIC difference is $-2(20)+2(2)=-36$, favoring the richer model. The BIC difference is $-2(20)+2\log(1000)\approx-40+13.8155=-26.1845$, also favoring the richer model. The gain exceeds both thresholds: AIC requires $\Delta\ell>2$, while BIC requires $\Delta\ell>\log(1000)\approx6.9078$. Different penalties do not imply different rankings in every comparison.

Diagnosing a Disagreement and Avoiding Common Mistakes

When the criteria disagree, first verify that the models use the same observations, response, and comparable likelihood. Confirm the number of estimated parameters in each fit and use the appropriate independent-unit count for $n$. Then calculate $\Delta\ell$, $q$, and the two criterion differences directly. This distinguishes a genuine tradeoff from a mismatch in likelihoods, data, or parameter counting.

A common mistake is to conclude that BIC must always choose the simpler model because its penalty is larger. The correction is to compare the fit gain with both thresholds: a sufficiently large $\Delta\ell$ can make both criteria choose the richer model. Another mistake is to treat AIC and BIC as interchangeable estimates of the same objective. They are not: AIC is oriented toward predictive discrepancy, while BIC’s usual interpretation is tied to marginal likelihood and model identification under assumptions.

Neither criterion settles the scientific question on its own. AIC’s predictive motivation does not guarantee good predictions under severe misspecification, and BIC’s identification motivation does not ensure that the true model is present in the candidate set. Irregular fits, boundary estimates, separation in logistic regression, or non-identifiable parameters can also undermine the regular approximations behind their interpretations. The relevant choice depends on the intended goal and whether the required modeling assumptions are plausible.

Key takeaway: For a richer model with $q$ additional parameters, AIC compares the likelihood gain with $q$, while BIC compares it with $q\log(n)/2$. Their disagreement is a predictable consequence of different targets, not evidence that one calculation must be wrong.

Check Your Understanding

Use the criteria and the stated assumptions to answer these questions.

A richer GLM adds three parameters to a simpler model. Write the AIC and BIC differences, richer minus simpler, in terms of $\Delta\ell$, $n$, and $q$.
For $n=200$ and $q=1$, give the interval of log-likelihood gains for which AIC selects the richer model and BIC selects the simpler model.
Why can BIC’s tendency to reject redundant terms as $n$ grows be useful for model identification but not automatically ideal for prediction?
In the nested true-model setting, what does Wilks’ theorem imply about the limiting probability that AIC retains $q$ redundant parameters?
Name two checks to make before interpreting an AIC–BIC disagreement between fitted GLMs.

What You'll Learn