Why AIC and BIC Choose Different NLME Models

Why Two Information Criteria Can Disagree

AIC and BIC both balance fit against complexity, but they impose different costs for adding parameters. In a nonlinear mixed-effects (NLME) analysis, this difference can matter when a more complex model improves the likelihood: the improvement may be large enough to offset AIC's penalty but not BIC's. The resulting disagreement is not a calculation error. It reflects the criteria's different penalty rules and aims.

AIC is commonly used to compare models with an emphasis on expected predictive information loss. BIC applies a penalty that grows with the number of independent sampling units and, under additional assumptions, is connected to approximating Bayesian model evidence. Neither criterion is a hypothesis test, and neither by itself establishes that a selected model is scientifically correct.

Definition: For a model with maximized log likelihood $\ell$, $K$ estimated parameters, and $n$ independent sampling units, the Akaike information criterion and Bayesian information criterion are $$ \mathrm{AIC}=-2\ell+2K,\qquad \mathrm{BIC}=-2\ell+K\log(n). $$ When comparing candidates fitted to the same data, the criterion with the smaller value ranks better under that criterion. For the BIC convention used here, $n$ is the number of independent subjects when the likelihood is built from independent subject-level contributions, not the total number of concentration records.

The parameter count $K$ includes all estimated parameters in the likelihood being compared, such as fixed effects, random-effects distribution parameters, and residual-error parameters. The count must follow the same convention for every candidate. The likelihoods must also be comparable: candidates should be fitted to the same response data and observations using the same likelihood basis. In particular, comparisons involving different fixed-effects structures should use likelihoods that are comparable for that purpose, typically maximum likelihood rather than likelihoods computed under different restricted-likelihood specifications.

The Penalty Difference in a Two-Model Comparison

Suppose a larger candidate adds $q$ estimated parameters to a smaller candidate. Let $G$ be the increase in twice the maximized log likelihood: $G=2(\ell_{\mathrm{large}}-\ell_{\mathrm{small}})$. A positive $G$ means the larger model fits better by likelihood. Subtracting the smaller model's criterion from the larger model's criterion gives the exact comparison.

\begin{aligned} \mathrm{AIC}_{\mathrm{large}}-\mathrm{AIC}_{\mathrm{small}}&=-G+2q,\\ \mathrm{BIC}_{\mathrm{large}}-\mathrm{BIC}_{\mathrm{small}}&=-G+q\log(n). \end{aligned}

A negative difference favors the larger model; a positive difference favors the smaller one. Thus AIC favors the larger model when $G>2q$, while BIC favors it when $G>q\log(n)$. When $n>e^2$, or about 7.389 independent subjects, $\log(n)>2$. In that case BIC charges more for each added parameter, so a disagreement is possible when $2q<G<q\log(n)$. At $G=2q$ or $G=q\log(n)$, the relevant criterion ties the models. If $n<e^2$, the BIC penalty per added parameter is smaller than AIC's, and the direction of a possible disagreement reverses.

This result follows directly from the definitions; it does not require a special NLME theorem. It isolates the source of disagreement: both criteria reward the same likelihood improvement, but they subtract different amounts for the extra parameters. The sample size changes BIC's penalty, whereas AIC's per-parameter penalty remains 2.

Worked Example: AIC Prefers the Larger Model but BIC Does Not

Consider two hypothetical NLME models fitted to the same concentration data from $n=120$ independent subjects. The larger model adds two estimated parameters. Assume both fits converged, used the same likelihood definition, and have no unresolved fitting warnings. The smaller model has $\ell=-820.0$ and $K=6$; the larger model has $\ell=-816.5$ and $K=8$.

The larger model's log likelihood is higher by $3.5$, so $G=2(3.5)=7.0$. The likelihood improvement is real as a numerical feature of these fitted candidates, but it must be compared with each criterion's complexity penalty.

For AIC, the smaller model gives $-2(-820.0)+2(6)=1640+12=1652$. The larger model gives $-2(-816.5)+2(8)=1633+16=1649$. The larger model's AIC is lower by $3.0$, so AIC ranks it ahead.

For BIC, $\log(120)\approx4.7875$. The smaller model gives $1640+6(4.7875)=1668.7$. The larger gives $1633+8(4.7875)=1671.3$. BIC ranks the smaller model ahead by about $2.575$ criterion units.

The algebra explains the result without relying only on rounded totals. Here $q=2$, so the AIC penalty increase is $2q=4$, while the BIC penalty increase is $q\log(120)\approx9.575$. The gain $G=7.0$ exceeds 4 but is less than 9.575. It therefore clears AIC's complexity cost but not BIC's.

Worked Comparisons at Different Sample Sizes and Fit Gains

The disagreement is not determined by the number of parameters alone. It depends jointly on the gain in fit, the number of added parameters, and—under BIC—the number of independent subjects. The next comparisons make those dependencies explicit.

Worked Example: One Added Parameter With a Smaller Subject Sample

Suppose two hypothetical models are fitted to the same data from $n=40$ independent subjects. The smaller model has $\ell=-500.0$ and $K=7$. The larger model adds one parameter, has $K=8$, and has $\ell=-498.6$. Assume both likelihoods are maximized comparably and both fits are stable.

The likelihood gain is $G=2[-498.6-(-500.0)]=2.8$. For AIC, the smaller model's value is $1000+14=1014$, and the larger model's value is $997.2+16=1013.2$. AIC favors the larger model by $0.8$.

For BIC, $\log(40)\approx3.6889$. The smaller model's value is $1000+7(3.6889)=1025.8$; the larger model's value is $997.2+8(3.6889)=1026.7$. BIC favors the smaller model by about $0.889$.

In threshold form, the added parameter requires a gain above 2 for AIC and above 3.6889 for BIC. Since $2<2.8<3.6889$, the rankings differ. The calculation uses 40 subjects, not a larger count obtained by adding the repeated records within each subject.

Worked Example: Both Criteria Favor the More Complex Model

Now compare hypothetical candidates fitted to the same data from $n=300$ independent subjects. The smaller model has $\ell=-1000.0$ and $K=5$. A larger model adds two parameters, giving $\ell=-994.0$ and $K=7$. Assume the estimates are stable and the likelihoods are comparable.

The likelihood gain is $G=2[-994.0-(-1000.0)]=12.0$. AIC for the smaller model is $2000+10=2010$; for the larger it is $1988+14=2002$. AIC favors the larger model by 8.0.

For BIC, $\log(300)\approx5.7038$. The smaller model's value is $2000+5(5.7038)=2028.5$. The larger model's value is $1988+7(5.7038)=2027.9$. BIC also favors the larger model, but only by about $0.592$.

The gain of 12.0 exceeds both penalties for two added parameters: $2q=4$ and $q\log(300)\approx11.408$. BIC's preference is comparatively slight, so it would be unwise to treat the ranking as a large or decisive practical advantage.

Conditions, Diagnostics, and Interpretation

A numerical ranking is informative only if the model comparison itself is sound. Before interpreting AIC or BIC, check the following conditions:

Same data and response scale. The candidates should use the same observations and response representation. A change in the data set can change $-2\ell$ for reasons unrelated to model quality.
Comparable likelihoods. Confirm that the values come from the same likelihood basis and estimation convention. A reported objective function should not be assumed to be a comparable maximized log likelihood unless its definition is known.
Converged, credible fits. Check optimizer status, gradient or convergence diagnostics, parameter estimates, and whether repeated starts produce consistent solutions. A local or failed fit can give a misleading likelihood ranking.
Consistent parameter counts. Count all estimated parameters included in the likelihood, using the same rule across models. Omitting a variance or residual parameter from one candidate distorts the penalty.
Correct BIC sample size. Use the number of independent sampling units appropriate to the likelihood factorization. In a subject-level NLME analysis, repeated observations from one subject are not automatically independent subjects.

An unusual estimate, a variance pushed to a boundary, or a parameter that is weakly identified can signal that a model's nominal complexity does not correspond to stable information in the data. Information criteria still have numerical definitions, but the ranking should be reported alongside fitting and model adequacy diagnostics. Where the likelihood is irregular, standard large-sample interpretations may be less reliable; the arithmetic alone does not resolve that concern.

AIC and BIC are relative criteria: their values do not measure absolute fit, and the difference is not a probability that one model is true. A lower AIC does not prove better performance for every prediction target. A lower BIC does not prove that the model is the data-generating model. Both rankings also depend on the set of candidates supplied for comparison.

How the Criteria Relate to Other Comparisons

For nested models, a likelihood-ratio test uses the same likelihood gain $G$, but compares it with a reference distribution under its assumptions and a chosen significance level. AIC and BIC instead compare $G$ with fixed complexity penalties. A test may not reject the smaller model while AIC favors the larger one; that is not inherently contradictory, because the methods answer different questions and use different decision rules. Boundary parameters and other irregularities can also affect likelihood-ratio test calibration.

For non-nested candidates, AIC and BIC can still rank models when the likelihoods and data are comparable. This does not turn their comparison into a formal test of one model against another. Predictive validation provides a different perspective by evaluating predictions on data not used for fitting; its result depends on the prediction target and validation design. These approaches are complementary rather than interchangeable.

Common Pitfall: Treating BIC's Larger Penalty as a Universal Verdict

A common mistake is: “BIC penalizes complexity more, so BIC is always the correct choice and the extra parameter must be unnecessary.” The first clause is conditional on $n>e^2$, and the conclusion does not follow. AIC and BIC encode different goals and assumptions; a ranking under either one is not a direct verdict on scientific necessity. A better practice is to state which criterion ranks which candidate, show the likelihood gain and penalty difference, then assess whether the model's scientific purpose, diagnostics, and prediction target support the choice.

Key takeaway: For $q$ added parameters, AIC favors the larger model when $G>2q$, and BIC favors it when $G>q\log(n)$. When the subject-level sample size makes BIC's penalty larger, a gain between those thresholds produces a predictable disagreement.

Check Your Understanding

For each problem, compare the criteria using the stated hypothetical values. Solutions are included so the threshold logic can be checked directly.

A larger NLME model adds two parameters, and the data contain 100 independent subjects. What interval of likelihood gains $G$ would make AIC favor the larger model and BIC favor the smaller one?
With $n=25$, one parameter is added and $G=3.2$. Which model does each criterion rank better?
For $n=200$, a model adds three parameters and has $G=18$. Do the criteria agree?
Why is it generally incorrect to substitute the total number of repeated concentration records for $n$ in a subject-level BIC calculation?

Solutions. (1) Here $q=2$, and $\log(100)\approx4.6052$. AIC favors the larger model for $G>4$, while BIC favors the smaller one for $G<2(4.6052)\approx9.210$. Thus disagreement occurs for $4<G<9.210$; equality at either endpoint gives a tie under that criterion. (2) The thresholds are $2q=2$ and $q\log(25)=\log(25)\approx3.219$. Since $2<3.2<3.219$, AIC favors the larger model and BIC the smaller. (3) The AIC threshold is $2q=6$, and the BIC threshold is $3\log(200)\approx15.895$. Since $18$ exceeds both, both criteria favor the larger model. (4) The repeated records within a subject contribute to that subject's joint likelihood; they do not, solely by being multiple records, create additional independent subjects. Use the independent sampling units appropriate to the likelihood construction.