Why a Lower Information Criterion May Not Mean a Better Model
AIC and BIC are useful for comparing fitted models, but their numerical values are meaningful comparatively only when the fits describe the same observed response under compatible likelihood conventions. A model can have a smaller reported criterion simply because it was fit to different records or because its likelihood was expressed on a different scale. In those cases, selecting the smaller number is not a sound comparison.
This tutorial focuses on recognizing those invalid comparisons. It assumes the usual definitions of AIC and BIC for models fitted by maximum likelihood. The critical question is not just, “Which number is smaller?” It is, “Were these numbers calculated from comparable likelihoods for the same data?”
The penalty terms address model complexity; they do not repair an incompatible comparison. AIC and BIC cannot make two different response datasets equivalent, turn a quasi-likelihood into a full likelihood, or remove a data-dependent difference in likelihood conventions. The word “same” here concerns the data being modeled, not merely the number of rows: the observed outcomes, observational units, and likelihood target must be aligned.
What Must Match for a Valid Comparison?
Before interpreting a difference in AIC or BIC, check three aspects of the fits. First, were the models evaluated on the same observed responses? Second, do the reported likelihoods use the same probability model and compatible conventions? Third, are the likelihood values genuine maximized log-likelihoods, rather than deviance or quasi-likelihood quantities treated as if they were log-likelihoods?
| Check | Why it matters | Typical warning sign |
|---|---|---|
| Response observations | The likelihood must describe the same outcomes for the criteria to compare model fit on a common basis. | Different complete-case records because predictors have different missing values. |
| Likelihood definition | Likelihood values must be on a compatible scale and based on the same response representation. | Comparing grouped-binomial and individual-Bernoulli log-likelihoods without accounting for their constant difference. |
| Likelihood existence and interpretation | Ordinary AIC and BIC use a maximized full likelihood. | Using a quasi-Poisson fit or substituting residual deviance for \(-2\ell\). |
| BIC sample size | The penalty uses the number of independent observational units represented by the likelihood. | Using different or inconsistent definitions of \(n\) across candidate fits. |
A model fit after dropping records with missing predictor values is not automatically comparable with a model fitted to all records, even if both use the same response variable and family. AIC and BIC then describe fit to different observed outcomes. A fair information-criterion comparison generally requires refitting the candidates on the same eligible records, provided that doing so still answers the intended modeling question.
Likelihood conventions also matter. For a common data set, adding the same constant to every candidate log-likelihood does not change their AIC ranking. But if one reported likelihood contains a data-dependent constant and another does not, comparing their raw criteria mixes model fit with the convention used to calculate the likelihood. A common response in different units can also require a density transformation and its Jacobian; without expressing both likelihoods for the same response scale, their raw values need not be comparable.
Why a Common Constant Cancels
The cancellation property explains why likelihood conventions matter. Suppose two candidate models use the same observations, and a reporting convention adds the same constant \(C\) to each model’s maximized log-likelihood. Let the models have log-likelihoods \(\ell_1,\ell_2\) and parameter counts \(k_1,k_2\). The AIC difference is:
The constant disappears, so the AIC difference is unchanged. The same cancellation occurs for BIC if both models use the same \(n\): its difference is \(-2(\ell_1-\ell_2)+(k_1-k_2)\log(n)\). But if a constant is included for one likelihood representation and not the other, or if the likelihoods concern different response data, this cancellation argument does not apply. A numerical difference then need not measure the relative support of the candidate models.
Worked Examples
Worked Example: Different Complete-Case Samples
A hypothetical insurer fits two Poisson GLMs for the number of roadside-assistance calls per policy. Model A uses vehicle age; Model B adds a regional indicator. Model A retains 240 policies, while Model B retains 218 because the regional indicator is missing for 22 policies. The reported maximized log-likelihoods and parameter counts are \(\ell_A=-98.2,\ k_A=4\) and \(\ell_B=-96.0,\ k_B=5\). Calculate the reported AIC values and decide whether they support choosing Model B.
Method. Calculate each reported AIC from \(-2\ell+2k\), then check whether the models were fit to the same outcomes. This avoids the common mistake of interpreting a lower AIC without checking the records used.
For Model A, \(-2\ell_A=-2(-98.2)=196.4\), and \(2k_A=2(4)=8\). Therefore, \(\mathrm{AIC}_A=196.4+8=204.4\).
For Model B, \(-2\ell_B=-2(-96.0)=192.0\), and \(2k_B=2(5)=10\). Therefore, \(\mathrm{AIC}_B=192.0+10=202.0\). The reported value for Model B is lower by \(204.4-202.0=2.4\).
Conclusion. These values do not provide a valid AIC comparison: Model A’s likelihood describes 240 policies, whereas Model B’s describes 218. The difference of 2.4 combines model fit with the change in response records. To compare the candidates by AIC, refit both on the same 218 policies and recalculate both likelihoods and AIC values. The resulting comparison would apply to those 218 policies, not automatically to the original 240.
Worked Example: Grouped and Individual Binomial Likelihoods
A hypothetical study records outcomes for 10 independent groups, each containing two trials. In every group, one trial succeeds and one fails. One fit reports a grouped-binomial log-likelihood of \(-40.000\) with \(k=3\). An equivalent fit represents the 20 trial outcomes individually as Bernoulli observations. Calculate the individual-level log-likelihood and both AIC values. Explain why the raw AIC values should not be compared across these representations.
Method. The grouped-binomial likelihood includes the binomial coefficient \(\binom{2}{1}=2\) for each group. The individual Bernoulli likelihood does not include this count of possible within-group arrangements. The grouped log-likelihood therefore exceeds the individual-level log-likelihood by \(10\log(2)\), a constant determined by these observed outcomes.
There are 10 groups, each with \(\log\binom{2}{1}=\log(2)\), so the total difference is \(10\log(2)=10(0.693147)=6.93147\). Thus the individual-level log-likelihood is \(-40.000-6.93147=-46.93147\).
For the grouped fit, \(\mathrm{AIC}_{\mathrm{grouped}}=-2(-40.000)+2(3)=80+6=86.000\). For the individual fit, \(\mathrm{AIC}_{\mathrm{individual}}=-2(-46.93147)+6=93.86294+6=99.86294\). Rounding only at the final step, the individual-level AIC is 99.863 to three decimal places.
Conclusion. The difference is \(99.86294-86.000=13.86294=2(6.93147)\), exactly twice the likelihood constant. The discrepancy comes from the data representation, not a difference in model fit. The two values should not be compared as though they were calculated under one likelihood convention. Within either representation, candidate models fitted consistently to the same outcomes can still be ranked by their AIC differences, because the common constant cancels. This solution avoids the trap of treating different absolute AIC values as evidence that one representation fits better.
Worked Example: A Quasi-Poisson Fit Is Not an Ordinary AIC Candidate
A hypothetical portfolio analyst fits a quasi-Poisson GLM and reports residual deviance 120, estimated dispersion 1.5, and \(k=4\) estimated regression parameters. A colleague calculates \(120+2(4)=128\) and calls it the model’s AIC, then compares it with a Poisson fit whose maximized log-likelihood is \(-61.2\) and whose parameter count is 4. Assess the comparison.
Method. Ordinary AIC requires a maximized full log-likelihood. A quasi-likelihood specifies a mean-variance relationship but does not, by itself, specify a normalized probability distribution and full likelihood. Also, residual deviance is not generally equal to \(-2\ell\).
The Poisson fit has an ordinary AIC of \(-2(-61.2)+2(4)=122.4+8=130.4\). The colleague’s calculation for the quasi-Poisson fit is \(120+8=128\), but this substitutes residual deviance for \(-2\ell\). In general, deviance is defined relative to a saturated model; it is a likelihood-ratio quantity, not the model’s negative twice log-likelihood. More importantly, the quasi-Poisson fit does not supply the full likelihood required by ordinary AIC.
Conclusion. The value 128 is not an ordinary AIC, and it cannot validly be compared with 130.4. The difference in dispersion does not turn the quasi-likelihood into a full likelihood. Specialized quasi-likelihood criteria require their own stated definitions and compatible conventions; they should not be invented by adding a penalty to residual deviance. This avoids the trap of treating the familiar AIC formula as a license to use any fit statistic in place of log-likelihood.
A Practical Validity Check
When a software output presents AIC or BIC values side by side, the comparison still needs a modeling check. A useful sequence is:
Verify that the same outcomes and observational units contributed to every fit. Check missing-data exclusions, filtering, and any changes to the analysis population.
Check that the likelihoods refer to compatible outcomes and scales. If data are grouped or transformed, establish whether constants or Jacobian terms differ.
Ordinary AIC and BIC require maximized likelihoods. Do not substitute residual deviance, a quasi-likelihood value, or another fit statistic without a separately justified criterion.
Use a consistent number of independent observational units for \(n\), and count parameters according to the likelihood-based model being compared.
If a check fails, do not rank the models by the reported criteria. Align the fits or choose a comparison objective appropriate to the question.
A practical complication is that software may silently fit each model to the records available for its own predictors. Equal reported sample sizes do not guarantee identical observations, so check the actual records or analysis subsets. Likewise, two likelihoods may describe the same outcomes while differing by a constant because one is grouped and one is expanded. If that constant is common across all models within each representation, rankings within a representation can remain valid, but cross-representation AIC or BIC levels are not directly comparable.
For BIC, the observation count must correspond to the independent units represented by the likelihood. For example, a grouped-binomial likelihood may represent groups with trial totals, while an expanded Bernoulli likelihood represents individual trials. The appropriate \(n\) depends on the sampling and likelihood formulation; it should not be switched casually to create a preferred penalty. If candidates use different definitions of \(n\), the BIC penalties may no longer represent a like-for-like comparison.
These checks matter because information criteria are summaries, not universal scores. A lower number is evidence in a comparison only when the candidate likelihoods target the same observed data under compatible definitions. If those requirements are not met, report that the criteria are not comparable rather than presenting a misleading ranking. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, decide whether the comparison is valid and calculate any requested criteria.
- Two Poisson GLMs use the same 80 policy records. Model A has \(\ell=-52.0,\ k=3\); Model B has \(\ell=-50.5,\ k=4\). Calculate both AIC and BIC values. Use \(\log(80)=4.3820\). Which model has the lower AIC, and which has the lower BIC? (Answer: AIC values are 110.0 for A and 109.0 for B, so B has lower AIC. BIC values are 117.146 for A and 118.528 for B, so A has lower BIC.)
- A logistic model using age is fitted to 500 records. A second model adds a health indicator and is fitted to 463 records because of missing values. Its reported AIC is 3 units lower. Is that enough to select the second model? (Answer: No. The response records differ, so the AIC values do not provide a valid like-for-like comparison; refit both models on the same records.)
- For 8 groups with two trials each and one success per group, a grouped-binomial log-likelihood is \(-30.000\), with \(k=2\). The individual Bernoulli representation uses the same outcomes and parameter count. Calculate the individual log-likelihood and both AIC values. Use \(\log(2)=0.693147\). (Answer: The individual log-likelihood is \(-30-8\log(2)=-35.5452\); grouped AIC is \(60+4=64.000\), and individual AIC is \(71.0904+4=75.0904\). The raw values are not comparable across representations.)
- A quasi-binomial GLM has residual deviance 74 and \(k=5\). An ordinary binomial GLM has maximized log-likelihood \(-42\) and \(k=5\). A candidate claims the quasi-binomial AIC is \(74+10=84\). Identify the error and state whether 84 can be compared with the ordinary binomial AIC. (Answer: The calculation substitutes residual deviance for negative twice the maximized log-likelihood, and the quasi-binomial fit does not provide an ordinary full likelihood. The binomial AIC is \(-2(-42)+10=94\); 84 is not an ordinary AIC and cannot be compared with 94.)