AIC, BIC, And Non-Nested Likelihood Comparisons

Why Compare Models Without a Nested Test?

Nonlinear likelihood analysis often produces several plausible specifications for the same outcome. The models may use different functional forms or parameterizations, and neither may be a restricted version of the other. The maximized likelihood alone cannot settle the comparison: a more flexible model can usually fit the observed data at least as well, simply because it has more parameters.

Akaike’s information criterion (AIC) and the Bayesian information criterion (BIC) balance fit against model complexity. They provide a way to rank candidate models fitted to the same data, including non-nested models. Their rankings are useful, but they are not formal tests of whether two models are equal, and they do not establish that the top-ranked model is true.

Here, \(\hat{\theta}\) is the maximum likelihood estimate under the particular model. Count all estimated free parameters in \(k\), including nuisance parameters such as an estimated variance or dispersion parameter. Do not count quantities fixed in advance. The \(n\) in the usual BIC formula is the number of independent observational units contributing to the likelihood, not automatically the number of rows if the likelihood’s sampling structure says otherwise.

Calculating and Reading the Criteria

Both criteria start with \(-2\ell(\hat{\theta})\), a measure that decreases as the maximized likelihood improves. They add a penalty for model complexity. AIC adds \(2\) for each estimated parameter. BIC adds \(\log n\) for each parameter, so its per-parameter penalty exceeds AIC’s when \(n>e^2\), approximately 7.39. In most applications with more observations, BIC penalizes additional parameters more heavily.

The values are relative to the candidate set: an AIC of 100 has no universal interpretation as “good” or “bad.” The difference between criteria for two models is often more informative than either value alone. Comparisons require the same observed data and a consistently defined likelihood. In particular, likelihoods that omit different constants cannot safely be compared by their reported AIC or BIC unless the likelihood values are first put on a common scale.

What the Penalties Represent

AIC’s penalty is motivated by prediction. Evaluating a fitted model on the same observations used to estimate it tends to make its fit look better than its fit on new observations. AIC corrects for this optimism, approximately estimating expected out-of-sample deviance for a future sample of the same size. The correction relies on regularity conditions and, in its usual form, correct specification of the likelihood. If the likelihood is misspecified, a different penalty may be needed.

The scale of the prediction calculation matters. Let the training sample contain \(n\) independent observations, and suppose an independent evaluation sample from the same correctly specified model contains \(m\) observations. Write \(I\) for the per-observation Fisher information at the true parameter \(\theta_0\), and suppose there are \(k\) regular, identifiable parameters. The MLE has approximate covariance \(I^{-1}/n\). A quadratic approximation to the evaluation log likelihood gives $$ E\!\left[\ell_{\mathrm{eval}}(\hat{\theta})-\ell_{\mathrm{eval}}(\theta_0)\right] \approx -\frac{m}{2}\operatorname{tr}\!\left(I\,\frac{I^{-1}}{n}\right) =-\frac{km}{2n}. $$ This is the expected evaluation log-likelihood loss from estimating \(\theta\) rather than using the true \(\theta_0\). It is about \(k/2\) only when \(m=n\). In the training sample, fitting the \(k\) parameters produces an approximate expected log-likelihood gain of \(k/2\) over using \(\theta_0\). Thus, for evaluation on a same-sized independent sample, the expected gap between fitted training log likelihood and evaluation log likelihood is approximately \(k\). On the deviance scale, which is \(-2\) times log likelihood, that gap is \(2k\), the AIC penalty.

BIC has a different motivation. Under regularity conditions, a fixed-dimensional model, and suitable prior distributions that are positive near the maximum, a large-sample approximation to the model’s integrated likelihood gives $$ -2\log(\text{integrated likelihood})\approx -2\ell(\hat{\theta})+k\log n+\text{terms not growing with }n. $$ This Laplace approximation explains BIC’s penalty. In settings where the candidate models and priors make a Bayesian model comparison meaningful, BIC differences can approximate differences in model evidence. The approximation does not make BIC a universal probability that a model is true; its accuracy depends on the assumptions behind the approximation.

Non-Nested Comparisons Are Not Likelihood-Ratio Tests

For nested models, a likelihood-ratio statistic can sometimes be compared with a chi-square reference distribution under appropriate regularity conditions. A non-nested pair has no such general reference distribution for the difference in maximized log likelihood. In particular, the fact that one model has a higher maximized likelihood does not make a likelihood-ratio test valid. AIC and BIC instead apply their own fit-complexity criteria; they do not turn the likelihood difference into a formal test of model equality.

The distinction is about the question being answered. An information criterion ranks candidates according to a stated fit-penalty rule. A formal test assesses a specified null hypothesis using a sampling distribution under assumptions. If the question is whether a restriction holds, use an appropriate testing method only when its hypotheses and reference distribution are justified. Do not interpret a lower AIC or BIC as a p-value or as evidence that the other model is statistically rejected.

Practical Checks and Common Pitfalls

Before reporting an information-criterion comparison, verify that the candidate models were fit to the same observations and that their likelihoods refer to the same response and likelihood scale. Check that \(k\) includes all estimated parameters. Also confirm that the numerical optimization has converged to a credible maximum: a criterion calculated from a failed or inferior fit can produce a misleading ranking.

A frequent mistake is to conclude that the model with the smallest AIC or BIC has been proven correct. That reasoning treats a relative ranking as a formal test. The correction is to state only that the model is preferred by that criterion among the specified candidates, and to explain whether the comparison is aimed at predictive performance (AIC) or uses BIC’s large-sample model-evidence approximation. Neither criterion protects against a poor candidate set, misspecification, or unreliable estimation.

Another mistake is to compare reported criterion values from different samples or from likelihoods that omit different constants. Such differences need not reflect model quality. Establish a common data set and likelihood convention first; otherwise, the numerical ranking may have no valid interpretation.