Why Information-Criterion Comparisons Go Wrong
AIC and BIC combine goodness of fit with a penalty for model complexity. Their formulas are short, but a comparison can still go wrong if the parameter count is inconsistent, the criterion direction is reversed, or candidate models use different observations. A numerically smaller value is meaningful only when the values are calculated on a compatible basis.
This tutorial uses Gaussian linear regression to make those checks explicit. The same careful habits apply whenever information criteria are used: identify the likelihood, count estimated parameters, confirm the data being compared, and interpret the result as a ranking under stated conditions—not as proof that one model is certainly true.
For this model, the maximized Gaussian log-likelihood gives the following full criteria:
The \(p+1\) counts the \(p\) regression coefficients and the estimated variance. When every candidate uses the same response and the same \(n\) observations, the term \(n[\ln(2\pi)+1]\) is common to all candidates. It can be omitted without changing their ranking. The resulting reduced expressions are
These reduced expressions omit the variance penalty as well as the common likelihood term. This is valid for ranking models because the omitted variance penalty is the same for every candidate: it is \(2\) for AIC and \(\ln n\) for BIC. Thus, use either the full formulas for every model or the reduced formulas for every model. Do not compare a full value for one model with a reduced value for another.
Why the Gaussian Fit Term Uses RSS
The fit term is not an arbitrary substitution. Under the stated Gaussian model, the likelihood for a fixed regression fit is maximized over \(\sigma^2\) at \(\widehat{\sigma}^2_{\mathrm{MLE}}=\mathrm{RSS}/n\). Substituting this estimate into the log-likelihood explains the logarithm of the mean squared residual.
For residuals \(e_i=Y_i-\widehat{Y}_i\), the log-likelihood as a function of \(\sigma^2\) is \(-\frac{n}{2}\ln(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^{n}e_i^2\).
Since \(\sum e_i^2=\mathrm{RSS}\), differentiation with respect to \(\sigma^2\) gives \(-n/(2\sigma^2)+\mathrm{RSS}/[2(\sigma^2)^2]=0\). Solving yields \(\widehat{\sigma}^2_{\mathrm{MLE}}=\mathrm{RSS}/n\).
Then \(-2\widehat{\ell}=n[\ln(2\pi)+1+\ln(\mathrm{RSS}/n)]\). Adding the AIC or BIC penalty produces the full formulas above.
The derivation relies on the Gaussian likelihood and its maximum-likelihood variance estimate. If the response distribution, likelihood, or fitted data differ, these particular RSS formulas need not apply; use the appropriate maximized likelihood and parameter count instead.
Worked Comparisons and the Traps They Expose
Worked Example: Counting the Variance Parameter
Two Gaussian regression models use the same response and the same \(n=50\) observations. Model A has an intercept and one slope, with \(\mathrm{RSS}_A=50\). Model B has an intercept and two slopes, with \(\mathrm{RSS}_B=44\). Compare them using AIC and BIC. Report reduced values to six decimal places.
Method. Count regression coefficients first: \(p_A=2\) and \(p_B=3\). The full criteria count one additional estimated parameter, \(\sigma^2\); the reduced formulas count only the regression coefficients because the variance penalty is common.
For Model A, \(\mathrm{RSS}_A/n=50/50=1\), so \(\ln(1)=0\). Therefore,
For Model B, \(\mathrm{RSS}_B/n=44/50=0.88\), and \(\ln(0.88)=-0.1278333715\). The fit contribution is \(50\ln(0.88)=-6.391668575\). Thus,
Both reduced criteria are lower for Model B, so both prefer it among these two candidates. As a check, the full AIC values add \(2\) to each reduced value, and the full BIC values add \(\ln(50)=3.912023\) to each. Those shared additions cannot change the ranking.
Common mistake avoided: Counting only slopes, or counting the variance parameter for one model but not the other, changes the penalty inconsistently. State the counting convention and apply it to every candidate.
Worked Example: Lower Is Better, but the Criteria Can Disagree
Two models are fitted to the same \(n=60\) observations. Model C has \(p_C=2\) regression coefficients and \(\mathrm{RSS}_C=72\). Model D has \(p_D=1\) coefficient and \(\mathrm{RSS}_D=70\). Compare the reduced criteria, rounding to six decimal places.
For C, \(\mathrm{RSS}_C/n=1.2\), \(\ln(1.2)=0.1823215568\), and \(\ln(60)=4.094344562\). Therefore,
For D, \(\mathrm{RSS}_D/n=70/60=7/6\), and \(\ln(7/6)=0.1541506798\). Hence the fit contribution is \(60\ln(7/6)=9.249041\), giving
For both criteria, D has the lower value and is preferred in this comparison. The rule is minimize AIC or BIC; maximizing either reverses the intended ranking. Here D also has a lower RSS, so its improved fit and smaller penalty both favor it.
Common mistake avoided: Do not assume the model with more coefficients must win because it fits more closely, or that a criterion should be maximized. Compare the complete criterion values and select the smaller one.
Worked Example: Different Samples Make a Ranking Invalid
A hypothetical analyst reports Model E with \(n_E=100\), \(\mathrm{RSS}_E=90\), and \(p_E=3\), and Model F with \(n_F=80\), \(\mathrm{RSS}_F=70\), and \(p_F=2\). The models were fitted to different observations. Calculate their reduced BIC values to illustrate the issue, then compare them on a common set of 80 observations where refitting gives \(\mathrm{RSS}_E=76\) and \(\mathrm{RSS}_F=70\).
On their original samples, E has \(\mathrm{RSS}_E/n_E=0.9\). Since \(\ln(0.9)=-0.1053605157\) and \(\ln(100)=4.605170186\),
F has \(\mathrm{RSS}_F/n_F=0.875\), with \(\ln(0.875)=-0.1335313926\) and \(\ln(80)=4.382026635\). Thus,
It would be invalid to declare F the winner from these two values. The models use different responses-in-the-sample: their RSS values summarize errors on different observations, and their sample-size terms also differ. Instead, use the stated common set of 80 observations and the refitted RSS values. For E, \(76/80=0.95\), \(\ln(0.95)=-0.0512932944\), and \(80\ln(0.95)=-4.103464\). For F, \(70/80=0.875\), so its fit contribution remains \(-10.682511\). Both now use \(\ln(80)=4.382026635\):
On the common sample, F has the lower reduced BIC. This is a valid criterion-based ranking under the stated Gaussian assumptions. The arithmetic has been checked using the unrounded logarithms; rounding is to six decimal places.
Common mistake avoided: A lower number does not rescue a comparison made on different samples. Fit every candidate to the same observations before comparing the criteria.
Constants, Parameter Counts, and Interpretation
AIC and BIC are often shown in reduced form because shared terms cancel in a ranking. That convenience has limits. Omit a term only after confirming it is genuinely common to all models being compared. In the Gaussian formulas here, the likelihood constant is common when \(n\), the response, and the likelihood specification are common; the variance penalty is common when every model estimates the same one variance parameter. If sample sizes differ, even the likelihood constant \(n[\ln(2\pi)+1]\) differs. If models use different response scales or transformed responses, their likelihoods may also require terms absent from the simple RSS expressions. Use a consistent likelihood-based calculation rather than mixing formulas.
The selected model is not thereby proven to be the true model, nor is the criterion a probability that the model is correct. It is the preferred candidate under a particular criterion, a specified candidate set, and the assumptions used to calculate the likelihood. Different reasonable candidate sets or criteria can produce different rankings. A small difference should not be turned into certainty; interpret it in context and consider whether the assumptions and available predictors support the comparison.
| Check | Question to ask | Why it matters |
|---|---|---|
| Parameter count | Did every model count its regression coefficients and estimated variance consistently? | The complexity penalty depends on how many parameters are estimated. |
| Direction | Did I select the smaller criterion value? | AIC and BIC are minimized, not maximized. |
| Sample | Were the same observations used for every candidate? | RSS and likelihood values from different samples are not directly comparable. |
| Constants | Were omitted terms common to all candidates? | Only common terms can be dropped without affecting the ranking. |
| Conclusion | Have I described a criterion-based preference rather than certainty? | The ranking is conditional on the candidate set and assumptions. |
Check Your Understanding
For each question, use the Gaussian regression conventions stated above. Reduced criteria are sufficient when candidates use the same observations and response.
- A model has an intercept and two slopes, fitted to \(n=30\) observations with \(\mathrm{RSS}=30\). What are \(p\), \(\mathrm{AIC}_{\mathrm{red}}\), and \(\mathrm{BIC}_{\mathrm{red}}\), to four decimal places? (Answer: \(p=3\), AIC \(=6.0000\), BIC \(=10.2036\).)
- On the same \(n=25\) observations, Model A has \(p=1\), \(\mathrm{RSS}=25\); Model B has \(p=2\), \(\mathrm{RSS}=20\). Which has lower AIC and BIC? (Answer: Model B; its reduced AIC is \(-1.5786\) and reduced BIC is \(0.8592\), versus A's \(2.0000\) and \(3.2189\).)
- Two models have reduced BIC values of \(5.1\) and \(2.8\), but the first was fitted to 90 observations and the second to 70 different observations. Can the second be declared preferred from these values alone? (Answer: No; refit both models on a common set of observations before ranking them.)
- Two models use the same \(n=50\) observations. Model G has \(p=2\), \(\mathrm{RSS}=55\); Model H has \(p=3\), \(\mathrm{RSS}=50\). Compute their reduced BIC values to four decimal places and select the lower one. (Answer: G \(=12.5896\), H \(=11.7361\); H is preferred by BIC.)
- A candidate has the smallest BIC among four models, but another model would be available if the candidate set were expanded. What conclusion is justified? (Answer: It is the BIC-preferred model among the four candidates under the stated assumptions, not necessarily the true or certainly best model.)