Why Calculate Both Rankings?
AIC and BIC combine model fit with a penalty for the number of estimated parameters. A model with more regression terms can reduce the residual sum of squares (RSS), but the criterion’s penalty may outweigh that improvement. Because the AIC and BIC penalties differ, the two criteria can rank the same candidate regressions differently.
The calculations below use hypothetical regression analyses. Each candidate is fitted to the same response observations, so the sample size and the omitted likelihood constants are common across candidates. That condition matters: a score difference is useful for ranking models only when the candidates are fitted to comparable data.
Some calculations use only the change in the fit term relative to a reference model. This is valid because, for a fixed sample size, the term \(n\ln(RSS_{\text{ref}}/n)\) is common and cancels from every comparison. In the examples, the reported relative score is \(n\ln(RSS/RSS_{\text{ref}})+2k\) for AIC or \(n\ln(RSS/RSS_{\text{ref}})+k\ln(n)\) for BIC. These are not absolute criterion values, but their rankings are identical to those from the displayed formulas.
Why the RSS Appears in the Criteria
The fit term follows from the Gaussian likelihood. For a regression with fitted values \(\widehat{y}_i\), the residual sum of squares is \(RSS=\sum_{i=1}^n(y_i-\widehat{y}_i)^2\). At fixed regression coefficients and variance, the normal log-likelihood contains the term \(-RSS/(2\sigma^2)\). Maximizing over the regression coefficients minimizes RSS.
After fitting the coefficients, the maximum-likelihood estimate of the error variance is \(\widehat{\sigma}^2=RSS/n\). Substituting this estimate into the log-likelihood gives the following form, up to terms that do not depend on the candidate model:
AIC adds \(2k\) to this maximized-likelihood term, while BIC adds \(k\ln(n)\). Thus a lower RSS improves both scores, but a larger parameter count increases both. This substitution relies on the Gaussian likelihood and its variance estimate; it is not a universal formula for every regression model or response distribution.
Worked Rankings
Worked Example: AIC and BIC Prefer Different Models
A hypothetical insurer fits three Gaussian regressions to the same \(n=120\) policy-level observations, using the same response and response scale. The models have the following RSS values and total parameter counts, with \(k\) including the error variance. Rank the candidates by AIC and BIC.
| Candidate | RSS | Total parameters, \(k\) |
|---|---|---|
| A | 240 | 3 |
| B | 210 | 5 |
| C | 205 | 6 |
Method. Use candidate A as the reference. Its RSS is 240, so its relative fit term is \(120\ln(240/240)=0\). For each other candidate calculate \(120\ln(RSS/240)\), then add the applicable penalty. The common reference term is omitted from all three scores.
Candidate A. Its relative fit term is zero. Therefore, \(AIC_A=0+2(3)=6.000\), and \(BIC_A=0+3\ln(120)=3(4.787491743)=14.36247523\).
Candidate B. The RSS ratio is \(210/240=0.875\), and \(\ln(0.875)=-0.1335313926\). Its relative fit term is \(120(-0.1335313926)=-16.02376711\). Thus \(AIC_B=-16.02376711+2(5)=-6.02376711\), while \(BIC_B=-16.02376711+5(4.787491743)=7.91369161\).
Candidate C. The RSS ratio is \(205/240=41/48\), and \(\ln(41/48)=-0.157628944\). Its relative fit term is \(120(-0.157628944)=-18.91547333\). Thus \(AIC_C=-18.91547333+2(6)=-6.91547333\), while \(BIC_C=-18.91547333+6(4.787491743)=9.80947713\).
Ranking and check. The AIC scores rank C, then B, then A, from lowest to highest. The BIC scores rank B, then C, then A. Candidate C improves RSS relative to B by only 5, while using one more parameter. That small fit improvement is enough to give C the lower AIC, but not enough to offset BIC’s larger added penalty, \(\ln(120)\), rather than 2. Each calculated fit term becomes more favorable as RSS decreases, and each penalty increases with \(k\), as expected. The common mistake avoided is declaring one criterion’s winner to be the winner under both.
Worked Example: A Modest RSS Reduction Is Not Enough
A hypothetical study of \(n=100\) equipment-repair records compares a smaller Gaussian regression with \(RSS=100\) and \(k=2\) against a larger one with \(RSS=96.5\) and \(k=4\). Both use exactly the same records. Calculate relative scores using the smaller model as the reference, and determine whether the larger model ranks better.
Fit change. The larger model’s relative fit term is \(100\ln(96.5/100)=100\ln(0.965)=-3.56271776\).
Smaller model. Its relative fit term is zero, so \(AIC=2(2)=4.000\) and \(BIC=2\ln(100)=2(4.605170186)=9.21034037\).
Larger model. Its parameter count is four, so \(AIC=-3.56271776+2(4)=4.43728224\). Its BIC is \(BIC=-3.56271776+4(4.605170186)=14.85796298\).
Ranking and check. Both criteria favor the smaller model: its relative AIC of 4.000 is below 4.437, and its relative BIC of 9.210 is below 14.858. The larger model’s improvement in fit does not compensate for the two additional parameters. The common mistake avoided is treating any reduction in RSS as sufficient reason to select the larger model.
Worked Example: AIC Favors the Larger Model but BIC Does Not
A hypothetical analyst models the average daily claim amount for \(n=30\) days. A baseline Gaussian regression has \(RSS=60\) and \(k=3\); an expanded regression has \(RSS=55\) and \(k=4\). Both are fitted to the same 30 days. Use the baseline as the reference and compare the scores.
Baseline scores. The relative fit term is zero. Therefore, \(AIC_{\text{base}}=2(3)=6.000\), and \(BIC_{\text{base}}=3\ln(30)=3(3.401197382)=10.20359215\).
Expanded-model fit term. The RSS ratio is \(55/60=11/12\). Since \(\ln(11/12)=-0.0870113770\), the relative fit term is \(30(-0.0870113770)=-2.61034131\).
Expanded scores. Its AIC is \(-2.61034131+2(4)=5.38965869\). Its BIC is \(-2.61034131+4(3.401197382)=10.99444822\).
Ranking and check. AIC favors the expanded model because \(5.390<6.000\). BIC favors the baseline because \(10.204<10.994\). The expanded model’s better fit offsets the AIC increase of 2 for one extra parameter, but it does not offset the BIC increase of \(\ln(30)=3.4012\). The common mistake avoided is comparing the RSS improvement with the wrong penalty for the criterion being used.
How to Make a Reliable Comparison
A calculation is only as meaningful as the comparison it represents. Before ranking models, check that the candidates use the same response variable, the same response scale, and the same observations. If one candidate omits records with missing values while another uses them, the sample sizes differ and the likelihood terms are not directly comparable by simply ranking these scores. Also verify that all parameter counts use one convention and include the same types of estimated quantities.
Check that candidates are fitted to the same observations and response scale under a suitable common likelihood framework.
Use the number of observations in the fitted sample, the model’s residual sum of squares, and a consistent total parameter count.
Evaluate \(n\ln(RSS/n)\) and add \(2k\) for AIC or \(k\ln(n)\) for BIC. Alternatively, subtract a common reference fit term.
Order the AIC scores from smallest to largest, then do the same for BIC. Do not assume their rankings must agree.
AIC and BIC are criteria for comparing fitted candidates, not measures of absolute model quality. A lower score identifies the preferred model among the candidates under that criterion; it does not prove that the model is correctly specified or that its predictions will be best in every setting. If the rankings disagree, report the disagreement rather than silently selecting whichever result is more convenient.
Common Calculation Traps
- Choosing the largest score. Both AIC and BIC are minimized. A negative relative score can be perfectly valid because the shared absolute likelihood constant was omitted.
- Using the wrong sample size. The \(n\) in the fit term and \(\ln(n)\) is the number of observations used to fit the candidate, not the number of predictors or the population size.
- Changing the parameter-count convention. Count the intercept and, under the convention here, the error variance. Use one convention for every candidate.
- Comparing noncomparable fits. A common-reference calculation cancels constants only when the candidates use the same observations and response scale. It does not repair a comparison based on different data.
- Rounding too early. Keep unrounded logarithms and scores through the comparison, then round the displayed results. Close rankings can otherwise be reversed by premature rounding.
Check Your Understanding
For each problem, compare the criteria only where the stated candidates use the same observations and response. Parameter counts include the error variance.
- A pair of Gaussian regressions fitted to the same \(n=50\) observations has \((RSS,k)=(100,2)\) and \((90,3)\). Which model ranks better under AIC and BIC? (Answer: The larger model ranks better under both; its AIC difference is approximately \(-3.2680\) and its BIC difference is approximately \(-1.3560\), relative to the smaller model.)
- Two regressions use the same \(n=80\) records and have \((RSS,k)=(160,3)\) and \((150,5)\). Which model does each criterion prefer? (Answer: AIC prefers the larger model; BIC prefers the smaller model. The larger-minus-smaller score differences are approximately \(-1.1631\) for AIC and \(3.6010\) for BIC.)
- For the same \(n=40\) observations, compare a model with \((RSS,k)=(75,4)\) with one having \((RSS,k)=(70,5)\). Which model does each criterion rank lower? (Answer: AIC ranks the larger model lower; BIC ranks the smaller model lower. The larger-minus-smaller differences are approximately \(-0.7597\) and \(0.9292\), respectively.)
- Two Gaussian regressions use the same \(n=64\) observations and have \((RSS,k)=(128,3)\) and \((120,4)\). Calculate the larger-minus-smaller differences and identify each criterion’s preferred model. (Answer: The AIC difference is approximately \(-2.1305\), so AIC prefers the larger model; the BIC difference is approximately \(0.0284\), so BIC narrowly prefers the smaller model.)
- A candidate regression uses 92 observations because some responses are missing, while its comparison model uses 100 observations. Can their standard AIC or BIC scores be ranked directly as if both were fitted to a common sample? (Answer: No. They do not use the same observations, so the likelihood comparison is not directly comparable; fit both candidates to a common sample or use a justified method for handling the differing data.)