Why Compare Several Criteria?
A model comparison is not just a search for the smallest numerical score. Adjusted R-squared, AIC, BIC, and Mallows Cp summarize fit and complexity in different ways, so they can prefer different candidate models. A useful comparison calculates each criterion consistently, identifies what each one favors, and states what the comparison cannot establish.
The calculations below use ordinary least-squares regression with an intercept. Let \(n\) be the common number of observations, \(k\) the number of predictors (not counting the intercept), \(RSS\) the residual sum of squares, and \(TSS\) the total sum of squares about the sample mean of the response. For Mallows Cp, \(p=k+1\) counts the regression coefficients, including the intercept. For Gaussian AIC and BIC, the parameter count also includes the error variance, so it is \(k+2\).
The Four Criteria
For a candidate model with \(k\) predictors, the coefficient of determination is \(R^2=1-RSS/TSS\). Adjusted R-squared modifies this quantity to account for the model’s residual degrees of freedom:
A larger adjusted R-squared is preferred among the models being compared. Unlike ordinary \(R^2\), adjusted R-squared can decrease when a predictor is added, because the reduction in residual sum of squares may be too small to compensate for using another degree of freedom.
For Gaussian least-squares models fitted to the same response and observations, convenient versions of AIC and BIC omit constants common to all candidates:
A smaller AIC or BIC is preferred within the specified comparison. Both penalize model complexity, but BIC’s penalty per added parameter, \(\ln(n)\), exceeds AIC’s penalty of 2 when \(n>e^2\), approximately 7.39. They can therefore rank the same candidates differently. These are relative scores: an AIC or BIC value by itself is not a probability, a test statistic, or evidence that the model is correct.
Mallows Cp compares a candidate model’s residual sum of squares with an estimate of error variance from a specified full model. If the full model has \(p_{\mathrm{full}}\) regression coefficients, define \(\widehat{\sigma}^{\,2}_{\mathrm{full}}=RSS_{\mathrm{full}}/(n-p_{\mathrm{full}})\). Then for a candidate with \(p=k+1\) regression coefficients,
A common diagnostic is whether Cp is reasonably small and near \(p\), the number of regression coefficients. It is not a rule that the candidate with the numerically smallest Cp must be chosen: the estimate of error variance and the adequacy of the full model matter.
Worked Example: Comparing All Four Criteria
Worked Example: Comparing All Four Criteria
Suppose a hypothetical data set has \(n=40\) observations and \(TSS=200\). Three least-squares candidates, all fitted to the same response and observations, have the following residual sums of squares. Candidate A has one predictor, candidate B has two, and candidate C has three.
| Candidate | Predictors \(k\) | \(RSS\) |
|---|---|---|
| A | 1 | 130 |
| B | 2 | 120 |
| C | 3 | 118 |
Adjusted R-squared. For A, the residual degrees of freedom are \(40-1-1=38\), so \(R^2_{\mathrm{adj}}=1-(130/38)/(200/39)=0.3329\). For B, they are \(40-2-1=37\), giving \(1-(120/37)/(200/39)=0.3676\). For C, they are \(40-3-1=36\), giving \(1-(118/36)/(200/39)=0.3608\). Adjusted R-squared favors B.
AIC. The total parameter counts, including the error variance, are \(k+2\). For A, \(40\ln(130/40)+2(3)=53.15\). For B, \(40\ln(120/40)+2(4)=51.94\). For C, \(40\ln(118/40)+2(5)=53.27\). The smaller AIC is B’s, so AIC favors B.
BIC. Using \(\ln(40)\approx3.6889\), the scores are \(40\ln(130/40)+3\ln(40)=58.21\) for A, \(40\ln(120/40)+4\ln(40)=58.70\) for B, and \(40\ln(118/40)+5\ln(40)=61.72\) for C. BIC favors A, the model with one predictor.
Mallows Cp. Treat C as the specified full model. Its residual variance estimate is \(\widehat{\sigma}^{\,2}_{\mathrm{full}}=118/(40-4)=118/36=3.278\), since it has four regression coefficients, including the intercept. The candidates have \(p=2,3,4\) regression coefficients. Thus \(C_{p,A}=130/(118/36)-40+2(2)=3.661\), \(C_{p,B}=120/(118/36)-40+2(3)=2.610\), and \(C_{p,C}=118/(118/36)-40+2(4)=4.000\). B has the smallest Cp; C’s Cp equals its \(p\), while B’s is fairly close to its \(p=3\).
The numerical results do not give a unanimous winner: adjusted R-squared and AIC favor B, BIC favors A, and the Cp values require considering both their size and their relation to \(p\). The residual sum of squares falls as predictors are added, but that fact alone does not settle whether the extra predictors are worth the added complexity.
Why Adjusted R-Squared Can Change Direction
The adjusted R-squared formula has a direct interpretation through residual mean squares. For fixed data, \(TSS/(n-1)\) is the same across candidates. Therefore, maximizing adjusted R-squared is equivalent to minimizing \(RSS/(n-k-1)\), the residual sum of squares per residual degree of freedom.
This explains why adjusted R-squared and ordinary \(R^2\) can disagree. Ordinary \(R^2\) cannot decrease when a predictor is added to a least-squares model fitted to the same data, because the larger model can reproduce the smaller model by setting the added coefficient to zero. Adjusted R-squared includes a degrees-of-freedom adjustment and can decrease instead.
Worked Example: A BIC Preference That Reverses
Worked Example: A BIC Preference That Reverses
Consider two hypothetical least-squares candidates fitted to the same \(n=100\) observations and response. The smaller model has \(k=1\) predictor and \(RSS=200\); the larger has \(k=2\) predictors and \(RSS=185\). Their BIC values, using the same-response Gaussian formula, are:
For the smaller model, the total parameter count is \(k+2=3\): \(100\ln(200/100)+3\ln(100)=83.13\). For the larger model, it is \(k+2=4\): \(100\ln(185/100)+4\ln(100)=79.94\).
BIC favors the larger model by approximately \(83.13-79.94=3.19\) points. The fit improvement is large enough in this comparison to outweigh BIC’s additional complexity penalty. This result applies to these candidates and these observations; it does not establish that the larger model is the true model or that its added predictor has a causal effect.
Worked Example: Reading Mallows Cp Against Its Target
Worked Example: Reading Mallows Cp Against Its Target
Suppose a hypothetical sample has \(n=50\), and a specified full model has three predictors plus an intercept, so \(p_{\mathrm{full}}=4\). If its residual sum of squares is 123, the full-model variance estimate is \(\widehat{\sigma}^{\,2}_{\mathrm{full}}=123/(50-4)=123/46=2.674\). Three candidates have residual sums of squares 160, 125, and 123, with one, two, and three predictors, respectively.
Their coefficient counts are \(p=2,3,4\). For the one-predictor candidate, \(C_p=160/(123/46)-50+2(2)=160(46)/123-46=13.84\), well above \(p=2\). For the two-predictor candidate, \(C_p=125/(123/46)-50+2(3)=125(46)/123-44=2.748\), close to \(p=3\). For the full candidate, \(C_p=123/(123/46)-50+2(4)=46-50+8=4.000\), exactly \(p=4\).
The first candidate’s large Cp is a warning that its residual error is high relative to the full-model variance estimate. The other two values are close to their respective coefficient counts. This pattern does not prove that either candidate is correct; it makes clear why interpreting Cp requires the target \(p\) and the source of the variance estimate, not just a ranking from smallest to largest.
Making a Responsible Comparison
Before reporting a preferred candidate, check that each criterion is being used on a comparable basis. In particular, AIC and BIC comparisons based on the formulas above require the same response scale and observations; changing either can alter the likelihood and its score. Count the intercept and error variance correctly in their parameter penalties. For adjusted R-squared, count predictors in \(k\) and use the residual degrees of freedom \(n-k-1\). For Cp, count regression coefficients in \(p\) and state which full model supplied \(\widehat{\sigma}^{\,2}_{\mathrm{full}}\).
A criterion ranking is conditional on the candidates supplied to it. It does not assess every possible model, verify that model assumptions hold, or quantify uncertainty in the fitted coefficients. The conclusions are also not hypothesis tests: a lower score does not provide a p-value or a probability that the model is true. State the criterion, the candidates compared, and any meaningful disagreement. If several criteria select different candidates, report that disagreement rather than presenting one ranking as a universal verdict.
- Adjusted R-squared rewards fit while adjusting for residual degrees of freedom; larger is preferred.
- AIC and BIC trade fit against parameter-count penalties; smaller is preferred, with BIC generally penalizing complexity more strongly at larger \(n\).
- Mallows Cp is interpreted relative to the candidate’s number of regression coefficients and a full-model variance estimate.
- No criterion can rescue an unsuitable candidate set or turn an observational association into a causal conclusion.
Common Mistake
Correction: Least-squares RSS cannot increase when predictors are added to nested models on the same observations. That decrease alone does not account for complexity. Compare the specified criteria, check their assumptions and parameter counts, and describe the result as a preference among the candidates—not proof that one model is true.
Check Your Understanding
Use the formulas and distinctions in this workshop to answer the following questions.
- Why might adjusted R-squared fall when a predictor is added even though ordinary \(R^2\) cannot fall?
- In the Gaussian AIC and BIC formulas used here, what parameters are included in the count \(k+2\)?
- For \(n=40\), which has the larger complexity penalty per parameter: AIC or BIC? Explain.
- Why must a Mallows Cp interpretation identify both \(p\) and the full model used to estimate error variance?
- If AIC and BIC favor different candidates, what can be concluded—and what cannot be concluded—from that disagreement?