Why the Same Models Can Have Different Winners
Model-selection criteria do not ask exactly the same question. Adjusted R-squared compares residual variation with total variation while accounting for residual degrees of freedom. Mallows’ Cp compares residual sum of squares with an estimated error variance and adds a parameter penalty. AIC and BIC use a likelihood-based measure of fit, but apply different complexity penalties. It is therefore possible for all four to rank the same candidates differently without any calculation being wrong.
The diagnostic task is to make the comparison consistent first: use the same observations and response, count parameters the same way, and use the appropriate variance estimate. Then separate each criterion into its fit component and its penalty. This tutorial is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
What Each Criterion Ranks
For adjusted R-squared, the residual degrees of freedom are \(n-k\), while the total sum of squares has \(n-1\) degrees of freedom. A larger adjusted R-squared is preferred. Mallows’ Cp uses a variance estimate from the reference full model; a smaller Cp is preferred. For AIC and BIC under Gaussian linear regression, terms common to all candidate models can be omitted when ranking them.
Here \(AIC^*\) and \(BIC^*\) are ranking scores, not necessarily the full reported criteria: constants shared by all candidate models have been dropped. Smaller AIC and BIC scores are preferred. In the full Gaussian likelihood, the variance parameter is also estimated; its contribution is common across candidates when comparing these models, so it does not change their ranking. The parameter count \(k\) in the displayed ranking formulas is the coefficient count, used consistently across all candidates.
| Criterion | Fit component | Complexity adjustment | Preferred direction |
|---|---|---|---|
| Adjusted R-squared | RSS relative to TSS | Residual degrees of freedom | Larger |
| Mallows’ Cp | RSS relative to \(\widehat{\sigma}^2\) | \(2k\), alongside \(-n\) | Smaller |
| AIC | \(n\ln(RSS/n)\) | \(2k\) | Smaller |
| BIC | \(n\ln(RSS/n)\) | \((\ln n)k\) | Smaller |
The common sample requirement matters: \(n\), \(TSS\), and the likelihood basis must be comparable. If candidates use different observations, a change in a score can reflect a changed data set rather than a better model. Likewise, Cp depends on the chosen estimate \(\widehat{\sigma}^2\), so that estimate must be identified rather than silently changed between candidates.
A Short Justification for Adjusted R-squared Rankings
For candidates fitted to the same response and observations, \(TSS/(n-1)\) is a fixed positive quantity. Rewrite adjusted R-squared as follows:
Since the denominator \(TSS/(n-1)\) is fixed and positive, maximizing adjusted R-squared is equivalent to minimizing \(RSS/(n-k)\). Thus, a smaller RSS does not automatically win: the reduction must be considered together with the change in residual degrees of freedom. This algebraic result explains one source of disagreement with criteria that penalize model size in another way.
Worked Example: Four Criteria, Four Rankings
Worked Example: Four Criteria, Four Rankings
A hypothetical analyst compares four nested Gaussian regression candidates for a portfolio-level response. Each model uses the same \(n=40\) observations and has an intercept. The candidates have \(k=2,3,4,5\) fitted coefficients and respective RSS values \(500,465,445,432\). The total sum of squares is \(TSS=1000\). A specified full reference model has \(k_F=20\) coefficients and \(RSS_F=160\). Find the preferred candidate under each criterion.
Method and setup. First estimate the error variance for Cp using the reference model’s residual degrees of freedom:
The candidates have \(q=k-1\), or \(1,2,3,4\), predictors. The following table gives each ranking score. Adjusted R-squared values are shown to six decimal places; other scores are shown to three decimals.
| \(k\) | \(RSS\) | Adjusted \(R^2\) | \(C_p\) | \(AIC^*\) | \(BIC^*\) |
|---|---|---|---|---|---|
| 2 | 500 | 0.486842 | 26.500 | 105.029 | 108.407 |
| 3 | 465 | 0.509865 | 24.125 | 104.126 | 109.193 |
| 4 | 445 | 0.517917 | 23.625 | 104.368 | 111.123 |
| 5 | 432 | 0.518629 | 24.000 | 105.182 | 113.626 |
Check the adjusted R-squared for \(k=4\). Its residual degrees of freedom are \(40-4=36\), so \(R^2_{\mathrm{adj}}=1-(445/36)/(1000/39)=1-17355/36000=0.5179167\). The \(k=5\) value is \(1-(432/35)/(1000/39)=1-16848/35000=0.5186286\). The largest value is at \(k=5\), so adjusted R-squared selects that candidate.
Check Cp for \(k=3\) and \(k=4\). The \(k=3\) score is \(465/8-40+2(3)=58.125-34=24.125\). The \(k=4\) score is \(445/8-40+2(4)=55.625-32=23.625\). Comparing the full table, Cp is smallest at \(k=4\), so Cp selects that candidate.
Check the AIC ranking. For \(k=3\), \(AIC^*=40\ln(465/40)+6=40\ln(11.625)+6=104.126\). The lowest AIC score in the table is at \(k=3\), so AIC selects it.
Check the BIC ranking. Since \(\ln(40)=3.688879\), the \(k=2\) score is \(40\ln(500/40)+2\ln(40)=101.029+7.378=108.407\). The table shows this is the lowest BIC score, so BIC selects \(k=2\).
Diagnosis. The four decisions are adjusted R-squared: \(k=5\); Cp: \(k=4\); AIC: \(k=3\); BIC: \(k=2\). The rankings do not contradict one another arithmetically. Each criterion trades improvement in fit against complexity differently. This example uses Cp’s reference-model variance estimate explicitly; replacing it with another estimate could change the Cp ranking.
Common mistake avoided: Choosing the candidate with the smallest raw RSS would select \(k=5\), but that ignores every complexity adjustment. Raw RSS can identify the best fit among these nested candidates, not the preferred model under the stated criteria.
Worked Example: AIC and BIC Disagree on an Added Coefficient
Worked Example: AIC and BIC Disagree on an Added Coefficient
A hypothetical claims-frequency analysis compares two Gaussian regression candidates on the same \(n=50\) observations. The smaller model has \(k=2\) coefficients and \(RSS=640\); the larger has \(k=3\) and \(RSS=600\). Calculate the larger-minus-smaller score difference for AIC and BIC, then state each selection.
Method. The shared fit term \(n\ln(RSS/n)\) changes by \(50\ln(600/640)\), and the larger model adds one coefficient. Therefore,
The difference is negative, so the larger model has the lower AIC score and AIC selects it.
For BIC, the added coefficient receives a penalty of \(\ln(50)\), not \(2\):
This difference is positive, so the smaller model has the lower BIC score and BIC selects it. The example isolates the penalty difference: both criteria use the same likelihood-based fit change, but their complexity penalties are not the same.
Common mistake avoided: Reusing the AIC penalty of \(2\) for BIC would incorrectly produce the AIC difference \(-1.226926\) and select the larger model. BIC uses \(\ln(n)\) per added coefficient, which here is \(\ln(50)=3.912023\).
Worked Example: Why Cp Depends on the Variance Estimate
Worked Example: Why Cp Depends on the Variance Estimate
A hypothetical severity model compares candidates fitted to the same \(n=32\) observations. The smaller candidate has \(k=2\), \(RSS=360\); the larger has \(k=3\), \(RSS=330\). Let \(TSS=800\), and suppose the full reference model gives \(\widehat{\sigma}^2=20\). Compare the criteria and then assess how the Cp decision changes if the variance estimate were \(10\).
Adjusted R-squared. For \(k=2\), the residual degrees of freedom are \(30\): \(R^2_{\mathrm{adj}}=1-(360/30)/(800/31)=1-12/(800/31)=1-0.465=0.535000\). For \(k=3\), they are \(29\): \(R^2_{\mathrm{adj}}=1-(330/29)/(800/31)=1-10230/23200=0.559052\). Adjusted R-squared selects the larger candidate.
Cp with the stated estimate. The scores are \(360/20-32+2(2)=18-28=-10\) and \(330/20-32+2(3)=16.5-26=-9.5\). Since \(-10\) is smaller, Cp selects the smaller candidate. A negative Cp is not itself an error; the ranking between candidates is what matters.
AIC and BIC. The larger-minus-smaller AIC difference is \(32\ln(330/360)+2=32\ln(11/12)+2=-2.784364+2=-0.784364\), so AIC selects the larger model. The BIC difference is \(-0.784364+\ln(32)=-0.784364+3.465736=2.681372\), so BIC selects the smaller model.
Change the Cp variance estimate. With \(\widehat{\sigma}^2=10\), the larger-minus-smaller Cp difference is \((330-360)/10+2=-3+2=-1\). The larger model now has the smaller Cp score. Thus the Cp choice changes when the variance estimate changes, while the adjusted R-squared, AIC, and BIC comparisons here do not use that Cp estimate.
Common mistake avoided: Treating the variance estimate as an irrelevant detail can produce a confident but unsupported Cp ranking. State where \(\widehat{\sigma}^2\) came from and use that same value for all candidate Cp scores.
How to Diagnose a Disagreement
When rankings differ, do not immediately assume that a criterion has been misapplied. Audit the comparison in a fixed order:
Check that the response, observations, and response scale are the same for every candidate. A score comparison across different samples may not isolate model quality.
Here \(k\) includes the intercept. If another convention is used, translate it before calculating; do not count the intercept in one model but omit it in another.
Adjusted R-squared uses residual degrees of freedom; Cp uses \(\widehat{\sigma}^2\); AIC uses \(2k\); BIC uses \((\ln n)k\).
Maximize adjusted R-squared, but minimize Cp, AIC, and BIC. A correct score with the wrong direction still gives the wrong selection.
State which candidates improve fit and which criterion’s adjustment changes the ranking. A criterion’s selected model is not automatically the universally best model for every purpose.
AIC and BIC share the same likelihood-based fit term in these Gaussian comparisons, but their penalties differ with sample size. Adjusted R-squared and Cp use different fit summaries again. These differences are the explanation to investigate after confirming the arithmetic, not evidence by themselves that one score is wrong.
Common Traps
- Using the same penalty for AIC and BIC. AIC adds \(2k\); BIC adds \((\ln n)k\). For \(n\) greater than \(e^2\), the BIC penalty per coefficient is larger.
- Comparing raw RSS as if it were a criterion. In nested least-squares models, adding predictors cannot increase RSS, but a criterion’s complexity adjustment can still favor the smaller model.
- Changing the Cp variance estimate mid-comparison. The same specified reference-model estimate must be used across all candidate Cp scores.
- Forgetting residual degrees of freedom. Adjusted R-squared uses \(n-k\) in the residual variance estimate, not \(n-1\).
- Reporting only the winner. Showing the fit measure, penalty, and selected direction makes a disagreement diagnosable and helps reveal a parameter-count or arithmetic error.
Check Your Understanding
For each problem, use the stated conventions and compare only the candidates on the same sample.
- Two candidates use \(n=25\), \(TSS=500\), and have \((k,RSS)=(2,240)\) and \((3,220)\). Calculate adjusted R-squared for both and identify the preferred candidate. (Answer: \(0.4991\) for \(k=2\), \(0.5200\) for \(k=3\); adjusted R-squared selects \(k=3\).)
- For \(n=30\), a smaller model has \(k=2\), \(RSS=270\), and a larger model has \(k=3\), \(RSS=250\). Calculate the larger-minus-smaller AIC difference. (Answer: \(30\ln(250/270)+2=-0.3088\); AIC selects the larger model.)
- For \(n=60\), two candidates have RSS values \(800\) and \(760\), with the latter model having one additional coefficient. Calculate the larger-minus-smaller BIC difference. (Answer: \(60\ln(760/800)+\ln(60)=1.0167\); BIC selects the smaller model.)
- A larger candidate reduces RSS from \(440\) to \(410\) and adds one coefficient. With \(\widehat{\sigma}^2=12\), calculate the larger-minus-smaller Cp difference and state the choice. (Answer: \((410-440)/12+2=-0.5000\); Cp selects the larger model.)
- In a comparison with \(n=50\), a candidate has a slightly smaller RSS but a larger BIC score. Is that possible, and what must be checked before interpreting it? (Answer: Yes; the complexity penalty can outweigh the fit improvement. Confirm the same observations and response scale, consistent coefficient counts, and the BIC formula using \(\ln(50)\) per coefficient.)