One Regression Table, Four Ways to Compare Models
A regression output table may report several candidate models fitted to the same response and observations. The models trade fit against complexity: adding predictors cannot increase the residual sum of squares (RSS) for nested least-squares models, but a more complicated model may not be worth that improvement. Adjusted R-squared, Mallows’ Cp, AIC, and BIC quantify this tradeoff in different ways.
The practical task is not to make the four measures agree. It is to calculate each using its own definition, then identify which candidate it prefers. The methods below are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Definitions and Conventions
The model assumptions matter. The adjusted R-squared formula below uses an intercept and the usual residual and total degrees of freedom. Mallows’ Cp uses an estimate of error variance from the full model, not a separate estimate from each candidate. The Gaussian regression AIC and BIC formulas assume normally distributed, independent errors with a common variance. All candidates must use the same observations and response for the comparisons in this tutorial.
- Adjusted R-squared: \(R^2_{\mathrm{adj}}=1-\dfrac{RSS/(n-k)}{TSS/(n-1)}\). Prefer the largest value.
- Mallows’ Cp: \(C_p=\dfrac{RSS}{\widehat{\sigma}_F^2}-n+2k\), where \(\widehat{\sigma}_F^2=RSS_F/(n-k_F)\) is the full model’s residual mean square. Smaller Cp is preferred; its size is also assessed relative to \(k\).
- Gaussian regression AIC: up to a constant shared by all candidates, \(AIC=n\ln(RSS/n)+2K\).
- Gaussian regression BIC: up to a constant shared by all candidates, \(BIC=n\ln(RSS/n)+K\ln n\).
These are not interchangeable formulas. Adjusted R-squared compares residual mean square with the response’s sample variance. Cp scales RSS by a full-model variance estimate and adds a coefficient-count term. AIC and BIC use the Gaussian likelihood’s RSS term and penalize the number of estimated parameters. A larger adjusted R-squared is preferred; smaller Cp, AIC, and BIC scores are preferred.
Why Adjusted R-squared Uses Degrees of Freedom
Ordinary \(R^2=1-RSS/TSS\) tends not to decrease when predictors are added to a least-squares model. Adjusted R-squared instead compares two variance estimates: the residual mean square \(RSS/(n-k)\) and the sample variance of the response \(TSS/(n-1)\). Substituting these estimates into “one minus residual variance divided by response variance” gives
This is the stated formula. The degrees of freedom account for the regression coefficients estimated from the data, including the intercept. As a result, adding a predictor can lower adjusted R-squared if the reduction in RSS is too small to offset the reduced residual degrees of freedom.
Worked Comparison Across All Four Criteria
Worked Example: Ranking Four Models From One Output Table
A hypothetical portfolio study fits four nested regressions to the same \(n=40\) observations, with an intercept in every model. The response has \(TSS=1{,}000\). The full model has four predictors and \(RSS_F=700\). The table gives each candidate’s predictor count and RSS. Calculate all four criteria and identify each preferred candidate.
| Model | Predictors \(p\) | Coefficients \(k=p+1\) | RSS |
|---|---|---|---|
| M1 | 1 | 2 | 950 |
| M2 | 2 | 3 | 800 |
| M3 | 3 | 4 | 735 |
| M4 (full) | 4 | 5 | 700 |
Method and full-model variance. Start with the quantities needed by all four calculations. The full model has \(40-5=35\) residual degrees of freedom, so
Adjusted R-squared. For M1, \(RSS/(n-k)=950/(40-2)=950/38=25\), giving \(1-25/(1{,}000/39)=0.025\). For M2, the residual mean square is \(800/37\), giving \(1-(800/37)/(1{,}000/39)\approx0.157\). For M3, it is \(735/36\), giving approximately \(0.204\). For M4, it is \(700/35=20\), giving \(1-20/(1{,}000/39)=0.220\). The largest value is M4.
Mallows’ Cp. Apply the same full-model variance estimate to each candidate:
The smallest Cp is 4.75, so M3 has the minimum Cp. Its value is also near its coefficient count, \(k=4\), which is a useful calibration check.
AIC and BIC. For these criteria, \(K=k+1\), so the values of \(K\) are 3, 4, 5, and 6. The common term \(n[\ln(2\pi)+1]\) is omitted. For instance, M1 has \(40\ln(950/40)+2(3)\approx132.703\) for AIC and \(40\ln(950/40)+3\ln(40)\approx137.770\) for BIC. Applying the same formulas to each row gives:
| Model | Adjusted R-squared | Cp | AIC score | BIC score |
|---|---|---|---|---|
| M1 | 0.025 | 11.50 | 132.703 | 137.770 |
| M2 | 0.157 | 6.00 | 127.829 | 134.585 |
| M3 | 0.204 | 4.75 | 126.440 | 134.884 |
| M4 | 0.220 | 5.00 | 126.488 | 136.621 |
Decision and check. Adjusted R-squared prefers M4, Cp prefers M3, AIC prefers M3, and BIC prefers M2. Each decision follows its own direction of preference: maximize adjusted R-squared and minimize the other three scores. As a check, each nested model’s RSS is at least as large as the full model’s RSS of 700.
Common mistake avoided: Do not use the number of predictors in place of \(k\). For example, M3 has three predictors but four regression coefficients, including the intercept. Cp uses \(k=4\); AIC and BIC count \(K=5\), adding the error-variance parameter.
Two More Calculation Patterns
Worked Example: A Close AIC–BIC Comparison
Two hypothetical Gaussian regressions use the same \(n=25\) observations. Model A has \(k=3\) regression coefficients and \(RSS=250\); Model B has \(k=4\) and \(RSS=220\). Calculate the relative AIC and BIC scores and identify the preferred model for each.
Method. For AIC, the parameter counts are \(K_A=4\) and \(K_B=5\). The RSS per observation is \(250/25=10\) for A and \(220/25=8.8\) for B:
AIC prefers B because its score is lower. For BIC, use \(\ln(25)\approx3.218876\):
BIC prefers A, by a small margin. The calculations use natural logarithms and omit only a constant shared by the two models. Common mistake avoided: \(K\) includes the variance parameter even though it does not appear in the regression equation. Forgetting it gives incorrect absolute relative scores under the displayed convention, even when a shared one-parameter omission happens not to change the ranking.
Worked Example: Mallows’ Cp and the Full-Model Reference
A hypothetical study has \(n=20\) observations. Its full model has \(k_F=4\) regression coefficients and \(RSS_F=160\). Two candidate models have \((k,RSS)=(2,200)\) and \((3,170)\). Calculate their Cp values and choose the model with the smaller Cp.
Method. Estimate error variance from the full model first. Its residual degrees of freedom are \(20-4=16\), so \(\widehat{\sigma}_F^2=160/16=10\). For the first candidate,
For the second candidate,
The second candidate has the smaller Cp, 3 rather than 4. Its Cp also equals its coefficient count \(k=3\). Common mistake avoided: do not estimate \(\widehat{\sigma}^2\) separately from each candidate’s RSS and residual degrees of freedom. Cp uses the full model’s variance estimate for both candidates; otherwise the comparison no longer follows the stated Cp definition.
Reading the Rankings Carefully
A comparison table is useful only if its assumptions and conventions are consistent. Before interpreting a result, verify that the response, observations, and treatment of the intercept are the same across candidates. In particular, the simple AIC and BIC scores shown here omit a constant only because the candidates share the same \(n\) and response under the Gaussian model. If those conditions do not hold, this shortcut may not give a valid comparison.
- Check the direction. A larger adjusted R-squared is preferred; smaller Cp, AIC, and BIC are preferred.
- Check the counts. The \(k\) used in adjusted R-squared and Cp includes the intercept. The \(K\) used in AIC and BIC also includes the estimated error variance.
- Check Cp’s reference model. Use the specified full model’s residual mean square, with its own residual degrees of freedom.
- Check the likelihood assumptions. The AIC and BIC expressions here are for Gaussian linear regression with a common error variance.
- Report each result separately. Different criteria can select different models; calculate and state each ranking rather than assuming they must agree.
These checks are especially important when a problem supplies a table with model size, RSS, and a full-model estimate. A correct RSS calculation paired with the wrong degrees of freedom or parameter count still produces the wrong criterion. Keep the model definitions beside the output as you work, and show enough intermediate arithmetic to make each score auditable.
Check Your Understanding
For each item, use the stated convention and identify the requested value or model. Round numerical criterion values to three decimals where needed.
- A model uses \(n=18\) observations, has \(k=3\) regression coefficients including the intercept, \(RSS=270\), and \(TSS=340\). Calculate its adjusted R-squared. (Answer: 0.100)
- A candidate has \(n=30\), \(k=4\), and \(RSS=300\). The specified full model gives \(\widehat{\sigma}_F^2=12\). Calculate Cp. (Answer: 3)
- Two Gaussian regression candidates use \(n=16\). Model A has \(k=2\) and \(RSS=160\); Model B has \(k=3\) and \(RSS=130\). Using \(K=k+1\), which has the lower relative AIC? (Answer: Model B; AIC scores are approximately 42.841 for A and 41.529 for B.)
- Two Gaussian regression candidates use \(n=100\). Model A has \(k=2\), \(RSS=900\); Model B has \(k=3\), \(RSS=820\). Using \(K=k+1\), which has the lower relative BIC? (Answer: Model B; BIC scores are approximately 233.538 for A and 228.834 for B.)
- A full model has \(n=25\), \(k_F=5\), and \(RSS_F=200\). A candidate has \(k=3\) and \(RSS=250\). Calculate the candidate’s Cp using the full-model variance estimate. (Answer: \(\widehat{\sigma}_F^2=200/(25-5)=10\), so \(C_p=250/10-25+2(3)=6\).)