Ranking Regression Models With Cp, AIC, BIC, and Adjusted R-Squared

Why Regression Selection Criteria Can Disagree

A more complex regression model cannot have a larger residual sum of squares than a nested model fitted to the same observations: adding predictors can only maintain or improve the in-sample fit. But fit alone is not a sufficient selection rule. A criterion that rewards fit without accounting for the number of estimated coefficients can favor unnecessarily complex models. Mallows’ \(C_p\), AIC, BIC, and adjusted \(R^2\) each balance fit and complexity in a different way.

These criteria are used to compare candidate models fitted to the same response on the same observations and scale. Their rankings need not agree. A defensible interpretation reports what each criterion favors and recognizes that the choice reflects the criterion’s penalty, not a universal proof that one model is correct. These comparisons are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Four Criteria and Their Penalties

Mallows’ \(C_p\) compares a candidate model’s residual sum of squares with an estimate of the error variance, usually obtained from a suitably large full model. The full model’s residual degrees of freedom are \(n-p_{\text{full}}\), since \(p_{\text{full}}\) includes its intercept.

The complexity penalty is \(2p_m\). If the candidate model is adequate and the variance estimate is appropriate, \(C_p\) is often considered in relation to \(p_m\): a value near \(p_m\) is consistent with the idea that lack of fit is not adding much to the criterion. This is a diagnostic interpretation, not a guarantee that the model is adequate. A very small \(C_p\) is not automatically evidence that every modeling assumption is satisfied.

For Gaussian linear regression with independent errors of common variance, AIC and BIC can be calculated, up to constants common to all candidate models, as follows:

The smaller value is preferred under either criterion. In the full Gaussian likelihood, the error variance is also estimated. Counting it adds a common penalty term to all models—\(2\) for AIC and \(\log n\) for BIC—so those terms can be omitted when ranking candidates. The displayed formulas count the regression coefficients, including the intercept, in \(p_m\).

AIC’s penalty per coefficient is \(2\); BIC’s is \(\log n\). Thus, when \(n>e^2\), BIC applies a larger per-coefficient penalty than AIC. BIC will often prefer a smaller model when the fit improvement from added predictors is modest, although it need not always do so. AIC and BIC are not hypothesis tests, and their numerical values have no absolute interpretation when the candidate set or data changes.

Adjusted \(R^2\) modifies ordinary \(R^2\) to account for the residual degrees of freedom. With an intercept and \(p_m\) counting all fitted coefficients, it is

Unlike the other three criteria, larger adjusted \(R^2\) is preferred. The numerator is the candidate model’s residual mean square, while the denominator is the response’s total mean square about its sample mean. Adding a predictor raises ordinary \(R^2\) or leaves it unchanged, but adjusted \(R^2\) rises only if the reduction in \(RSS\) is sufficient to compensate for the lost residual degree of freedom.

Why These Formulas Have This Form

The adjusted \(R^2\) formula follows directly from its degrees-of-freedom adjustment. Ordinary \(R^2=1-RSS/TSS\) compares unadjusted sums of squares. For a regression with \(p_m\) fitted coefficients, the residual degrees of freedom are \(n-p_m\); the total degrees of freedom around the sample mean are \(n-1\). Replacing each sum of squares by its corresponding mean square gives

This explains why it is incorrect to use \(n-p_m\) with a coefficient count that excludes the intercept when the fitted model includes one: that would use the wrong residual degrees of freedom. The intercept must be counted consistently in this formula.

The AIC and BIC fit terms also have a likelihood justification. Under the stated Gaussian model, maximizing the likelihood over the error variance gives the maximum-likelihood estimate \(\widehat{\sigma}^2_{\text{ML}}=RSS_m/n\). Substituting it into minus twice the log likelihood yields

The first two terms inside the brackets are common to every candidate. AIC adds twice the parameter count, and BIC adds the parameter count times \(\log n\). Removing terms common to all models gives the displayed ranking formulas. This derivation relies on Gaussian errors for the likelihood expression; if that model is unsuitable, these particular Gaussian-regression formulas should not be treated as automatically appropriate.

Worked Rankings

Conventions, Conflicts, and Common Traps

For \(C_p\), AIC, and BIC, using a different coefficient-count convention consistently across every model changes the reported scores by a common amount. For example, excluding the intercept from every \(p_m\) changes the \(2p_m\) terms in AIC and \(C_p\) by the same amount for all candidates; it likewise changes the BIC penalties by a common amount. The rankings are unchanged. Mixing conventions across models, however, can distort those comparisons.

That constant-shift statement does not apply to adjusted \(R^2\). Its degrees-of-freedom ratio changes by model when \(p_m\) changes, so changing the count can change each model’s adjusted \(R^2\) by a different amount. For a model that includes an intercept, count the intercept in \(p_m\) and use \(n-p_m\).

• Do not select by \(RSS\) alone. A larger nested model cannot fit worse in sample, so the smallest \(RSS\) does not account for complexity.
• Do not reverse the direction. Smaller \(C_p\), AIC, and BIC are preferred; larger adjusted \(R^2\) is preferred.
• Do not compare raw criterion values across unrelated data sets. AIC or BIC values require a common response, scale, and observations for a direct ranking.
• Do not treat a near tie as a decisive difference. Report the values at sensible precision and describe the ranking as close when the differences are small.
• Do not treat a ranking as a model check. These criteria do not replace examining assumptions, residual behavior, or the purpose of the analysis.

When criteria conflict, explain the trade-off rather than averaging their scores. AIC often tolerates more complexity than BIC as sample size grows; adjusted \(R^2\) rewards a sufficiently large reduction in residual mean square; and \(C_p\) depends on the quality of the variance estimate from the full model. Each ranking is conditional on its formula and assumptions.