Comparing Non-Nested Regression Models

Why Compare Non-Nested Models?

Regression questions often admit more than one plausible model. One model might use a patient’s age to predict a response, while another uses a biomarker; neither model is necessarily a special case of the other. In this situation, a nested-model test is not the general tool for deciding which model is preferable. AIC, BIC, and Mallows \(C_p\) instead provide criteria for comparing model fit while accounting for complexity.

A useful comparison requires care about what is held fixed. The candidate models should use the same response, measured on the same scale, and the same observations. Then their fit statistics refer to the same data, and the criterion scores can be compared directly. A lower score indicates preference under that criterion; it does not establish that one model is true or that its predictors cause the response.

Conditions for a Fair Criterion Comparison

For AIC and BIC in Gaussian least-squares regression, assume that each candidate model uses the same \(n\) response observations and has independent errors with mean zero, constant variance, and a normal distribution. Each model is fitted by least squares, and its residual sum of squares is denoted by \(RSS\). The normality assumption supplies the Gaussian likelihood used in these criteria; independence and constant variance are part of the regression error model. In applications, residual plots and knowledge of the data collection help assess whether these assumptions are reasonable.

For Mallows \(C_p\), the candidate regressions are also fitted to the same response observations. The calculation uses an estimate of the common error variance from a suitably large reference model, typically a full model. That reference model should be adequate for representing the mean response, and its residual variance estimate should be relevant to the candidate models. The usual least-squares assumptions include independent errors with mean zero and a common variance.

For Gaussian AIC and BIC, let \(K\) count all estimated likelihood parameters, including the regression coefficients and the error variance. The usual least-squares formulas, omitting a constant that is identical for all candidates fitted to the same data, are as follows.

For Mallows \(C_p\), let \(p\) count the regression coefficients in the candidate model, including its intercept, and let \(\hat{\sigma}^2\) be the error variance estimate from the reference model. If the candidate model has residual sum of squares \(RSS\), then

The parameter-count conventions differ: AIC and BIC include the estimated error variance in \(K\), whereas \(p\) for \(C_p\) counts regression coefficients only. Use one convention consistently across candidates. Lower AIC, BIC, or \(C_p\) scores indicate a preferred candidate according to that criterion. The scores are rankings, not probabilities that models are correct.

How the Rankings Work

The fit part of Gaussian AIC and BIC improves when \(RSS\) decreases. Their penalty parts increase with the number of estimated parameters. To see how the comparison works, subtract the score for model A from the score for model B. The shared response observations make \(n\) the same for both models, so

and likewise

These expressions follow by subtracting the respective score formulas: the shared terms \(n\log(n)\) cancel, leaving the logarithm of the residual-sum-of-squares ratio, and the difference between the complexity penalties. If model B has smaller residual error, the logarithmic fit term favors B. If B also has more parameters, the penalty works against it. AIC and BIC can therefore disagree: for \(n>e^2\), BIC’s penalty per added parameter, \(\log(n)\), exceeds AIC’s penalty of 2.

Mallows \(C_p\) for Non-Nested Candidates

Mallows \(C_p\) can rank non-nested least-squares models when they share the response observations and use the same error-variance reference estimate. Its formula uses \(p\), the candidate’s number of regression coefficients including the intercept. The estimate \(\hat{\sigma}^2\) is commonly calculated as the reference model’s residual sum of squares divided by its residual degrees of freedom, \(n-p_{\text{full}}\), where \(p_{\text{full}}\) counts its regression coefficients. A smaller \(C_p\) can be preferred among the candidates, but the estimate and adequacy of the reference model matter.

Criterion Rankings Are Not Nested-Model Tests

A standard partial \(F\) test compares nested models: the smaller model’s restrictions must be expressible as constraints within the larger model. Its null hypothesis concerns those restrictions, and the test statistic is evaluated against an \(F\) distribution under the relevant linear-model assumptions. If two models are non-nested, neither supplies that restricted-versus-full structure. Simply subtracting their residual sums of squares and applying a partial \(F\) test is not justified.

AIC, BIC, and \(C_p\) answer a different question. They produce criterion scores that balance fit and complexity, without giving a \(p\)-value for a null hypothesis that one non-nested model is correct. A lower score is not the probability that the model is true, and the difference between scores is not itself a test statistic with a standard null distribution. These criteria are useful for comparing candidate predictive or explanatory descriptions, but their interpretation depends on the criterion, the candidate set, and the assumptions used to compute it.

Common Mistakes and a Practical Check

• Comparing unlike data: Scores from models fitted to different observations, different response transformations, or different response scales are not directly comparable using these formulas. Confirm the response and data rows match.
• Counting parameters inconsistently: For Gaussian AIC and BIC, include the estimated error variance in \(K\). For \(C_p\), \(p\) counts regression coefficients, including the intercept.
• Treating the winner as proven true: A lower criterion score means only that the model is preferred by that criterion among the candidates compared.
• Using a nested-model test for non-nested candidates: A partial \(F\) test requires nested models. AIC, BIC, and \(C_p\) are rankings, not substitutes for that test or a source of a \(p\)-value.

Before comparing scores, check that all models use the same response observations, that their assumptions are plausible, and that the parameter counts follow the criterion’s convention. Then state which criterion ranks which candidate lower, without claiming more than the comparison supports.