Comparing Nested And Non-Nested Regression Models

Why Model Nesting Is Not Required for Information Criteria

Regression candidates are often arranged as a sequence: a smaller model uses a few predictors, and a larger model adds more. In that case the smaller model is nested within the larger one. But candidates can also use different predictors without either model being a restricted version of the other. A model using age and account tenure, for example, need not contain a model using only region indicators, or be contained by it.

AIC and BIC can compare either kind of candidate. They balance fit, measured through the maximized likelihood, against a penalty for the number of estimated parameters. Nesting matters for some hypothesis tests, but it is not a condition for comparing information criteria. The comparison still needs a consistent likelihood basis: here, Gaussian linear-regression models fitted to the same response and observations.

Definition: Two regression models are nested when the smaller model can be obtained from the larger model by imposing restrictions on its parameters. Equivalently, the set of mean vectors allowed by the smaller model is contained in the set allowed by the larger model. If neither model's allowed set is contained in the other's, they are non-nested.

For example, an intercept-and-\(x_1\) model is nested within an intercept-\(x_1,x_2\) model: setting the \(x_2\) coefficient to zero recovers the smaller model. By contrast, an intercept-\(x_1\) model and an intercept-\(x_2\) model are generally non-nested. Neither is obtained from the other merely by setting one of its coefficients to zero.

AIC and BIC for Gaussian Regression

Assume that each candidate has the form \(Y_i=\boldsymbol{x}_i^\mathsf{T}\boldsymbol{\beta}+\varepsilon_i\), where the errors are independent and identically distributed as \(N(0,\sigma^2)\). The design matrix is treated as fixed, and the candidate is fitted by maximum likelihood. Write \(n\) for the common number of observations and \(\mathrm{RSS}\) for the residual sum of squares. Let \(k\) count all estimated parameters, including the regression coefficients and the error variance. With an intercept, a model with \(p\) predictors therefore has \(k=p+2\).

For candidates fitted to the same response and observations, terms common to all candidates can be omitted from the criteria. The resulting comparison formulas are:

\mathrm{AIC}=n\log(\mathrm{RSS}/n)+2k,\qquad \mathrm{BIC}=n\log(\mathrm{RSS}/n)+k\log n.

Lower values indicate the preferred candidate among those being compared under the stated setup. AIC and BIC do not require one candidate to be nested within another. Their penalties differ: AIC adds \(2\) per estimated parameter, while BIC adds \(\log n\). Thus BIC's per-parameter penalty exceeds AIC's when \(n>e^2\), though the resulting rankings still depend on both fit and complexity.

Important condition: These simplified RSS formulas assume Gaussian linear regression and a common response, sample, and likelihood convention across candidates. They are not a license to compare unrelated RSS values. If the models use different observations or different likelihoods, the omitted terms may not be common, and this direct comparison may not be valid.

Why the RSS Formula Works

The connection between fit and the information criteria follows from the Gaussian likelihood. For a candidate with fitted means \(\widehat{\mu}_i\), its log-likelihood is

\ell(\boldsymbol{\beta},\sigma^2) =-\frac{n}{2}\log(2\pi\sigma^2) -\frac{1}{2\sigma^2}\sum_{i=1}^n(Y_i-\widehat{\mu}_i)^2.

For fixed regression coefficients, maximizing this expression over \(\sigma^2\) gives \(\widehat{\sigma}^2=\mathrm{RSS}/n\). Substituting that estimate into the likelihood gives the maximized log-likelihood. The definition of AIC is \(-2\ell_{\max}+2k\), and the definition of BIC is \(-2\ell_{\max}+k\log n\). Substitution yields the displayed RSS formulas, apart from terms common to candidates.

-2\ell_{\max} =n\left\{\log(2\pi)+1+\log(\mathrm{RSS}/n)\right\}.

The terms \(n\{\log(2\pi)+1\}\) are identical when \(n\) is common, so they do not affect the ordering. This derivation also explains why a smaller RSS helps but does not settle the comparison: adding parameters can improve fit while increasing the penalty.

Worked Comparisons

Worked Example: Comparing Two Non-Nested Predictor Sets

A hypothetical insurer studies annual claim cost for \(n=40\) policyholders. Both candidates use the same response and the same 40 records, with independent normal errors of common variance. Model A has an intercept and predictors \(x_1,x_2\), with \(\mathrm{RSS}_A=80\). Model B has an intercept and predictor \(x_3\), with \(\mathrm{RSS}_B=90\). Neither predictor set contains the other. Compare the candidates by AIC and BIC.

Method. Use the Gaussian-regression information criteria. Model A estimates three regression coefficients and one variance parameter, so \(k_A=4\). Model B estimates two regression coefficients and one variance parameter, so \(k_B=3\).

For Model A, \(\mathrm{RSS}_A/n=80/40=2\), and \(\log 2=0.693147\). Therefore,

\mathrm{AIC}_A=40(0.693147)+2(4)=27.7259+8=35.7259.

For Model B, \(\mathrm{RSS}_B/n=90/40=2.25\), and \(\log 2.25=0.810930\). Thus,

\mathrm{AIC}_B=40(0.810930)+2(3)=32.4372+6=38.4372.

For BIC, \(\log 40=3.688879\). The two values are

\mathrm{BIC}_A=27.7259+4(3.688879)=42.4814, \qquad \mathrm{BIC}_B=32.4372+3(3.688879)=43.5038.

Conclusion and check. Model A has the lower AIC and the lower BIC, so both criteria prefer it among these candidates. Its RSS is also lower, but the ranking is not based on RSS alone: the calculation includes the parameter penalties. The models are non-nested, so describing one as a reduced version of the other would be incorrect. The common mistake avoided here is rejecting an information-criterion comparison merely because the candidates are non-nested.

Worked Example: Nested Candidates Receive Different AIC and BIC Rankings

For \(n=100\) hypothetical accounts, a model with an intercept and \(x_1\) has \(\mathrm{RSS}_S=108\). A larger model with an intercept and predictors \(x_1,x_2,x_3\) has \(\mathrm{RSS}_L=100\). Both are fitted under independent normal errors with a common variance. Compare the rankings.

Method. The smaller model is nested within the larger because setting the coefficients of \(x_2\) and \(x_3\) to zero gives the smaller model. The smaller model has \(k_S=2+1=3\); the larger has \(k_L=4+1=5\), counting the variance parameter in each.

The smaller model has \(\mathrm{RSS}_S/n=1.08\), with \(\log(1.08)=0.076961\). The larger has \(\mathrm{RSS}_L/n=1\), with \(\log 1=0\). Therefore,

\mathrm{AIC}_S=100(0.076961)+2(3)=7.6961+6=13.6961, \qquad \mathrm{AIC}_L=100(0)+2(5)=10.

AIC prefers the larger model, since \(10<13.6961\). For BIC, \(\log 100=4.605170\), so

\mathrm{BIC}_S=7.6961+3(4.605170)=21.5116, \qquad \mathrm{BIC}_L=0+5(4.605170)=23.0259.

BIC prefers the smaller model, since \(21.5116<23.0259\). The fit improvement is enough to offset AIC's added penalty, but not BIC's. The check is consistent with the criteria's penalties: the larger model estimates two extra parameters, costing \(4\) AIC units and \(2\log 100=9.2103\) BIC units.

Common mistake avoided. Do not assume that nesting forces AIC and BIC to agree. Both criteria compare fit and complexity, but their penalties differ, so a ranking disagreement is possible.

Worked Example: Do Not Apply a Nested-Model Test to Non-Nested Candidates

A hypothetical study compares two models for the same \(n=30\) observations under independent normal errors with common variance. Model C has an intercept and \(x_1\), with \(\mathrm{RSS}_C=45\). Model D has an intercept and \(x_2\), with \(\mathrm{RSS}_D=50\). Neither contains the other. Compare them using AIC and identify the test-related trap.

Method. Each candidate estimates an intercept, one slope, and a variance, so \(k_C=k_D=3\). Since \(45/30=1.5\) and \(50/30=5/3\), with \(\log 1.5=0.405465\) and \(\log(5/3)=0.510826\),

\mathrm{AIC}_C=30(0.405465)+6=12.16395+6=18.16395, \qquad \mathrm{AIC}_D=30(0.510826)+6=15.32478+6=21.32478.

AIC prefers Model C because \(18.16395<21.32478\). The equal parameter counts mean the AIC ranking follows the likelihood fit here, but it remains a criterion comparison, not a test of whether \(x_1\) has a statistically significant effect.

Common mistake avoided. One might form a likelihood-ratio statistic from the two maximized likelihoods and compare it to a usual chi-square reference distribution as if one model were nested in the other. That ordinary nested-model comparison is not justified: the candidates are non-nested, so the required restriction relationship is absent. Use the applicable information criterion for this comparison; do not assign a nested-test significance level to it.

Interpret the Ranking, Not More Than It Says

A lower information criterion identifies a preferred candidate under that criterion and the comparison conditions. It does not establish that the selected model is the true data-generating model, prove that its predictors are causal, or guarantee better performance on new observations. AIC and BIC also need not select the same candidate; that disagreement reflects their different complexity penalties, not an arithmetic contradiction.

For nested models, separate inferential questions may be addressed using methods whose assumptions and reference distributions apply to the specified restrictions. For non-nested models, the ordinary nested likelihood-ratio test is not automatically available. In either setting, keep the tasks distinct: information criteria rank candidates by a fit-penalty tradeoff, whereas a hypothesis test addresses a defined null hypothesis under its own conditions.

First establish that the response, observations, and likelihood basis match.
Count all estimated parameters consistently, including the variance under the convention used here.
Compute the criterion for each candidate; do not rank only by RSS.
State which criterion prefers which candidate, and avoid claiming that the winner is necessarily true or significant.

Key takeaway: Nesting is not required for AIC or BIC comparisons. It is important for the usual nested-model likelihood-ratio test. For information criteria, use a common likelihood basis, apply the appropriate parameter penalty, and interpret the result as a relative ranking among the candidates considered.

Check Your Understanding

For each problem, assume Gaussian linear regression with independent errors of common variance and use \(k\) to count regression coefficients plus the variance parameter. Give criterion differences as larger candidate minus smaller candidate where requested.

Two candidates use the same \(n=25\) observations. Model S has \(k=3\) and RSS \(60\); Model L has \(k=4\) and RSS \(50\). What is \(\mathrm{AIC}_L-\mathrm{AIC}_S\), and which candidate does AIC prefer? (Answer: \(25\log(5/6)+2=-2.558\); AIC prefers Model L.)
For \(n=50\), Model A has \(k=3\) and RSS \(100\), while Model B has \(k=4\) and RSS \(90\). Calculate \(\mathrm{BIC}_B-\mathrm{BIC}_A\) and state the BIC choice. (Answer: \(50\log(0.9)+\log 50=-1.356\); BIC prefers Model B.)
Two non-nested candidates have the same \(n=36\) and the same parameter count. Their RSS values are \(72\) and \(80\). Which has lower AIC and BIC, and why? (Answer: The candidate with RSS \(72\) has lower AIC and BIC because its fit term is lower and the penalties are equal.)
At \(n=64\), a larger model estimates two more parameters than a smaller model. What upper bound must the RSS ratio \(\mathrm{RSS}_L/\mathrm{RSS}_S\) satisfy for BIC to prefer the larger model? (Answer: \(\mathrm{RSS}_L/\mathrm{RSS}_S<\exp\{-2\log(64)/64\}=0.8781\), approximately.)
Model E uses an intercept and \(x_1\); Model F uses an intercept and \(x_2\), with both fitted to the same observations. Can the ordinary nested-model likelihood-ratio test be assumed valid just because both are Gaussian regressions? What comparison can still be made? (Answer: No; neither candidate is generally nested in the other. AIC or BIC can compare them if the common-response, common-sample likelihood conditions hold.)

What You'll Learn