Nested and Non-Nested Model Comparisons

Why Model Structure Matters

Two GLMs can be fitted to the same response data and still require different comparison methods. If one model is a restricted version of another, a likelihood-ratio test can assess whether the added flexibility improves fit beyond what might be expected by chance. If neither model is a restricted version of the other, simply subtracting their log-likelihoods does not produce the usual likelihood-ratio test.

The key question is not which model has the larger likelihood: a more flexible model fitted by maximum likelihood will generally fit at least as well. The key question is whether the candidates have the relationship and satisfy the assumptions required by the proposed comparison. Correctly identifying that structure is central to GLM comparison aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: Two models are nested when every probability distribution allowed by the smaller model is also allowed by the larger model. Equivalently, the smaller model can be obtained from the larger by imposing restrictions on its parameters. Models are non-nested when neither model’s set of allowed distributions is contained in the other’s.

For example, a logistic regression with a linear age effect is nested within a logistic regression with both linear and quadratic age effects: the smaller model results when the quadratic coefficient is set to zero. By contrast, a model using territory and a model using vehicle type are generally non-nested if neither set of predictors is a restriction of the other. Sharing some predictors does not, by itself, make models nested.

When a Likelihood-Ratio Test Applies

Let \(M_0\) be the restricted model and \(M_1\) the larger model, with \(M_0\) nested in \(M_1\). Write \(\ell_0\) and \(\ell_1\) for their maximized log-likelihoods. The likelihood-ratio statistic is

G^2 = 2(\ell_1-\ell_0).

Under the null hypothesis that the restrictions in \(M_0\) are correct, and under standard regularity conditions, \(G^2\) has approximately a chi-square distribution with degrees of freedom equal to the difference in the models’ numbers of freely estimated parameters. The test asks whether the larger model’s improvement in fit is statistically compelling given the extra parameters. It does not show that the larger model predicts better on new data.

Conditions for the usual test: The models must be nested and fitted by maximum likelihood to the same response observations using the same response likelihood and compatible assumptions. The parameter restrictions must be regular: parameters should be identifiable, the null value should ordinarily be in the interior of the larger model’s parameter space, and the model should not have unresolved rank or convergence problems. The chi-square approximation is generally a large-sample result.

Other details must also be aligned. If the models use different rows because of missing values, their maximized likelihoods do not describe the same observed response data. If one fit uses weights or an offset, these should be consistent with the comparison being made. A quasi-likelihood fit, such as a quasi-Poisson model, does not supply the ordinary likelihood required for the standard likelihood-ratio test. Boundary parameters or other nonstandard conditions can also change the null distribution; the routine chi-square reference distribution should not be assumed in those cases.

Why the Statistic Has This Form

The likelihood ratio compares how well the two fitted models account for the same observed data. If \(L_0\) and \(L_1\) are the maximized likelihoods, then the ratio \(L_0/L_1\) is at most 1 because the larger model includes all parameter choices available to the smaller one. Taking minus twice its logarithm gives

-2\log\left(\frac{L_0}{L_1}\right) = -2(\log L_0-\log L_1) = 2(\ell_1-\ell_0).

Thus a larger value means the restricted model loses more likelihood relative to the larger model. Under the null hypothesis and regularity conditions, Wilks’ theorem gives the approximate chi-square reference distribution, with degrees of freedom reflecting the number of independent restrictions. This is why the test uses both a likelihood difference and a parameter-dimension difference; neither alone is enough.

Worked Example: Testing a Quadratic Age Effect

A hypothetical insurer models whether an applicant lapses during a year using logistic regression. Both models use the same 240 applicants and the same Bernoulli response likelihood. Model \(M_0\) has linear age as its only predictor. Model \(M_1\) adds squared age. Their maximized log-likelihoods are \(-128.2\) and \(-124.9\), respectively. At the 5% significance level, test whether the quadratic term improves fit.

Method and structure. \(M_0\) is nested in \(M_1\), because setting the coefficient of squared age to zero recovers \(M_0\). The models share the response, observations, and likelihood. Each has one intercept; \(M_0\) has one age coefficient and \(M_1\) has age and squared-age coefficients. The difference in parameter counts is \(3-2=1\).

Calculate the statistic. The log-likelihood improvement is \(-124.9-(-128.2)=3.3\). Therefore,

$$ G^2=2(3.3)=6.6. $$

Make the decision. Under the null hypothesis that the quadratic coefficient is zero, compare \(6.6\) with the 5% critical value from a chi-square distribution with 1 degree of freedom, approximately \(3.841\). Since \(6.6>3.841\), reject the null at the 5% level. The p-value is approximately \(0.0102\).

Check and trap avoided. The larger model has the higher log-likelihood, as it must when the nested models are maximized on the same data. The test uses the one additional parameter, not the total number of parameters in \(M_1\). The result is evidence about the quadratic term’s contribution to fit; it does not establish predictive superiority.

Non-Nested Models Need Different Evidence

When two candidates are non-nested, their log-likelihoods can still be compared as part of a criterion, but their difference does not have the usual chi-square distribution with degrees of freedom obtained by subtracting parameter counts. There is no set of restrictions that turns one candidate into the other, so the nested-model derivation does not apply.

AIC is one possible criterion when candidates are fitted by maximum likelihood to the same response observations and their likelihoods are comparable. For a model with \(k\) estimated parameters and maximized log-likelihood \(\ell\),

\mathrm{AIC}=-2\ell+2k.

A smaller AIC indicates a more favorable balance of fit and parameter count among the candidates being compared. AIC does not turn non-nested models into a likelihood-ratio test, and a small AIC difference is not a significance test. The earlier material on model comparison also emphasizes checking that the likelihoods refer to a common data set and scale.

Worked Example: Comparing Non-Nested Claim-Frequency Models

A hypothetical portfolio’s annual claim counts are modeled with Poisson GLMs on the same 500 policies. Model \(A\) uses vehicle age and a three-level territory factor. Model \(B\) uses vehicle age and annual mileage. Neither model can be obtained from the other by setting a subset of its coefficients to zero. Suppose their maximized log-likelihoods are \(-91.4\) and \(-89.8\), and they estimate \(k_A=4\) and \(k_B=5\) parameters, including intercepts.

Check the relationship. Both models use vehicle age, but \(A\) includes territory and \(B\) includes mileage. The territory effects cannot be removed from \(A\) to create the mileage effect in \(B\); nor can a restriction of \(B\) create territory effects. They are non-nested.

Reject the tempting test. The raw log-likelihood difference is \(-89.8-(-91.4)=1.6\). Doubling it gives \(3.2\), but treating \(3.2\) as a chi-square statistic with \(5-4=1\) degree of freedom is invalid. The degrees-of-freedom subtraction applies to a nested comparison, not to these alternatives.

Compare AIC instead. For \(A\), \(\mathrm{AIC}_A=-2(-91.4)+2(4)=182.8+8=190.8\). For \(B\), \(\mathrm{AIC}_B=-2(-89.8)+2(5)=179.6+10=189.6\). The difference is \(190.8-189.6=1.2\), so AIC favors \(B\), but only by a small amount.

Check. The calculations use each model’s own parameter count and the same 500 observations. The conclusion is a modest AIC preference, not a statistically significant finding from a likelihood-ratio test.

Same Response, Different Link: Still Not Necessarily Nested

Two GLMs can have the same response distribution and predictors but use different link functions. That does not automatically make one model a restricted version of the other. For instance, a logistic model and a probit model with the same predictors generally impose different relationships between the predictors and the response probability. Neither link is obtained from the other by setting a coefficient to zero.

If both are fitted by maximum likelihood to the same Bernoulli observations, AIC can provide a common criterion for comparison. But a standard nested-model likelihood-ratio test is not justified merely because the response and predictors match. Always examine the model families themselves, not just the variable lists.

Worked Example: Logistic Versus Probit Incidence Models

A hypothetical study records claim incidence for the same 300 insured units. A logistic model and a probit model both use an intercept and two predictors, and both use a Bernoulli likelihood. Their maximized log-likelihoods are \(-76.4\) for logistic and \(-75.9\) for probit. Each estimates \(k=3\) parameters. Decide what comparison is appropriate.

Identify the structure. The response observations and predictors are common, but the link functions differ. Neither model is obtained by imposing a coefficient restriction within the other, so they are non-nested candidates.

Calculate AIC. The logistic model has \(\mathrm{AIC}=-2(-76.4)+2(3)=152.8+6=158.8\). The probit model has \(\mathrm{AIC}=-2(-75.9)+2(3)=151.8+6=157.8\).

Interpret and check. AIC favors the probit fit by \(158.8-157.8=1.0\), a small difference. A likelihood-ratio test should not be formed by doubling the log-likelihood difference: the models are non-nested. The answer avoids confusing a shared response and predictor set with a nested model relationship.

A Practical Decision Sequence

Before calculating a test statistic or criterion, work through the comparison in a fixed order.

Check the response records.
Confirm that both fits use the same response observations and comparable likelihood definitions. If not, the direct likelihood comparison is not valid as stated.

Check model nesting.
Ask whether parameter restrictions on one model reproduce the other. A shared predictor or a larger parameter count does not establish nesting.

Choose the evidence.
For a regular nested comparison fitted by maximum likelihood, use \(2(\ell_1-\ell_0)\) and the difference in parameter dimensions. For non-nested candidates, use an appropriate comparable criterion such as AIC rather than a standard nested-model likelihood-ratio test.

State what the result means.
A test addresses the specified restrictions under its assumptions. An information criterion ranks candidates under its penalty. Neither conclusion should be overstated as proof of performance on new data.

Common Exam Traps

Using a likelihood-ratio test for non-nested models. Doubling the log-likelihood difference does not make the statistic chi-square; the nesting argument is essential.
Subtracting total parameter counts without checking restrictions. A parameter-count difference gives the usual degrees of freedom only when the models are nested and the restrictions are regular.
Comparing fits on different rows. A larger likelihood may reflect a different set of responses, not a better model for the same data. Refit on common observations when that is the intended comparison and is methodologically appropriate.
Assuming every GLM fit has an ordinary likelihood. Quasi-likelihood methods do not supply the ordinary likelihood needed for the standard test or ordinary AIC formula.
Equating a lower AIC with statistical significance. AIC is a penalized comparison criterion, not a p-value or a test of a coefficient.

Key takeaway: Use a likelihood-ratio test only when the smaller model is genuinely nested in the larger one, both are fitted by maximum likelihood to comparable common data, and the regularity conditions support the chi-square approximation. For non-nested alternatives, choose a comparison method suited to that structure, such as AIC when its comparability conditions are met.

Check Your Understanding

For each problem, identify the comparison structure before interpreting the calculation.

A Poisson model has maximized log-likelihood \(-54.0\) and 3 parameters. A nested model adds two freely estimated coefficients and has log-likelihood \(-51.5\), using the same observations and likelihood. Calculate the likelihood-ratio statistic and degrees of freedom. At the 5% level, is the improvement significant? (Answer: \(G^2=2[-51.5-(-54.0)]=5.0\), with 2 degrees of freedom; the 5% critical value is about 5.991, so do not reject. The p-value is about 0.0821.)
Two non-nested models use the same observations. Model X has \(\ell=-44.2\) and \(k=3\); Model Y has \(\ell=-42.8\) and \(k=4\). Calculate both AIC values and state the preference. (Answer: \(\mathrm{AIC}_X=88.4+6=94.4\); \(\mathrm{AIC}_Y=85.6+8=93.6\); AIC favors Y by 0.8, not as a significance test.)
A regular nested logistic model adds one coefficient. Its log-likelihood improves by 2.0, so the likelihood-ratio statistic is 4.0. At the 5% level, what is the conclusion? (Answer: With 1 degree of freedom, \(4.0>3.841\); reject the restriction at 5%, with p-value about 0.0455.)
Logistic and probit GLMs use the same binary responses and predictors, but neither link model is a restriction of the other. Is the difference in their maximized log-likelihoods a standard likelihood-ratio test? (Answer: No. They are non-nested; a comparable criterion such as AIC may be used if its likelihood and data conditions hold.)
Two otherwise nested GLMs report maximized log-likelihoods, but one fit used 180 records and the other used 160 because of missing predictor values. Can the reported values be used directly for the usual likelihood-ratio test? (Answer: No. The fits do not use the same response observations; refit both on a common appropriate set before making that direct comparison.)

What You'll Learn