Deviance Differences for Nested GLMs

Why Compare Nested GLMs by Deviance?

A larger generalized linear model (GLM) can often fit the observed data at least as well as a smaller one, because it has more freedom to represent patterns in the response. The important question is whether that improved fit is substantial relative to the number of additional parameters. For two suitably fitted nested GLMs, the difference in their deviances provides a likelihood-based measure of that improvement.

A deviance difference is not simply a score for choosing whichever model has the smaller number. It quantifies the improvement in fit when moving from a restricted model to a larger model, and can be compared with a chi-square reference distribution under specific assumptions. The calculation is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: Two GLMs are nested when the smaller, or reduced, model can be obtained from the larger, or full, model by imposing restrictions on its parameters. For example, a model with predictors \(x_1\) and \(x_2\) is nested within one with predictors \(x_1\), \(x_2\), and \(x_3\): the reduced model is obtained by setting the coefficient of \(x_3\) to zero.

For a model fitted by maximum likelihood, let \(\ell\) denote its maximized log-likelihood and let \(\ell_{\text{sat}}\) denote the log-likelihood of the saturated model, which fits the observed response as closely as the model family permits. The deviance is twice the log-likelihood gap between these models.

D = 2(\ell_{\text{sat}}-\ell)

The saturated-model term is the same when both models use the same responses, observations, response distribution, and likelihood convention. This makes it cancel when the two deviances are subtracted.

The Deviance-Difference Test

Write \(D_R\) for the reduced model's deviance and \(D_F\) for the full model's deviance. The test statistic is the reduced model's deviance minus the full model's deviance. Its degrees of freedom are the number of parameters in the full model minus the number in the reduced model, counting only parameters that differ between the models.

G^2 = D_R-D_F = 2(\ell_F-\ell_R), \qquad \nu = p_F-p_R

Here, \(p_R\) and \(p_F\) are the numbers of freely estimated model parameters in the reduced and full models. The comparison tests the null hypothesis that the restrictions defining the reduced model are correct. Under the null, and under regularity conditions, \(G^2\) is approximately distributed as a chi-square random variable with \(\nu\) degrees of freedom. A small upper-tail probability is evidence that the additional parameters improve fit beyond what would usually be expected from random variation under the reduced model.

Conditions for the standard test: The models must be nested and fitted by maximum likelihood to the same response observations under the same response distribution and likelihood convention. The usual chi-square approximation also relies on regularity conditions, including identifiable parameters and a sufficiently large sample. For standard Poisson and binomial GLMs, this is the familiar likelihood-ratio comparison. If the fitting method does not provide a valid likelihood, or overdispersion makes the assumed variance model inappropriate, the ordinary chi-square calibration may not be justified.

A negative computed difference is a warning, not evidence that the larger model is worse in a valid nested maximum-likelihood comparison. Since the full model includes every parameter setting available to the reduced model, its maximized likelihood cannot be lower. Thus \(D_F\) cannot exceed \(D_R\), apart from minor numerical discrepancies in software output. A larger apparent violation suggests a mismatch in data, model specification, fitting method, or reported values.

Why the Formula Works

The identity between the deviance difference and the likelihood-ratio statistic follows directly from the definition of deviance. Apply that definition to both models:

\begin{aligned} D_R-D_F &=2(\ell_{\text{sat}}-\ell_R)-2(\ell_{\text{sat}}-\ell_F)\\ &=2\ell_{\text{sat}}-2\ell_R-2\ell_{\text{sat}}+2\ell_F\\ &=2(\ell_F-\ell_R). \end{aligned}

The saturated-model terms cancel because the comparison uses the same observations and response likelihood. To see why the difference is nonnegative, note that the full model contains the reduced model: every reduced-model parameter setting is available in the full model by satisfying the relevant restrictions. Maximizing over the larger set cannot produce a lower likelihood, so \(\ell_F \geq \ell_R\), and therefore \(2(\ell_F-\ell_R)\geq 0\). The chi-square reference distribution is an additional large-sample result; it does not follow from the algebra alone.

Confirm comparability.
Check that the observations, response distribution, likelihood convention, and fitting method agree between the models.

Confirm nesting.
Identify the restrictions that turn the full model into the reduced model.

Calculate the statistic and degrees of freedom.
Compute \(D_R-D_F\) and \(p_F-p_R\).

Interpret the evidence.
Compare the statistic with a chi-square distribution having the calculated degrees of freedom, if the test assumptions are reasonable.

Worked Comparisons

Worked Example: Testing Two Additional Poisson Predictors

A hypothetical insurer models claim counts for the same 80 policy groups using Poisson GLMs with a log link. The reduced model contains an intercept and one rating variable; the full model adds two predictors. Both models use the same observations and exposure treatment, and are fitted by maximum likelihood. The reduced model has deviance 31.6 on 77 residual degrees of freedom; the full model has deviance 23.2 on 75 residual degrees of freedom. Test whether the additional predictors improve fit at the 5% level.

Method. Use the likelihood-ratio test based on the difference in residual deviances. The two models are nested because the reduced model is recovered by setting both added coefficients to zero.

Step 1: Calculate the statistic. \(G^2=31.6-23.2=8.4\).

Step 2: Calculate the degrees of freedom. The residual degrees of freedom fall from 77 to 75, so the full model estimates two additional parameters: \(\nu=77-75=2\).

Step 3: Find the upper-tail probability. For a chi-square distribution with 2 degrees of freedom, the upper-tail probability at 8.4 is \(P(\chi^2_2\geq 8.4)=e^{-8.4/2}=e^{-4.2}\approx 0.0150\).

Conclusion and check. Since \(0.0150<0.05\), there is evidence against the restrictions that both added coefficients are zero. The full model's deviance is lower, and \(8.4\) is positive, as required for this nested maximum-likelihood comparison. The common mistake avoided here is using 8.4 degrees of freedom; the statistic is 8.4, but the degrees of freedom are the two additional parameters.

Worked Example: One Added Term in a Logistic GLM

A hypothetical benefits program models whether an application is approved using a binomial logistic GLM. Both models use the same 120 application records. A reduced model includes an intercept and applicant age; a full model adds a binary indicator for whether documents were submitted online. The residual deviances are 42.0 and 38.5, respectively, with residual degrees of freedom 118 and 117. At the 5% level, is the additional term supported by this comparison?

Method. Compare the nested models with the deviance-difference likelihood-ratio test. The full model can reproduce the reduced model by setting the online-document coefficient to zero.

Step 1: Calculate the statistic and degrees of freedom. \(G^2=42.0-38.5=3.5\), and \(\nu=118-117=1\).

Step 2: Calculate the approximate p-value. For one degree of freedom, the chi-square upper-tail probability at 3.5 is approximately \(0.0614\).

Step 3: Interpret the result. Because \(0.0614>0.05\), the comparison does not provide sufficient evidence at the 5% level to reject the restriction that the added coefficient is zero. The full model fits the observed data better, but this deviance reduction is not statistically significant at the stated threshold.

Conclusion and check. The statistic is positive, and the reduction of one residual degree of freedom agrees with one additional parameter. The common mistake avoided is treating any decrease in deviance as conclusive evidence for keeping a term; the size of the decrease must be judged relative to its degrees of freedom.

Worked Example: A Three-Parameter Factor Addition

A hypothetical Poisson model describes event counts in 50 independent service regions. Both models use the same count data and a log link. The reduced model estimates 10 parameters and has deviance 64.8; the full model adds a four-level factor, estimates 13 parameters, and has deviance 55.6. Assume the model fits are maximum-likelihood fits and the usual chi-square approximation is reasonable. Test the factor as a whole at the 5% level.

Method. A four-level factor represented with an intercept reference category contributes three additional coefficients. The null hypothesis sets all three coefficients to zero, so the comparison tests the factor jointly.

Step 1: Calculate the deviance difference. \(G^2=64.8-55.6=9.2\).

Step 2: Calculate degrees of freedom. \(\nu=13-10=3\). Equivalently, the factor has four levels but adds three parameters because one level is the reference.

Step 3: Calculate the p-value. For 3 degrees of freedom, the chi-square upper tail at 9.2 is \(e^{-9.2/2}(1+9.2/2)=e^{-4.6}(5.6)\approx 0.0563\).

Conclusion and check. Since \(0.0563>0.05\), the factor does not meet the 5% threshold in this comparison. The deviance falls by 9.2 while the model gains three parameters; the degrees of freedom are three, not four. The common mistake avoided is counting factor levels rather than the number of independent coefficients added.

Interpretation and Common Traps

The deviance difference measures how much closer the full model's maximized likelihood is to the saturated model's likelihood than the reduced model's is. It is a measure of improvement in in-sample fit, assessed relative to the number of restrictions being tested. It does not, by itself, establish that the full model predicts better on new data or that the added terms have a practically important effect.

The comparison is a joint test of all restrictions needed to obtain the reduced model. For instance, a factor with several added coefficients is assessed as a group. A small p-value indicates evidence that at least one restriction is inconsistent with the data; it does not identify which coefficient is responsible.

A deviance difference should not be given the ordinary chi-square interpretation automatically. In particular, a quasi-likelihood fit does not supply the ordinary full likelihood required for the likelihood-ratio derivation. Likewise, if overdispersion indicates that the assumed response variation is too small, the usual reference distribution may give misleading evidence. The model and error assumptions must be credible before interpreting the p-value.

Common exam trap: Subtract in the wrong order, then report a negative test statistic as if it were meaningful. If \(D_R=18.4\) and \(D_F=21.1\), the calculation \(18.4-21.1=-2.7\) is a warning. It is not a valid likelihood-ratio statistic for nested maximum-likelihood fits on the same data: the full model cannot have a worse maximized likelihood. Check whether the model labels were reversed, the data or weights differ, or the fits failed. When the comparison is valid, use \(D_R-D_F\), which is nonnegative apart from rounding or numerical error.

Use the correct degrees of freedom. Count the difference in freely estimated parameters, not the number of predictors as loosely described. A factor can contribute several parameters.
Keep the data basis fixed. Deviances from fits on different subsets do not have the cancellation used in the derivation.
Distinguish a smaller deviance from compelling evidence. The size of the reduction must be considered with its degrees of freedom and the test assumptions.
State the scope of the conclusion. The test evaluates the added restrictions in the fitted likelihood model; it is not, by itself, a measure of predictive performance or practical value.

Key takeaway: For nested maximum-likelihood GLMs fitted comparably, subtract the full-model deviance from the reduced-model deviance and use the added parameter count as the chi-square degrees of freedom. Interpret the result only when the likelihood and large-sample assumptions are appropriate.

Check Your Understanding

For each problem, identify the deviance-difference statistic and its degrees of freedom before interpreting the result.

A hypothetical Poisson GLM comparison on the same 60 regions gives reduced deviance 24.7 and full deviance 19.1. The full model adds two parameters. Calculate the statistic and degrees of freedom. (Answer: \(G^2=5.6\), \(\nu=2\).)
Two nested binomial logistic GLMs use the same applications. The reduced deviance is 16.0 and the full deviance is 12.0; one parameter is added. The chi-square upper-tail probability is approximately 0.0455. At the 5% level, what is the conclusion? (Answer: \(G^2=4.0\), \(\nu=1\); reject the reduced-model restriction at 5%.)
A Poisson model has deviance 40.2 and 52 residual degrees of freedom; its nested expansion has deviance 34.4 and 49 residual degrees of freedom. Calculate the test statistic and degrees of freedom, and state the comparison's p-value approximately. (Answer: \(G^2=5.8\), \(\nu=3\), \(p\approx0.122\).)
A full maximum-likelihood model contains the reduced model's terms and adds two coefficients, but the reported deviances are 27.6 for the reduced model and 28.0 for the full model. What should be checked before interpreting a test? (Answer: The reported difference is \(-0.4\), which is not valid for comparable nested maximum-likelihood fits; check data, model labels, fitting convergence, and output.)
A four-level factor is added to a logistic GLM using reference coding. The reduced and full deviances are 51.3 and 43.9. How many degrees of freedom does the factor add, and what is the statistic? (Answer: \(\nu=3\), \(G^2=7.4\).)

What You'll Learn