Deviance Differences and Nested Model Comparisons

Why Compare Deviances?

A generalized linear model can be judged by how much likelihood it leaves unexplained relative to a saturated model, which fits each observation as closely as the model permits. But for a scientific question, the more useful comparison is often between two models: does adding a specified set of parameters improve the fit enough to justify the added complexity? For nested models, the difference in their residual deviances answers this question through a likelihood-ratio statistic.

The algebraic identity is exact when both deviances are defined from the same likelihood and data. A chi-square reference distribution is a separate, asymptotic claim: it requires a true null model and regularity conditions. Keeping those two claims distinct prevents a common error—treating every numerical deviance difference as automatically chi-square distributed.

Definitions and the Exact Identity

Definition: Let \(M_R\) be a reduced model nested in a full model \(M_F\), meaning every distribution allowed by \(M_R\) is also allowed by \(M_F\). Write \(\ell_R\) and \(\ell_F\) for the maximized log-likelihoods under the two models, based on the same observed data. If \(\ell_{\mathrm{sat}}\) is the maximized log-likelihood under a saturated model for those data, the residual deviance of model \(M\) is \(D_M=2(\ell_{\mathrm{sat}}-\ell_M)\). The likelihood-ratio statistic for testing \(M_R\) against \(M_F\) is \(G^2=2(\ell_F-\ell_R)\).

The fitted full model cannot have a lower maximized likelihood than the reduced model: the reduced model's possible parameter values are included in the full model. Thus \(G^2\geq 0\). The deviance difference has the same direction: subtract the full-model residual deviance from the reduced-model residual deviance.

Theorem: For nested models fitted to the same observations, with residual deviances defined from the same saturated likelihood, \(D_R-D_F=G^2\).

Proof. Substitute the definition of each residual deviance and cancel the common saturated log-likelihood:

D_R-D_F =2(\ell_{\mathrm{sat}}-\ell_R)-2(\ell_{\mathrm{sat}}-\ell_F) =2(\ell_F-\ell_R) =G^2.

This identity does not depend on a large-sample approximation. It follows directly from the definitions. Its practical use does depend on using a common likelihood: changing the observations, response distribution, or treatment of nuisance parameters can invalidate the cancellation. \(\square\)

In a fixed-dispersion exponential-family model, terminology needs care. If \(D_M\) means twice the difference in the full log-likelihoods, the identity above applies directly. Some conventions instead report an unscaled deviance \(D_M^{(0)}\), with the common dispersion factor omitted. If the dispersion \(\phi\) is fixed and common to both models, then \(G^2=(D_R^{(0)}-D_F^{(0)})/\phi\). For Poisson and binomial models, the usual dispersion is one, so this distinction does not change the statistic. A comparison with estimated dispersion requires additional care and is not covered by this fixed-dispersion identity in its unscaled form.

Worked Comparisons

Worked Example: Two Nested Poisson Mean Models

Suppose four independent Poisson counts are observed, with group labels \(A,A,B,B\), and no offsets. The reduced model has one common mean; the full model has a separate mean for each group. The counts are \(1,3,2,6\). Find the likelihood-ratio statistic and verify it by subtracting residual deviances.

The reduced-model estimate is the overall mean, \((1+3+2+6)/4=3\). The full-model estimates are the group means: \(\hat\mu_A=(1+3)/2=2\) and \(\hat\mu_B=(2+6)/2=4\). For independent Poisson observations, the log-likelihood is \(\ell=\sum_i[y_i\log(\mu_i)-\mu_i-\log(y_i!)]\). The factorial terms cancel in the likelihood difference. The fitted means sum to 12 under both models, so the \(-\mu_i\) terms also cancel. Therefore

G^2=2\{4\log 2+8\log 4-12\log 3\} \approx 1.359.

For Poisson observations, the residual deviance at fitted means \(\hat\mu_i\) is \(2\sum_i[y_i\log(y_i/\hat\mu_i)-(y_i-\hat\mu_i)]\), with the convention \(0\log 0=0\). In the full model, the \((y_i-\hat\mu_i)\) terms sum to zero within each group, giving

D_F=2\{1\log(1/2)+3\log(3/2)+2\log(2/4)+6\log(6/4)\} \approx 3.139.

For the reduced model, the fitted mean is 3 for all four observations, and the residual terms again sum to zero overall:

D_R=2\{1\log(1/3)+3\log(3/3)+2\log(2/3)+6\log(6/3)\} \approx 4.499.

Thus \(D_R-D_F\approx 4.499-3.139=1.360\), agreeing with the likelihood calculation up to rounding. The full model adds one parameter, so the usual large-sample comparison uses one degree of freedom if the regularity conditions hold.

Worked Example: Comparing Two Binomial Models

Two groups each contain 10 individuals. The numbers of successes are 2 in group 1 and 8 in group 2. Compare a model with one common success probability to a model with a separate probability for each group.

The common-probability estimate is \(10/20=0.5\). The separate estimates are \(0.2\) and \(0.8\). The binomial coefficients are identical under the two fits and cancel. The log-likelihood difference is

\ell_F-\ell_R =4\log(0.2)+16\log(0.8)-20\log(0.5) \approx 3.855.

Hence \(G^2\approx 2(3.855)=7.710\). The separate-probability model fits each group exactly, so its residual deviance is zero. The reduced-model residual deviance is

D_R =2\left\{2\log(0.2/0.5)+8\log(0.8/0.5) +8\log(0.8/0.5)+2\log(0.2/0.5)\right\} \approx 7.710.

Therefore \(D_R-D_F\approx 7.710\), as required. The full model adds one free probability parameter, so the difference in model dimensions is one. A chi-square approximation may be used only if its regularity conditions are reasonable; the identity itself does not assert that this small example has an exact chi-square distribution.

Worked Example: The Dispersion Scaling in a Normal Model

For independent normal observations with known variance \(\phi=2\), suppose nested fits have residual sums of squares 20 for the reduced model and 14 for the full model. Here the unscaled residual deviance is the residual sum of squares. The log-likelihood contains the term \(-\mathrm{RSS}/(2\phi)\), so

G^2=\frac{20-14}{2}=3.

The unscaled deviance difference is 6, not 3. Dividing by the common known dispersion gives the likelihood-ratio statistic. If the full model adds one identifiable coefficient and the null is true, the regular large-sample reference is \(\chi^2_1\); the calculation alone does not establish that approximation.

When Is the Chi-Square Approximation Justified?

Wilks's theorem gives the standard reference distribution. In its usual finite-dimensional form, suppose independent observations follow a correctly specified, regular parametric model with parameter \(\theta\in\mathbb{R}^p\). The null model is a smooth submodel of codimension \(q\), the true parameter is an interior point of that submodel, and the model is identifiable with nonsingular Fisher information. Assume the log-likelihood is sufficiently differentiable, the model satisfies local asymptotic normality, and the restricted and unrestricted maximum-likelihood estimators exist, are consistent, and are root-\(n\) close to the true parameter. With fixed \(p\) and \(q\), these conditions yield

G^2=2(\ell_F-\ell_R)\xrightarrow{d}\chi^2_q \qquad\text{under the null.}

Here \(q\) is the difference in the dimensions of the full and reduced parameter spaces, not the number of observations and not the numerical difference in residual degrees of freedom unless those quantities happen to agree. In a generalized linear model with regular design and an interior true parameter, \(q\) is commonly the number of additional regression coefficients. The result concerns a fixed-dimensional, large-sample limit; it is not an exact finite-sample distribution.

Derivation of Wilks's limit: Use smooth local coordinates \((\alpha,\beta)\), where the null is \(\beta=0\), \(\alpha\in\mathbb{R}^{p-q}\), and \(\beta\in\mathbb{R}^{q}\). Let \(I\) be the per-observation Fisher information at the null parameter, partitioned into \(\alpha\) and \(\beta\) blocks. Local asymptotic normality gives, for bounded \(h\),

\ell(\theta_0+h/\sqrt n)-\ell(\theta_0) =h^\mathsf{T}\Delta_n-\tfrac12 h^\mathsf{T}Ih+o_p(1), \qquad \Delta_n\xrightarrow{d}N(0,I).

The regularity assumptions ensure that the maximizing local parameters are root-\(n\) bounded and that this quadratic approximation applies to both maximizations. Write the blocks as \(I_{\alpha\alpha}, I_{\alpha\beta}, I_{\beta\alpha}, I_{\beta\beta}\), and define the efficient information for \(\beta\) by

I_{\beta\cdot\alpha} =I_{\beta\beta}-I_{\beta\alpha}I_{\alpha\alpha}^{-1}I_{\alpha\beta}.

For a fixed \(h_\beta\), maximizing the quadratic expression over \(h_\alpha\) gives \(h_\alpha=I_{\alpha\alpha}^{-1}(\Delta_\alpha-I_{\alpha\beta}h_\beta)\). Substitution leaves, apart from a term common to both models, the quadratic expression \(h_\beta^\mathsf{T}W_n-\tfrac12h_\beta^\mathsf{T}I_{\beta\cdot\alpha}h_\beta\), where \(W_n=\Delta_\beta-I_{\beta\alpha}I_{\alpha\alpha}^{-1}\Delta_\alpha\). Under the null, \(W_n\) converges to a mean-zero normal vector with covariance \(I_{\beta\cdot\alpha}\). The reduced model fixes \(h_\beta=0\); the full model maximizes over \(h_\beta\). Its maximizer is \(I_{\beta\cdot\alpha}^{-1}W_n\), so twice the difference in the two maximized quadratic expressions is

W_n^\mathsf{T}I_{\beta\cdot\alpha}^{-1}W_n+o_p(1).

A quadratic form of this kind converges to the sum of squares of \(q\) independent standard normal variables, which has a \(\chi^2_q\) distribution. This establishes the stated limit under the assumptions, rather than merely appealing to the name of the theorem. \(\square\)

The approximation can fail when a null parameter lies on the boundary, a parameter is unidentified under the null, information is singular, or the model dimension grows with sample size. It also requires the null hypothesis to be true for the stated central chi-square limit. Under alternatives, the limiting behavior is different. In particular, the exact deviance identity remains an algebraic fact even when a chi-square calibration is unsuitable.

Common Mistakes and Practical Checks

Reversing the subtraction. The comparison is \(D_R-D_F\), because the reduced model generally has the larger residual deviance. Reversing it produces a nonpositive quantity, not the likelihood-ratio statistic.
Assuming the identity proves the reference distribution. The identity is exact; the chi-square result is asymptotic and needs the null and regularity conditions stated above.
Using the wrong degrees of freedom. Use the difference in parameter-space dimensions for regular nested models. Do not substitute the sample size or infer the answer from the size of a deviance value.
Comparing incompatible fits. Both deviances must use the same observations and likelihood convention. If observations are missing differently across models, the saturated likelihoods need not cancel.
Ignoring dispersion scaling. For a common known dispersion other than one, an unscaled deviance difference must be divided by that dispersion to obtain the likelihood-ratio statistic.

A sound comparison therefore has two stages: first verify that the models are genuinely nested and compute the likelihood-ratio statistic from compatible fits; then assess whether the sampling regime and parameter values justify the chi-square approximation. These checks make the deviance difference an interpretable test of the added model terms, rather than just a subtraction of fit summaries.

Check Your Understanding

Use the definitions and conditions above to check your reasoning.

Why does the saturated log-likelihood cancel when two residual deviances from nested models are subtracted?
A full model has three more free parameters than its reduced model. What is the usual Wilks reference distribution under the null, and what conditions must still be checked?
If an unscaled deviance difference is 8 and the common known dispersion is 2, what is the likelihood-ratio statistic?
Does the exact identity \(D_R-D_F=G^2\) imply that the statistic has an exact chi-square distribution in a finite sample? Explain.
Name two circumstances in which the usual chi-square approximation may fail even though the models are nested.