Why Nested Deviance Differences Equal Likelihood-Ratio Statistics

Why Compare Deviances?

A generalized linear model comparison often asks whether a larger model fits sufficiently better than a smaller, nested model. The likelihood-ratio statistic measures that improvement on the log-likelihood scale. Deviance measures lack of fit relative to a saturated model, so it is natural to ask how the two quantities are connected. The connection is an exact algebraic identity, provided the models are fitted to the same response data under the same likelihood.

The key point is that each model's deviance is measured from the same saturated-model reference. Subtracting the smaller model's deviance from the larger model's deviance cancels that reference, leaving exactly twice the difference in maximized log likelihoods. This tutorial derives that statement and clarifies an important convention: in models with a non-unit dispersion parameter, the conventional unscaled deviance difference may need to be divided by dispersion to equal the likelihood-ratio statistic.

Likelihood-Ratio Statistic and Deviance

Definition: Let $M_0$ and $M_1$ be two models for the same observed response vector, with the same likelihood function and parameter space for the sampling distribution, and suppose $M_0$ is nested in $M_1$. Write $\ell_0$ and $\ell_1$ for their maximized log likelihoods. The likelihood-ratio statistic for comparing $M_0$ with $M_1$ is $$ G^2 = 2(\ell_1-\ell_0). $$ If a maximum is not attained, the corresponding supremum of the log likelihood is used.

“Nesting” means that every distribution allowed by $M_0$ is also allowed by $M_1$. Thus the larger model can reproduce every fit available to the smaller model, although it may fit better. A likelihood-ratio comparison is not defined by comparing two unrelated likelihood functions: the data, sampling model, and likelihood convention must agree.

Definition: Let $\ell_{\mathrm{sat}}$ be the maximized log likelihood in a saturated model, or its supremum if necessary. The exact likelihood deviance of a fitted model $M$ is $$ D_{\mathrm{lik}}(M)=2\{\ell_{\mathrm{sat}}-\ell_M\}, $$ where $\ell_M$ is the maximized log likelihood under $M$. The saturated model is the common reference that can fit the observed responses as freely as the sampling model permits.

In the usual GLM setting, deviance is also commonly expressed through an exponential-family formula. With dispersion $\phi$, some conventions call the quantity without a dispersion divisor the deviance $D(M)$; the likelihood deviance is then $D(M)/\phi$. When $\phi=1$, the two conventions agree. This distinction will be made explicit below.

The Exact Identity

Theorem: Suppose $M_0$ is nested in $M_1$, and both models use the same response observations, likelihood, and saturated-model reference. Then $$ G^2 = D_{\mathrm{lik}}(M_0)-D_{\mathrm{lik}}(M_1). $$ In particular, this difference is nonnegative.

The equality itself follows by cancellation; nesting supplies the nonnegativity. No large-sample approximation is involved in this identity.

Write both deviances.
Use the same saturated log likelihood in each expression: $D_{\mathrm{lik}}(M_j)=2(\ell_{\mathrm{sat}}-\ell_j)$, for $j=0,1$.

Subtract the larger-model deviance.
The shared term $2\ell_{\mathrm{sat}}$ cancels, leaving $2(\ell_1-\ell_0)$.

Use nesting for the sign.
Since every parameter value allowed by $M_0$ is available in $M_1$, maximization over $M_1$ cannot give a smaller log likelihood.

More explicitly, nesting implies $\ell_1\geq \ell_0$, so the statistic is nonnegative. The algebra is

\begin{aligned} D_{\mathrm{lik}}(M_0)-D_{\mathrm{lik}}(M_1) &=2(\ell_{\mathrm{sat}}-\ell_0)-2(\ell_{\mathrm{sat}}-\ell_1)\\ &=2(\ell_1-\ell_0)=G^2\geq 0. \end{aligned}

This proves the theorem. $\square$

The result is about maximized likelihoods, not about deviance values computed from arbitrary parameter estimates. If the fits are not maximized, subtracting their deviances still cancels the saturated reference, but it does not necessarily give the likelihood-ratio statistic for the models.

Dispersion and the Deviance Convention

For a common exponential-dispersion GLM representation, the observation-specific log likelihood has the form

\ell_i(\theta_i,\phi) =\frac{y_i\theta_i-b(\theta_i)}{a_i(\phi)}+c_i(y_i,\phi).

Here $\theta_i$ is the canonical parameter, $b$ is the cumulant function, and the functions $a_i$ and $c_i$ are fixed by the sampling distribution and the dispersion convention. For example, in a common convention $a_i(\phi)=\phi/w_i$, where $w_i$ is a known prior weight. If $\phi$ is held fixed and common to both models, the terms $c_i$ cancel in a likelihood difference. When $a_i(\phi)=\phi$, the log-likelihood difference is the corresponding unscaled deviance difference divided by $2\phi$.

Result: If the conventional unscaled deviance $D(M)$ is defined so that $$ \ell_{\mathrm{sat}}-\ell_M=\frac{D(M)}{2\phi} $$ for a common fixed dispersion $\phi>0$, then $$ G^2=\frac{D(M_0)-D(M_1)}{\phi}. $$ Thus the unscaled deviance difference equals the likelihood-ratio statistic when $\phi=1$, while the scaled deviance difference equals it for general fixed $\phi$.

This follows directly by applying the defining relation to each model and subtracting:

\begin{aligned} G^2 &=2(\ell_1-\ell_0)\\ &=2\left\{\frac{D(M_0)-D(M_1)}{2\phi}\right\}\\ &=\frac{D(M_0)-D(M_1)}{\phi}. \end{aligned}

The exact scaling depends on the likelihood and deviance definitions in use. For Bernoulli and Poisson GLMs, the usual dispersion is fixed at one, so the familiar deviance difference is the likelihood-ratio statistic. For a Gaussian model with known variance $\sigma^2$, the conventional residual deviance is a residual sum of squares, and its difference must be divided by $\sigma^2$ to obtain the likelihood-ratio statistic.

Worked Comparisons

Worked Example: Two Binomial Groups

Suppose two independent groups have binomial counts $Y_1=2$ of $n_1=10$ and $Y_2=6$ of $n_2=10$. Compare a smaller model with one common success probability $p$ to a larger model with separate probabilities $p_1,p_2$. The larger model is saturated for these two grouped observations.

Under the smaller model, the maximum-likelihood estimate is the pooled proportion: $$ \hat p=\frac{2+6}{10+10}=0.4. $$ The saturated estimates are $\hat p_1=2/10=0.2$ and $\hat p_2=6/10=0.6$. The binomial coefficients in the likelihood are identical under both models and cancel. The smaller-model deviance, relative to the saturated fit, is $$ \begin{aligned} D(M_0) &=2\left[ 2\ln\left(\frac{0.2}{0.4}\right) +8\ln\left(\frac{0.8}{0.6}\right) +6\ln\left(\frac{0.6}{0.4}\right) +4\ln\left(\frac{0.4}{0.6}\right) \right]\\ &=2[-1.386294+2.301457+2.432790-1.621860]\\ &\approx 3.4522. \end{aligned} $$ Each group-specific fitted proportion equals its observed proportion, so the larger model has deviance $D(M_1)=0$. Therefore $D(M_0)-D(M_1)\approx3.4522$.

The same value is obtained from the likelihood ratio. In the common log likelihood, the smaller model's log likelihood minus the saturated log likelihood is $$ 2\ln(0.4/0.2)+8\ln(0.6/0.8)+6\ln(0.4/0.6)+4\ln(0.6/0.4)\approx-1.7261. $$ Thus $2(\ell_1-\ell_0)\approx2(1.7261)=3.4522$, agreeing with the deviance difference.

Worked Example: Poisson Counts With Equal Exposure

Let $Y_1=2$ and $Y_2=8$ be independent Poisson counts, each with exposure one. Compare a common-mean model with mean $\mu$ against a larger model with separate means $\mu_1,\mu_2$. The maximum-likelihood estimates are $\hat\mu=5$ in the smaller model and $(\hat\mu_1,\hat\mu_2)=(2,8)$ in the saturated larger model.

For Poisson data, the deviance contribution from a count $y$ fitted with mean $\hat\mu$ is $2\{y\ln(y/\hat\mu)-(y-\hat\mu)\}$. The two counts sum to 10, as do their fitted means in the smaller model, so the linear terms cancel. Hence $$ \begin{aligned} D(M_0) &=2\left[2\ln(2/5)+8\ln(8/5)\right]\\ &=2[-1.832581+3.760604]\\ &\approx3.8560. \end{aligned} $$ The larger model fits each count exactly, so $D(M_1)=0$, and the deviance difference is approximately $3.8560$.

To verify the likelihood-ratio calculation, the Poisson log likelihood is $\ell=\sum_i\{y_i\ln(\mu_i)-\mu_i-\ln(y_i!)\}$. The factorial terms cancel between fits, and the linear mean terms sum to 10 under both fits. Therefore $$ \ell_1-\ell_0 =2\ln(2/5)+8\ln(8/5) \approx1.9280, \qquad G^2=2(1.9280)\approx3.8560. $$

Worked Example: Gaussian Model With Known Variance

Consider four independent observations $1,3,5,7$ from normal distributions with known variance $\sigma^2=4$. The smaller model gives all four observations a common mean. The larger model gives one mean to the first pair and another to the second pair.

The common-mean estimate is $\bar y=4$, giving residual sum of squares $$ (1-4)^2+(3-4)^2+(5-4)^2+(7-4)^2=9+1+1+9=20. $$ The two group means in the larger model are $2$ and $6$, giving residual sum of squares $$ (1-2)^2+(3-2)^2+(5-6)^2+(7-6)^2=1+1+1+1=4. $$ For known variance $\sigma^2$, the Gaussian log likelihood differs between these fits by minus the residual-sum-of-squares difference divided by $2\sigma^2$. Consequently, $$ G^2=\frac{20-4}{4}=4. $$ The conventional unscaled deviance difference is $20-4=16$, not $4$. Dividing by the known dispersion $4$ gives the likelihood-ratio statistic, as required.

Conditions, Interpretation, and a Common Pitfall

The identity is useful because a deviance table can express a likelihood-ratio comparison using two fitted models' lack of fit. But the cancellation works only with a common reference. Both models must use the same response data and likelihood, and the saturated-model likelihood must be the same. Changing the response subset, likelihood family, weights, or fixed dispersion can change the likelihood being compared; a numerical subtraction of reported deviances then need not be the likelihood-ratio statistic.

A further distinction is between an exact identity and a reference distribution. The difference equals the likelihood-ratio statistic exactly under the stated conditions. Deciding how to calibrate that statistic for a test, including its degrees of freedom or finite-sample behavior, is a separate question.

Common mistake: “The deviance difference is always the likelihood-ratio statistic, without qualification.”
Why it fails: In a Gaussian model with known variance 4, the example above has unscaled deviance difference 16 but likelihood-ratio statistic 4.
Correction: First check the deviance convention. The exact likelihood deviance difference equals the likelihood-ratio statistic; an unscaled deviance difference in a fixed-dispersion model may require division by dispersion.

Key takeaway: With a shared likelihood and saturated reference, subtracting the larger model's deviance from the smaller model's deviance cancels the saturated term and gives $2(\ell_1-\ell_0)$. Verify the dispersion scaling before treating a reported deviance difference as the likelihood-ratio statistic.

Check Your Understanding

Use the shared-reference identity and the dispersion convention to answer these questions.

State the exact likelihood-deviance identity for nested models $M_0$ and $M_1$, using the notation in this tutorial.
Why must the models use the same observations and saturated-model reference for the deviance subtraction to cancel correctly?
A fixed-dispersion model has conventional unscaled deviances $D(M_0)=31$ and $D(M_1)=23$, with $\phi=2$. What is the likelihood-ratio statistic under the stated convention?
In a Gaussian model with known variance 9, the smaller model has residual sum of squares 50 and the larger model has residual sum of squares 32. Find the deviance difference and the likelihood-ratio statistic.
Does the exact deviance identity by itself determine the reference distribution for a hypothesis test? Explain briefly.