Residual Deviance and Its Reference Distribution

Why Residual Deviance Needs a Reference Distribution

Residual deviance measures how much log likelihood is lost when a fitted model is used instead of a saturated model. A large value can indicate that the fitted model represents the data poorly, but deciding whether it is “large” requires a reference distribution. The familiar chi-square calibration is useful, but it is not automatic: its validity depends on how the data and model dimensions behave as information increases.

The central distinction is between a fixed number of response groups, each accumulating information, and a saturated model whose dimension grows with the sample size. In the first setting, regular likelihood theory can yield a chi-square limit. In the second, the usual fixed-dimensional likelihood argument does not apply. Sparse expected counts can also make the asymptotic approximation inaccurate even when a chi-square reference is justified in principle.

Definition: Let $\ell_{\mathrm{sat}}$ be the maximized log likelihood in the saturated model, and let $\ell(\hat{\theta})$ be the maximized log likelihood in the model being assessed. The residual deviance is $$ D=2\{\ell_{\mathrm{sat}}-\ell(\hat{\theta})\}. $$ It is twice the log-likelihood ratio statistic comparing the saturated model with the fitted model. Since the saturated model includes the fitted model, $D\geq 0$.

For independent Poisson counts $Y_j$ with fitted means $\hat{\mu}_j$, the contribution from cell $j$ is $$ D_j=2\left\{Y_j\log\left(\frac{Y_j}{\hat{\mu}_j}\right)-(Y_j-\hat{\mu}_j)\right\}, $$ where $0\log(0/\hat{\mu}_j)$ is defined to be zero. For grouped binomial observations $Y_j$ out of $n_j$, with fitted probabilities $\hat{\pi}_j$, it is $$ D_j=2\left\{Y_j\log\left(\frac{Y_j}{n_j\hat{\pi}_j}\right) +(n_j-Y_j)\log\left(\frac{n_j-Y_j}{n_j(1-\hat{\pi}_j)}\right)\right\}, $$ with zero-count terms interpreted by continuity. The total deviance is the sum of the contributions. These formulas are likelihood comparisons; neither formula, by itself, establishes a chi-square reference distribution.

When the Chi-Square Reference Is Justified

The relevant classical result is Wilks’ theorem. In its fixed-dimensional form, suppose the full model has a parameter $\eta$ in an open subset of $\mathbb{R}^d$, and the null model is a smooth, identifiable submodel of dimension $p<d$. Under the null, the true parameter must be an interior point of that submodel. The likelihood must be regular: in particular, the information is nonsingular, the maximum-likelihood estimates are consistent and locally well behaved, and the log likelihood has a uniform local quadratic expansion at the information scale. The dimensions $d$ and $p$ are held fixed as information increases.

Theorem (Wilks’ theorem): Under the null and the regularity conditions just stated, the likelihood-ratio statistic comparing the full model with the $p$-dimensional submodel satisfies $$ 2\{\ell_{\mathrm{full}}-\ell_{\mathrm{null}}\}\xrightarrow{d}\chi^2_{d-p}. $$ For residual deviance, the full model is the saturated model. If it has fixed dimension $d$ and the fitted model has dimension $p$, the asymptotic reference is therefore $\chi^2_{d-p}$.

Here is the quadratic argument behind the result. In local coordinates standardized by the information, write $Z_n$ for the normalized score. The local likelihood has the form $$ \ell_n(h)-\ell_n(0)=h^{\mathsf T}Z_n-\frac{1}{2}h^{\mathsf T}h+o_p(1), $$ uniformly over bounded local displacements $h$, with $Z_n\xrightarrow{d}N_d(0,I_d)$. The local tangent space of the null model is a $p$-dimensional subspace $S$ of $\mathbb{R}^d$. Regularity ensures the relevant maximizing displacements are local, so this expansion can be used to compare the two maximized likelihoods.

Proof. The quadratic part of the local log likelihood can be written as $$ h^{\mathsf T}Z_n-\frac{1}{2}h^{\mathsf T}h =\frac{1}{2}\|Z_n\|^2-\frac{1}{2}\|h-Z_n\|^2. $$ Over the full space, its maximum is $\frac{1}{2}\|Z_n\|^2$, attained at $h=Z_n$. Over $S$, the maximum is $\frac{1}{2}\|P_S Z_n\|^2$, attained at the orthogonal projection $P_S Z_n$. The likelihood-ratio statistic is twice the difference between these maxima, up to $o_p(1)$, and hence equals $$ \|Z_n\|^2-\|P_S Z_n\|^2+o_p(1) =\|(I_d-P_S)Z_n\|^2+o_p(1). $$ The matrix $I_d-P_S$ is an orthogonal projection of rank $d-p$. For $Z\sim N_d(0,I_d)$, the squared length of its projection onto a $(d-p)$-dimensional subspace has a $\chi^2_{d-p}$ distribution. Since $Z_n\xrightarrow{d}Z$, the continuous mapping theorem and the $o_p(1)$ remainder give the stated limit. $\square$

For a fixed number $m$ of independent response groups, if the saturated model has one free parameter per group and the fitted model has rank $p$, this gives the familiar residual-deviance reference $\chi^2_{m-p}$. This statement presumes regularity and increasing information in the groups; simply counting observations and fitted coefficients is not enough.

Why Deviance Resembles Pearson’s Statistic at Large Counts

A second useful fact explains why deviance and Pearson goodness-of-fit statistics often give similar results when expected counts are large. For a Poisson observation $Y$ and a fixed positive mean $\mu$, the deviance contribution is $$ D(Y,\mu)=2\left\{Y\log\left(\frac{Y}{\mu}\right)-(Y-\mu)\right\}. $$ If $Y=\mu+\delta$, set $x=\delta/\mu$. For $x$ near zero, Taylor expansion gives $(1+x)\log(1+x)-x=x^2/2+O(|x|^3)$. Substitution yields $$ D(Y,\mu)=\frac{\delta^2}{\mu}+O\left(\frac{|\delta|^3}{\mu^2}\right). $$ Thus the leading term is the Pearson contribution $(Y-\mu)^2/\mu$.

Proof of the expansion. For $x>-1$, define $g(x)=(1+x)\log(1+x)-x$. Then $g(0)=g'(0)=0$, $g''(0)=1$, and $g$ has a bounded third derivative on any closed interval around zero contained in $(-1,\infty)$. Taylor’s theorem therefore gives $g(x)=x^2/2+O(|x|^3)$ as $x\to0$. Since $Y=\mu(1+x)$, the deviance is $2\mu g(x)$, so $$ D(Y,\mu)=\mu x^2+O(\mu|x|^3) =\frac{\delta^2}{\mu}+O\left(\frac{|\delta|^3}{\mu^2}\right). $$ When $\mu$ grows and $\delta=O_p(\sqrt{\mu})$, the remainder is $O_p(\mu^{-1/2})$. This is a cellwise approximation; the chi-square result additionally requires regularity for the joint model and any estimated parameters. $\square$

Worked Examples

Worked Example: Equal-Proportion Multinomial Fit

A sample of 100 observations falls into four categories with counts $18,22,27,33$. Under the model, the four category probabilities are equal. The saturated multinomial model has dimension $4-1=3$, while the equal-proportion model has dimension zero because its probabilities are specified. Calculate the deviance and identify its asymptotic reference.

Under the model, each expected count is $100/4=25$. The multinomial deviance is $$ D=2\sum_{j=1}^4 Y_j\log\left(\frac{Y_j}{25}\right). $$ The four terms inside the sum are $$ 18\log(18/25)\approx-5.9131,\quad 22\log(22/25)\approx-2.8123, $$ $$ 27\log(27/25)\approx2.0779,\quad 33\log(33/25)\approx9.1619. $$ Their sum is approximately $2.5144$, so $D\approx2(2.5144)=5.0288$, or $5.029$ to three decimal places. With four fixed categories and a zero-dimensional null model, Wilks’ theorem gives the asymptotic reference $\chi^2_3$. The observed statistic can be compared with that reference, while remembering that its accuracy depends on the large-sample approximation.

Worked Example: Grouped Binomial Deviance With Sparse Groups

Consider five binomial groups, each with $n_j=4$, observed success counts $4,1,3,4,4$, and specified fitted success probabilities $0.6,0.4,0.4,0.4,0.6$. Calculate the residual deviance contributions. The small group sizes make this a useful warning about interpreting a computed statistic.

For each group, the fitted expected success and failure counts are $4\hat{\pi}_j$ and $4(1-\hat{\pi}_j)$. Applying the grouped binomial deviance formula gives, in group order, $$ \begin{aligned} D_1&=2\{4\log(4/2.4)\}\approx4.0866,\\ D_2&=2\{\log(1/1.6)+3\log(3/2.4)\}\approx0.3989,\\ D_3&=2\{3\log(3/1.6)+\log(1/2.4)\}\approx2.0207,\\ D_4&=2\{4\log(4/1.6)\}\approx7.3303,\\ D_5&=2\{4\log(4/2.4)\}\approx4.0866. \end{aligned} $$ Adding the contributions gives $D\approx4.0866+0.3989+2.0207+7.3303+4.0866=17.9231$. This calculation establishes the deviance value, not the quality of a chi-square approximation. Each group has only four trials, so large expected counts are absent. In addition, a reference degrees of freedom would depend on the dimension of the fitted model being compared with the saturated model; it cannot be inferred from the five contributions alone.

Worked Example: Individual Bernoulli Data and a Growing Saturated Model

Suppose 100 independent Bernoulli observations include 40 successes and 60 failures. Fit an intercept-only model, so the estimated success probability is $\hat{\pi}=40/100=0.4$. If the saturated model assigns a separate success probability to every observation, calculate the residual deviance and explain why the usual fixed-dimensional Wilks argument does not justify a $\chi^2_{99}$ reference.

The fitted intercept-only model assigns probability $0.4$ to each success and $0.6$ to each failure. The saturated model assigns probability one to each observed outcome, so its log likelihood is zero. Consequently, $$ D=2\left\{40\log(1/0.4)+60\log(1/0.6)\right\}. $$ Since $\log(2.5)\approx0.916291$ and $\log(1/0.6)\approx0.510826$, $$ D\approx2\{40(0.916291)+60(0.510826)\} =2(36.65164+30.64956) \approx134.6024. $$ The saturated model has 100 separate probabilities, so its dimension grows with the number of observations; the fitted model has dimension one. Wilks’ fixed-dimensional theorem does not cover this comparison. In particular, each saturated probability is informed by only one Bernoulli trial. The nominal dimension difference of 99 does not, by itself, validate a $\chi^2_{99}$ calibration.

Limits of the Reference and Common Pitfalls

The chi-square reference is most defensible when there are a fixed number of groups, each group contributes increasing information, the true model is interior and identifiable, and the information matrix is nonsingular. For a grouped Poisson model, this typically means that the number of cells stays fixed while expected counts grow. For grouped binomial data, it typically means that the number of groups stays fixed, the group sizes grow, and the true probabilities remain away from zero and one. The fitted-model dimension must also remain fixed.

By contrast, a large total sample size does not guarantee large information in every cell. A few large counts alongside many tiny expected counts can leave the chi-square approximation poor. Likewise, with one Bernoulli observation per unit, a saturated model has one parameter per observation. Even as the sample grows, that is a growing-dimensional comparison, not the fixed-dimensional setting of Wilks’ theorem. Alternatives or refinements may be appropriate in particular settings, but the ordinary fixed-degree residual-deviance rule cannot be assumed to settle them.

Mistake: “Residual deviance always has a chi-square distribution with observations minus coefficients degrees of freedom.” This treats a nominal dimension count as a theorem. The correction is to check that the saturated model has fixed dimension, the likelihood is regular, and the information grows in the required way.
Mistake: “A large sample total makes sparse cells harmless.” A large total can coexist with small expected counts in many cells. The correction is to inspect the cell-level information and the model structure, not just the total sample size.
Mistake: “A deviance calculation proves the calibration.” The deviance is a likelihood-based statistic whether or not its reference approximation is good. The correction is to justify the reference distribution separately from calculating the statistic.
Mistake: “Deviance and Pearson’s statistic are identical.” They have the same leading quadratic form under suitable large-count conditions, but differ at finite counts. The correction is to treat the Taylor approximation as an asymptotic relationship, not an exact identity.

Key takeaway: Residual deviance has an asymptotic $\chi^2_{d-p}$ reference under a regular, fixed-dimensional likelihood comparison. Sparse information or a saturated model whose dimension grows with the sample can invalidate that calibration, regardless of whether the deviance itself is easy to compute.

Check Your Understanding

Use the conditions and examples above to test whether you can distinguish a deviance value from its reference distribution.

In a regular model with a fixed $d$-dimensional full model and a $p$-dimensional null model, what is the limiting likelihood-ratio distribution under the null?
Why does the number of groups need to remain fixed in the standard grouped-data residual-deviance argument?
For the four-category example, why is the asymptotic degrees of freedom three rather than four?
What does the large-expected-count Taylor expansion say about Poisson deviance and Pearson’s statistic?
Why does an intercept-only fit to individual Bernoulli data not automatically have a $\chi^2_{99}$ residual-deviance reference?