Residual Deviance: A Fit Summary for One Model
After fitting a generalized linear model (GLM), we need summaries that describe how well its fitted means account for the observed responses. Residual deviance compares the fitted model's likelihood with the largest likelihood available when each observation can have its own mean. That comparison gives a measure of the likelihood still left unexplained by the fitted model. Residual degrees of freedom, in turn, describe the difference in dimension between the observation-level mean space and the fitted model's linear-predictor space.
These quantities are useful descriptions of a fit, but they are not, by themselves, a test comparing two scientific models. A residual deviance refers to one fitted model and its saturated reference; a nested-model test concerns the likelihood difference between two specified models. Keeping those roles separate prevents a fit summary from being given an unwarranted inferential interpretation.
Definition and Basic Properties
Suppose \(Y_1,\ldots,Y_n\) are independent responses with observed values \(y_1,\ldots,y_n\), and a GLM specifies a mean vector through a design matrix \(X\) and a link function. Write \(\ell(\hat\beta)\) for the fitted model's log-likelihood at its maximum, and \(\ell_{\mathrm{sat}}\) for the maximum log-likelihood in the saturated model. The saturated model allows a separate mean for each response observation, subject to the response distribution's permissible mean space.
The saturated model contains the fitted model's possible mean vectors, so its maximum likelihood cannot be smaller than the fitted model's maximum likelihood. Therefore residual deviance is nonnegative, provided the maxima exist (or are interpreted as suprema). If a saturated maximum is attained, zero deviance means that the fitted model attains the same likelihood as the saturated model. When the saturated fit is unique, this requires the fitted means to equal the saturated means.
An offset changes the specified linear predictor but does not add a freely estimated column to \(X\), so it does not increase the rank used in this count. If \(X\) has full column rank \(p\), then \(\operatorname{rank}(X)=p\) and the familiar expression is \(n-p\). In grouped-binomial data, \(n\) is the number of grouped response observations (rows), not the total number of underlying trials.
Why Residual Degrees of Freedom Equal \(n-\operatorname{rank}(X)\)
Proof. The map \(\beta\mapsto X\beta\) has image equal to the column space of \(X\). By the definition of matrix rank, this image has dimension \(r\). The saturated model permits an observation-level predictor vector in an \(n\)-dimensional space (for responses whose means are interior to their permissible ranges). The codimension of the fitted linear-predictor space is therefore \(n-r\). This dimension difference defines the conventional residual degrees of freedom, including when the coefficient vector is not uniquely represented because \(X\) is rank deficient. \(\square\)
This is a dimension count, not a claim that GLM residuals are independent or that their squared values have a particular distribution. In a Gaussian linear model it aligns with the familiar residual dimension in least squares. In other GLMs it remains the conventional rank-based count, but further distributional claims require additional assumptions.
Worked Example: Gaussian Regression With the Correct Least-Squares Fit
Take \(x=(0,1,2,3,4)\) and \(y=(1,2,2,4,6)\), and fit a Gaussian GLM with an intercept and a linear term in \(x\). The sample means are \(\bar{x}=2\) and \(\bar{y}=3\). The centered cross-product and squared-\(x\) sums are $$ \sum_{i=1}^5(x_i-2)(y_i-3)=12,\qquad \sum_{i=1}^5(x_i-2)^2=10. $$ Thus the least-squares slope is \(12/10=1.2\), and the intercept is \(3-1.2(2)=0.6\). The fitted values are \(0.6,1.8,3.0,4.2,5.4\), so the residuals are \(0.4,0.2,-1.0,-0.2,0.6\). Their squared sum is $$ 0.4^2+0.2^2+(-1.0)^2+(-0.2)^2+0.6^2 =0.16+0.04+1.00+0.04+0.36=1.60. $$ For the usual unscaled Gaussian GLM deviance, \(D_{\mathrm{res}}\) equals this residual sum of squares, so \(D_{\mathrm{res}}=1.60\). With an intercept and slope, the design matrix has rank 2, giving \(5-2=3\) residual degrees of freedom. If the Gaussian variance \(\sigma^2\) is treated as fixed, the twice-log-likelihood difference is \(1.60/\sigma^2\); the reported unscaled deviance remains \(1.60\).
Calculating Deviance for Count and Binary Responses
For independent Poisson observations with fitted means \(\hat\mu_i>0\), the unscaled deviance is $$ D_{\mathrm{res}}=2\sum_{i=1}^n\left\{y_i\log\left(\frac{y_i}{\hat\mu_i}\right)-(y_i-\hat\mu_i)\right\}, $$ where \(0\log(0/\hat\mu_i)\) is defined as zero. For grouped binomial observations, let \(m_i\) be the number of trials, \(y_i\) the number of successes, and \(\hat p_i\) the fitted success probability. Then $$ D_{\mathrm{res}}=2\sum_{i=1}^n\left\{y_i\log\left(\frac{y_i}{m_i\hat p_i}\right)+(m_i-y_i)\log\left(\frac{m_i-y_i}{m_i(1-\hat p_i)}\right)\right\}. $$ A term with zero numerator is defined to be zero by continuity. These expressions compare fitted means with the observation-specific saturated estimates: \(y_i\) for Poisson and \(y_i/m_i\) for grouped binomial data when those estimates are in the interior.
Worked Example: Poisson Intercept-Only Model
Suppose four independent Poisson counts are \(y=(2,1,4,3)\), and fit an intercept-only model. The maximum-likelihood estimate of the common mean is $$ \hat\mu=\bar y=\frac{2+1+4+3}{4}=2.5. $$ Because the observed and fitted totals are both 10, the \(\sum_i(y_i-\hat\mu)\) terms sum to zero. The residual deviance is therefore $$ D_{\mathrm{res}}=2\left\{2\log(2/2.5)+1\log(1/2.5)+4\log(4/2.5)+3\log(3/2.5)\right\}. $$ The four terms inside braces are approximately \(-0.4463\), \(-0.9163\), \(1.8800\), and \(0.5470\). Their sum is \(1.0644\), so \(D_{\mathrm{res}}\approx2.1288\). The design matrix is a column of four ones, of rank 1. Thus the residual degrees of freedom are \(4-1=3\). This calculation describes the fitted intercept-only model relative to its saturated reference; it does not yet compare that model with a different specified model.
Worked Example: Grouped-Binomial Intercept-Only Model
Consider three groups, each with 10 trials, and observed successes \(y=(2,5,8)\). Under a common-probability intercept-only model, the fitted probability is the pooled proportion: $$ \hat p=\frac{2+5+8}{10+10+10}=\frac{15}{30}=0.5. $$ For the first group, its deviance contribution is $$ 2\left\{2\log(2/5)+8\log(8/5)\right\} =2\{-1.8326+3.7600\}\approx3.8549. $$ The middle group has \(5\) successes and \(5\) failures, matching the fitted proportions, so its contribution is zero. The last group's contribution equals the first group's because its success and failure counts are reversed. Hence \(D_{\mathrm{res}}\approx3.8549+0+3.8549=7.7098\). There are three grouped response observations and the intercept-only design matrix has rank 1, giving \(3-1=2\) residual degrees of freedom—not \(30-1\). The likelihood is organized by the three grouped observations; the 30 trials are not 30 separate rows in this grouped-data model.
What These Quantities Do—and Do Not—Tell You
Residual deviance is a single-model fit summary: it quantifies the fitted model's likelihood loss relative to the saturated reference. Residual degrees of freedom give a rank-based measure of how many dimensions separate the observation-level model from the fitted linear-predictor space. A large deviance relative to its residual degrees of freedom may prompt investigation of fit, but that comparison alone does not establish a formal test result. Its interpretation depends on the response distribution, dispersion, sample size, and calibration assumptions.
A likelihood-ratio comparison between two specified models instead uses the difference between their maximized log-likelihoods. The saturated model is a reference for defining each model's residual deviance; it is not automatically the scientific alternative in a comparison of two proposed models. The connection between differences in residual deviance and likelihood-ratio statistics, and the conditions for inferential reference distributions, require separate treatment.
Common Mistakes
- Subtracting the number of columns instead of the rank. If columns are linearly dependent, they do not add fitted dimensions. Calculate \(\operatorname{rank}(X)\), then use \(n-\operatorname{rank}(X)\).
- Counting trials instead of grouped observations. For a grouped-binomial model with one row per group, the residual-df count starts from the number of groups, not the total number of Bernoulli trials.
- Calling residual deviance a test of the fitted model against a scientific alternative. It compares the fitted model to the saturated reference. A test between two specified models requires a different comparison and its own assumptions.
- Mixing unscaled deviance and a dispersion-scaled likelihood difference. For a Gaussian model with fixed variance, the twice-log-likelihood difference is RSS divided by \(\sigma^2\), while conventional unscaled deviance is RSS. State which quantity is being reported.
- Assuming a deviance-to-df ratio is automatically a calibrated test. That ratio may serve as a diagnostic summary, but it is not by itself proof that a chosen reference distribution or cutoff is valid.
Check Your Understanding
Try these questions before moving on.
- A GLM has 12 response rows and a design matrix of rank 4. What are its residual degrees of freedom, and why is rank the relevant quantity?
- For a grouped-binomial model, there are 5 groups with 20 trials per group. Which count is used as \(n\) in the conventional residual-df formula?
- In the Poisson example, why do the terms \(-(y_i-\hat\mu_i)\) sum to zero?
- Explain why the residual deviance of a fitted model is not automatically a test statistic comparing two scientific models.
- For a Gaussian model with known variance \(\sigma^2\) and residual sum of squares 8, state the unscaled deviance and the twice-log-likelihood difference.