Deviance As A GLM Fit Criterion

Why Compare a GLM With a Saturated Model?

A GLM produces fitted means or probabilities, but those fitted values alone do not summarize how closely the model reproduces the observed responses. Residual deviance measures the loss in log-likelihood when we use the fitted model instead of a saturated model: one that gives each observation its own parameter and therefore fits the observed data as closely as the specified response distribution allows.

Deviance is a likelihood-based measure of fit, not a direct measure of predictive accuracy and not, by itself, a test that a model is adequate. Its calculation depends on the response distribution and on what counts as an observation. The grouped-binomial and Poisson cases below make those details explicit.

Definition: For a specified response distribution, let $\ell_{\text{fit}}$ be the log-likelihood under the fitted GLM and let $\ell_{\text{sat}}$ be the log-likelihood under the corresponding saturated model. The residual deviance is $$ D = 2\left(\ell_{\text{sat}}-\ell_{\text{fit}}\right). $$ For a fitted model within the same likelihood setup as its saturated model, $D$ is nonnegative: the saturated model maximizes the likelihood by allowing a separate parameter for each observation.

The factor of 2 is conventional. Log-likelihoods use natural logarithms here. The formulas in this tutorial assume independent observations, the stated binomial or Poisson distributions, and no additional frequency or prior weights. If observations have weights or a different likelihood specification, the likelihood contributions must be adjusted to match that specification; these unweighted formulas should not be applied unchanged.

Deriving the Grouped-Binomial Deviance

For grouped-binomial observation $i$, suppose $Y_i\sim\operatorname{Binomial}(n_i,p_i)$, where $Y_i$ is the number of successes among $n_i$ trials. A fitted GLM supplies a fitted probability $\hat p_i$, so the fitted mean count is $n_i\hat p_i$. In the saturated model, the maximum-likelihood estimate for this observation is $\hat p_i^{\text{sat}}=y_i/n_i$, provided $0<y_i<n_i$. At the boundaries, the saturated estimate is 0 or 1.

The binomial log-likelihood contribution is $\log {n_i\choose y_i}+y_i\log(p_i)+(n_i-y_i)\log(1-p_i)$. When we subtract the fitted log-likelihood from the saturated log-likelihood, the binomial coefficient occurs in both and cancels. Substituting the saturated estimate gives the grouped-binomial deviance:

D_{\text{bin}} = 2\sum_i \left[ y_i\log\left(\frac{y_i}{n_i\hat p_i}\right) +(n_i-y_i)\log\left(\frac{n_i-y_i}{n_i(1-\hat p_i)}\right) \right].

The second term is essential: it represents failures, not successes. Equivalently, the comparison is between the observed success and failure counts and their fitted expected counts. If a count is zero, its contribution has the limiting value $0\log(0/a)=0$ for $a>0$. This convention follows from $\lim_{z\to0^+}z\log z=0$.

Worked Example: One Grouped-Binomial Observation

A hypothetical group has $n=8$ trials and $y=3$ successes. A fitted binomial GLM gives $\hat p=0.5$. Calculate the deviance contribution. The response distribution is $Y\sim\operatorname{Binomial}(8,p)$; the fitted parameter is $\hat p=0.5$, and the saturated estimate is $3/8=0.375$. No interest rate or mortality assumption applies to this statistical likelihood calculation.

Method. Compare both observed counts with their fitted expected counts. The fitted success count is $8(0.5)=4$, and the fitted failure count is also $8(1-0.5)=4$. The observed counts are 3 successes and $8-3=5$ failures.

Therefore, $D=2[3\log(3/4)+5\log(5/4)]$. The two log terms are $3\log(0.75)=3(-0.2876821)=-0.8630463$ and $5\log(1.25)=5(0.2231436)=1.1157178$. Their sum is $0.2526715$, so $D=2(0.2526715)=0.5053430$.

Rounded to four decimal places, the deviance contribution is 0.5053. A check is that the result is nonnegative. The common mistake this method avoids is omitting the failure-count term; using only the successes would not be the binomial likelihood comparison.

Summing Deviance Across Binomial Groups

For independent grouped-binomial observations, the total log-likelihood is the sum of the individual contributions. Consequently, total residual deviance is the sum of the group contributions. A group with a larger discrepancy between its observed counts and fitted expected counts will generally contribute more deviance, although the group size and fitted probability also matter.

Worked Example: Deviance for Two Binomial Groups

A hypothetical model is fitted to two independent groups. Group 1 has $n_1=10$, $y_1=4$, and fitted probability $\hat p_1=0.3$. Group 2 has $n_2=20$, $y_2=12$, and fitted probability $\hat p_2=0.5$. Each response follows its stated binomial distribution, with fitted parameters as given. No interest rate or mortality assumption applies.

Method. Calculate each group's success and failure terms, then add the contributions. For Group 1, the observed success and failure counts are 4 and 6. The fitted counts are $10(0.3)=3$ and $10(0.7)=7$. Thus, $D_1=2[4\log(4/3)+6\log(6/7)]$. The terms are $4(0.2876821)=1.1507283$ and $6(-0.1541507)=-0.9249041$. Hence $D_1=2(0.2258242)=0.4516484$.

For Group 2, the observed counts are 12 successes and 8 failures; the fitted counts are $20(0.5)=10$ and 10. Thus, $D_2=2[12\log(12/10)+8\log(8/10)]$. The terms are $12(0.1823216)=2.1878587$ and $8(-0.2231436)=-1.7851484$. Hence $D_2=2(0.4027103)=0.8054205$.

The total is $D=D_1+D_2=0.4516484+0.8054205=1.2570689$, or 1.2571 to four decimal places. The check is to add the unrounded group contributions before rounding. The common mistake avoided is treating the two observations as one binomial group; their fitted probabilities differ, so their likelihood contributions must be calculated separately.

Deriving the Poisson Deviance

For a Poisson response, suppose independent counts satisfy $Y_i\sim\operatorname{Poisson}(\mu_i)$, and the fitted GLM gives mean $\hat\mu_i$. The log-likelihood contribution, apart from notation, is $y_i\log(\mu_i)-\mu_i-\log(y_i!)$. The saturated model maximizes this contribution at $\hat\mu_i^{\text{sat}}=y_i$ for a positive observed count. Subtracting fitted log-likelihood from saturated log-likelihood cancels the factorial term and gives:

D_{\text{Pois}} = 2\sum_i\left[ y_i\log\left(\frac{y_i}{\hat\mu_i}\right) -(y_i-\hat\mu_i) \right].

When $y_i=0$, the logarithmic term is interpreted as zero, leaving the contribution $2\hat\mu_i$. This result is also obtained directly from the likelihood: for a zero count, the saturated mean is 0 and its log-likelihood contribution is 0, while the fitted contribution is $-\hat\mu_i$. Deviance is zero for a Poisson observation whose fitted mean equals its observed count.

Worked Example: Poisson Deviance With a Zero Count

A hypothetical Poisson GLM is fitted to three independent counts. The observed counts are $y=(0,2,5)$, and the fitted means are $\hat\mu=(1,2,4)$. The response distributions are $Y_i\sim\operatorname{Poisson}(\mu_i)$, with the fitted means specified above. No interest rate or mortality assumption applies.

Method. Apply the Poisson deviance contribution to each count. For the first observation, $y_1=0$ and $\hat\mu_1=1$, so the contribution is $2[0-(0-1)]=2$. For the second, $y_2=\hat\mu_2=2$, so the contribution is $2[2\log(1)-(2-2)]=0$.

For the third observation, $y_3=5$ and $\hat\mu_3=4$. Its contribution is $2[5\log(5/4)-(5-4)]$. Since $\log(1.25)=0.2231436$, this is $2[5(0.2231436)-1]=2(1.1157178-1)=0.2314355$.

The total deviance is $2+0+0.2314355=2.2314355$, or 2.2314 to four decimal places. Each contribution is nonnegative, and the exactly fitted second count contributes zero. The common mistake avoided is attempting to evaluate $\log(0/\hat\mu_1)$ as an ordinary logarithm; the zero-count convention gives a finite contribution of $2\hat\mu_1$.

What Deviance Can—and Cannot—Tell You

For a fixed data set, response definition, and likelihood, a smaller residual deviance means the fitted model has a log-likelihood closer to that of the saturated model. For two nested models fitted to the same observations under the same likelihood, the difference in their deviances is twice the difference in maximized log-likelihoods. This is the likelihood-ratio comparison expressed through deviance:

D_{\text{reduced}}-D_{\text{larger}} =2\left(\ell_{\text{larger}}-\ell_{\text{reduced}}\right).

This identity follows by writing each deviance as $2(\ell_{\text{sat}}-\ell_{\text{fit}})$ and subtracting: the common saturated log-likelihood cancels. The larger model has at least as high a maximized likelihood as the nested reduced model, so the difference is nonnegative. This comparison is meaningful only when both models use the same response observations and likelihood basis.

Raw deviance alone does not account for how many parameters were used. A more complex model can usually achieve a lower deviance, so the value should not be interpreted as proof that the model is useful or appropriate. Nor should a deviance value automatically be turned into a goodness-of-fit probability: such an interpretation requires additional conditions and a justified reference distribution. This tutorial uses deviance as a fit criterion and likelihood comparison, not as a stand-alone adequacy test.

Identify the response likelihood.
Determine whether the observations are grouped-binomial counts, Poisson counts, or another response type. Do not use a formula derived for a different distribution.

Find observed and fitted quantities.
For binomial data, record successes, failures, and fitted expected counts. For Poisson data, record each observed count and fitted mean.

Calculate and check contributions.
Use natural logarithms, apply the zero-count convention where needed, sum contributions, and check that the total is nonnegative.

Common Calculation Traps

Leaving out binomial failures. A grouped-binomial deviance includes both success and failure terms. The success term alone is not the likelihood comparison.
Using the wrong fitted expected count. For grouped-binomial data, the fitted success count is $n_i\hat p_i$, not merely $\hat p_i$. The fitted failure count is $n_i(1-\hat p_i)$.
Taking the logarithm of zero. A zero observed count contributes zero to the $y\log(y/\hat\mu)$ or corresponding binomial count term by continuity; it does not make the deviance undefined.
Comparing unrelated deviance values. A comparison requires the same response data and compatible likelihoods. A smaller number from a different response definition is not automatically evidence of a better model.
Calling deviance a prediction error. Deviance is based on likelihood relative to a saturated model. It is not the same quantity as a prediction metric such as RMSE.

Key takeaway: Residual deviance is twice the log-likelihood gap between a fitted GLM and its saturated counterpart. Derive the formula from the response likelihood, include every part of each observation's likelihood contribution, and interpret the result in the context of the same data and model family.

Check Your Understanding

For each problem, use natural logarithms and round the final deviance to four decimal places.

A hypothetical grouped-binomial observation has $n=10$, $y=2$, and fitted probability $\hat p=0.3$. Calculate its deviance, including both success and failure contributions. (Answer: 0.5146)
A hypothetical Poisson observation has $y=0$ and fitted mean $\hat\mu=2$. Calculate its deviance contribution using the zero-count convention. (Answer: 4.0000)
A hypothetical Poisson count is $y=3$, with fitted mean $\hat\mu=2$. Calculate its deviance contribution. (Answer: 0.4328)
Two independent hypothetical binomial groups each have $n=5$. The first has $y=1$ and $\hat p=0.4$; the second has $y=4$ and $\hat p=0.6$. Calculate the total deviance. (Answer: 1.8303)
A hypothetical grouped-binomial observation has $n=8$, $y=3$, and fitted probability $\hat p=0.5$. State the fitted success and failure counts, then calculate the deviance. (Answer: fitted counts 4 and 4; deviance 0.5053)

What You'll Learn