Null Deviance and Residual Deviance

Why Compare Deviances?

A generalized linear model (GLM) is often assessed by comparing its fit with a simple baseline: a model containing an intercept but no explanatory predictors. Deviance provides a common scale for describing both fits. The null deviance describes the baseline model’s lack of fit; the residual deviance describes the lack of fit remaining after fitting the predictors. Both are measured against the same benchmark, the saturated model.

This comparison is useful when interpreting a GLM summary or assessing how much the predictors have improved the fit. It is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. The difference is a descriptive measure of improvement; by itself, it does not establish that the improvement is statistically significant.

Definition: For a specified response family and data set, the saturated model fits each observation as well as that model can, subject to the response distribution. If \(\ell_{\mathrm{sat}}\) is its maximized log-likelihood and \(\ell_{\mathrm{fit}}\) is the maximized log-likelihood of a fitted model, the model’s (unscaled) deviance is

D=2\left(\ell_{\mathrm{sat}}-\ell_{\mathrm{fit}}\right).

The null deviance, \(D_{\mathrm{null}}\), uses the intercept-only fit. The residual deviance, \(D_{\mathrm{res}}\), uses the fit with the specified predictors. Both compare their fit with the same saturated model on the same observations.

The formulas below use independent responses and the usual maximum-likelihood GLM fit. An intercept-only model means no explanatory predictors; if a model includes a required offset, the null model retains that offset. Comparisons are meaningful only when the two deviances use the same response data, response family, weights, and likelihood convention. The examples use unweighted data without offsets.

Calculating Deviance for Common Responses

For independent Poisson observations \(Y_i\) with fitted means \(\hat{\mu}_i\), the saturated mean for observation \(i\) is \(y_i\). The Poisson deviance is

D_{\mathrm{Pois}} = 2\sum_{i=1}^n \left[ y_i\log\left(\frac{y_i}{\hat{\mu}_i}\right) -(y_i-\hat{\mu}_i) \right].

When \(y_i=0\), the term \(y_i\log(y_i/\hat{\mu}_i)\) is defined by its limit as zero, so that observation contributes \(2\hat{\mu}_i\). This convention avoids trying to evaluate \(\log(0)\).

For grouped binomial data with \(y_i\) successes out of \(m_i\) trials and fitted success probability \(\hat{p}_i\), the corresponding deviance is

D_{\mathrm{Bin}} = 2\sum_{i=1}^n \left[ y_i\log\left(\frac{y_i}{m_i\hat{p}_i}\right) +(m_i-y_i)\log\left(\frac{m_i-y_i}{m_i(1-\hat{p}_i)}\right) \right].

Terms with zero observed successes or failures are again evaluated by their limiting value of zero. These expressions illustrate why deviance depends on the response family: use the likelihood for the specified family, not a formula copied from a different model.

Worked Example: Poisson Null and Residual Deviance

A hypothetical insurer records claim counts for four equally exposed policy segments. The observed counts are \(1, 3, 4, 8\). The null model gives every segment fitted mean \(4\). A model with a two-level segment indicator gives fitted means \(2, 2, 6, 6\), respectively. Calculate both deviances and their difference.

Method. Apply the Poisson deviance contribution to each observation. The means in the second model are the averages within the two indicator groups: \((1+3)/2=2\) and \((4+8)/2=6\).

Null deviance. With \(\hat{\mu}_i=4\), the four contributions are

\begin{aligned} y_1=1 &: 2[1\log(1/4)-(1-4)]=3.2274113,\\ y_2=3 &: 2[3\log(3/4)-(3-4)]=0.2739076,\\ y_3=4 &: 2[4\log(4/4)-(4-4)]=0,\\ y_4=8 &: 2[8\log(8/4)-(8-4)]=3.0903549. \end{aligned}

Adding them gives \(D_{\mathrm{null}}=3.2274113+0.2739076+0+3.0903549=6.5916738\).

Residual deviance. Using fitted means \(2,2,6,6\), the contributions are

\begin{aligned} y_1=1,\ \hat{\mu}_1=2 &: 2[\log(1/2)-(1-2)]=0.6137056,\\ y_2=3,\ \hat{\mu}_2=2 &: 2[3\log(3/2)-(3-2)]=0.4327906,\\ y_3=4,\ \hat{\mu}_3=6 &: 2[4\log(4/6)-(4-6)]=0.7562791,\\ y_4=8,\ \hat{\mu}_4=6 &: 2[8\log(8/6)-(8-6)]=0.6029132. \end{aligned}

Thus \(D_{\mathrm{res}}=0.6137056+0.4327906+0.7562791+0.6029132=2.4056885\), and the reduction is \(6.5916738-2.4056885=4.1859853\), or \(4.1860\) to four decimal places. The predictor model fits better by this deviance measure. The common mistake to avoid is calling the residual deviance itself the improvement: the improvement is the difference between the two deviances.

Why Their Difference Measures Improvement

The key relationship follows directly from the definition. Write the two maximized log-likelihoods as \(\ell_{\mathrm{null}}\) and \(\ell_{\mathrm{res}}\). Since both deviances use the same saturated-model likelihood, subtracting cancels that benchmark:

\begin{aligned} D_{\mathrm{null}}-D_{\mathrm{res}} &=2(\ell_{\mathrm{sat}}-\ell_{\mathrm{null}}) -2(\ell_{\mathrm{sat}}-\ell_{\mathrm{res}})\\ &=2(\ell_{\mathrm{res}}-\ell_{\mathrm{null}}). \end{aligned}

Therefore, the reduction is twice the increase in maximized log-likelihood from the null fit to the predictor fit. This is a descriptive identity, not a probability statement. When the predictor model contains the null model and both are maximized on the same data under the same likelihood, the larger model can always reproduce the null fit by setting its predictor effects appropriately. Consequently, \(\ell_{\mathrm{res}}\geq\ell_{\mathrm{null}}\), and the deviance reduction cannot be negative, apart from possible numerical or fitting issues.

The reduction is not the residual deviance, nor is it a percentage unless a denominator is specified. A relative reduction can be reported as \((D_{\mathrm{null}}-D_{\mathrm{res}})/D_{\mathrm{null}}\), provided \(D_{\mathrm{null}}>0\); that percentage is a descriptive summary, not a test statistic with an automatic significance interpretation.

Worked Example: Binomial Deviance Against a Shared Benchmark

In a hypothetical study, two groups each contain four individuals. The first group has one success and the second has three. The null model uses one common probability; the group-indicator model estimates a separate probability for each group. Find the null deviance, residual deviance, and improvement.

Null fit. There are \(1+3=4\) successes among \(4+4=8\) trials, so the common fitted probability is \(\hat{p}=4/8=0.5\). In each group, the null model expects two successes and two failures. The first group’s deviance contribution is

2\left[1\log(1/2)+3\log(3/2)\right] =2[-0.6931472+1.2163953] =1.0464963.

For the second group the observed success and failure counts are reversed, so its contribution is also \(1.0464963\). Therefore \(D_{\mathrm{null}}=1.0464963+1.0464963=2.0929926\), or \(2.0930\) to four decimal places.

Predictor fit. The group-specific fitted probabilities are \(1/4=0.25\) and \(3/4=0.75\). These equal the observed proportions, so the fitted expected counts match the observed counts in both groups. Every binomial deviance term is zero, giving \(D_{\mathrm{res}}=0\). The improvement is \(2.0930-0=2.0930\).

A frequent arithmetic trap is to compute the first group’s contribution correctly and report it as the total. There are two groups, and each contributes \(1.0464963\); the total null deviance is their sum, \(2.0930\).

Interpreting a Model Summary

A model summary may report null deviance and residual deviance without displaying the observation-level calculations. The following steps keep the comparison clear.

Check what is being compared.
Confirm that both values refer to the same response, observations, family, weights, and offset structure. The null fit is an intercept-only baseline, not a saturated fit.

Subtract in the correct order.
Compute \(D_{\mathrm{null}}-D_{\mathrm{res}}\). A positive value describes the deviance reduction; reversing the subtraction changes its sign.

State what the result means.
The fitted predictors reduce the model’s deviance by that amount relative to the intercept-only fit. The remaining residual deviance still measures lack of fit relative to the saturated model.

Do not infer significance from subtraction alone.
A deviance difference may be used in a formal test under additional conditions. The reduction by itself does not supply a reference distribution, p-value, or guarantee of useful prediction.

Worked Example: Interpreting Reported Deviances

A hypothetical GLM for claim counts on 120 policies reports null deviance \(24.80\) and residual deviance \(9.36\). Describe the improvement and the remaining deviance, including the relative reduction.

Difference. The deviance reduction is \(24.80-9.36=15.44\).

Relative reduction. Dividing by the null deviance gives \(15.44/24.80=0.6225806\), or \(62.3\%\) to one decimal place. The residual deviance is \(9.36/24.80=0.3774194\), or \(37.7\%\) of the null deviance.

The predictors reduce deviance by \(15.44\) compared with the intercept-only fit, while \(9.36\) remains relative to the saturated benchmark. These summaries do not establish whether the reduction is statistically significant or whether the model is adequate for every purpose.

Common Traps

Comparing against different benchmarks. Null and residual deviance must both be measured against the saturated model for the same data and likelihood. Comparing numbers from different response subsets or different likelihood conventions does not yield the stated improvement.
Dropping a factor of two or a repeated contribution. The deviance definition has an outer factor of two, and each observation contributes once for every independent record or frequency-weighted replicate. Sum all contributions before rounding.
Assuming the residual deviance is the reduction. The reduction is \(D_{\mathrm{null}}-D_{\mathrm{res}}\); the residual deviance is what remains.
Treating improvement as proof of a good model. A predictor model can improve on the null fit and still leave substantial lack of fit, miss important structure, or perform poorly for prediction.
Calling the reduction a significance result. A formal likelihood-ratio test requires its own conditions and reference distribution. A later treatment of likelihood-ratio tests addresses that inferential step.

Key takeaway: Null deviance and residual deviance are each measured against the same saturated model. Their difference, \(D_{\mathrm{null}}-D_{\mathrm{res}}\), is twice the maximized log-likelihood gain over the intercept-only fit. Interpret it as a measure of improvement, not as a test result on its own.

Check Your Understanding

For each question, use the specified response family and compare fits on the same observations.

Two independent Poisson counts are \(1\) and \(3\). The null model fits mean \(2\) to each, and a saturated fit has residual deviance zero. Calculate the null deviance and its reduction relative to the saturated fit. (Answer: \(D_{\mathrm{null}}=1.0465\); reduction \(=1.0465\).)
Two binomial groups each have 10 trials, with 4 and 6 successes. The null model uses their pooled probability, and the group-specific model fits each observed proportion exactly. Find the null deviance and reduction. (Answer: \(D_{\mathrm{null}}=0.8054\); reduction \(=0.8054\).)
A model summary gives null deviance \(12.6\) and residual deviance \(8.1\). Calculate the reduction and its percentage of null deviance. (Answer: reduction \(=4.5\); relative reduction \(=35.7\%\).)
A Poisson data set contains four independent observations with counts \(2,2,6,6\). The intercept-only fitted mean is \(4\) for every observation. Calculate the null deviance using all four contributions. (Answer: \(D_{\mathrm{null}}=4.1860\).)
A predictor model and its intercept-only null model use the same response data and likelihood. Their reported deviances are \(31.4\) and \(18.9\), respectively. State the improvement and explain what can and cannot be concluded from it alone. (Answer: reduction \(=12.5\); the predictors improve fit descriptively, but the reduction alone does not establish statistical significance or overall adequacy.)

What You'll Learn