Deviance as Distance From the Saturated Model

Why Compare a Fitted Model With a Saturated Model?

A fitted generalized linear model restricts the mean responses through a chosen set of predictors and a link function. A natural way to describe the cost of those restrictions is to compare the model’s likelihood with the largest likelihood available when each observation can have its own mean. That unrestricted reference is called the saturated model. The likelihood gap is the GLM deviance.

Deviance is useful because it gives a common likelihood-based measure of relative fit. It is not a distance between coefficient vectors, nor is it automatically a test statistic with a known reference distribution. Its basic meaning is more direct: it measures how much log likelihood is lost by using the fitted model instead of the saturated reference.

Definition: Let $\ell_M$ be the log likelihood for a fitted model $M$, maximized over its allowed parameter values. Let $\ell_{\mathrm{sat}}$ be the supremum of the log likelihood in the saturated model, in which each observation has its own freely varying mean within the response family. The deviance is $$ D(M)=2\left(\ell_{\mathrm{sat}}-\ell_M\right). $$ If a maximum is not attained, use the corresponding supremum. This definition assumes the likelihoods being compared are finite.

The saturated model is defined relative to the observations and response family. For example, a saturated Poisson model gives every observed count its own Poisson mean; a saturated grouped-binomial model gives every group its own success probability. A saturated model does not necessarily reproduce every observation as a possible mean: it maximizes the likelihood separately for each observation, using boundary parameter values when necessary.

Basic Properties of the Likelihood Gap

The saturated model allows at least all the mean configurations allowed by a submodel $M$. Consequently, its supremum likelihood cannot be lower. The following result makes this precise.

Theorem: If every parameter configuration allowed by $M$ is also allowed by the saturated model, then $D(M)\geq 0$. If both maxima are attained, $D(M)=0$ exactly when a likelihood-maximizing configuration in $M$ also attains the saturated-model maximum.

Proof. Write $\mathcal{P}_M$ for the likelihood distributions allowed by $M$, and $\mathcal{P}_{\mathrm{sat}}$ for those allowed by the saturated model. By assumption, $\mathcal{P}_M\subseteq\mathcal{P}_{\mathrm{sat}}$. Taking suprema of the same log-likelihood function over these sets gives

\ell_M=\sup_{P\in\mathcal{P}_M}\ell(P) \leq \sup_{P\in\mathcal{P}_{\mathrm{sat}}}\ell(P) =\ell_{\mathrm{sat}}.

Subtracting $\ell_M$ and multiplying by two proves $D(M)\geq0$. If both maxima are attained, their difference is zero precisely when the maximum over $\mathcal{P}_M$ equals the saturated maximum. In that case a maximizing configuration in $M$ attains the saturated maximum as well. $\square$

This result explains the sign and scale in the definition. Reversing the likelihood gap would produce a nonpositive quantity. The factor of two is conventional and useful in likelihood-based inference, but the value remains a likelihood comparison—not, by itself, a calibrated measure of statistical significance.

Exponential-Family Form

A familiar expression for deviance follows when the independent responses have a common, fixed dispersion parameter $\phi>0$ and densities or mass functions of the form

f(y_i;\theta_i,\phi) = \exp\left\{ \frac{y_i\theta_i-b(\theta_i)}{\phi} +c(y_i,\phi) \right\}.

Here $\theta_i$ is the natural parameter, and the mean is $b'(\theta_i)$. Suppose that, for each observation, the saturated-model likelihood is maximized at $\theta_{i,\mathrm{sat}}$, or at a limiting boundary value if needed. Let $\hat{\theta}_i$ denote the fitted natural parameter under $M$. The terms $c(y_i,\phi)$ are identical in the two likelihoods, so they cancel.

Result: Under the stated common-dispersion exponential-family form, $$ D(M)=\frac{2}{\phi}\sum_{i=1}^{n} \left[ y_i(\theta_{i,\mathrm{sat}}-\hat{\theta}_i) -b(\theta_{i,\mathrm{sat}}) +b(\hat{\theta}_i) \right]. $$ The expression is understood by a limit when a saturated parameter is on the boundary.

Derivation. Independence makes the log likelihood the sum of the individual log likelihoods. For observation $i$, the saturated-minus-fitted log-likelihood contribution is

\begin{aligned} &\log f(y_i;\theta_{i,\mathrm{sat}},\phi) -\log f(y_i;\hat{\theta}_i,\phi)\\ &\quad = \frac{y_i\theta_{i,\mathrm{sat}}-b(\theta_{i,\mathrm{sat}})}{\phi} -\frac{y_i\hat{\theta}_i-b(\hat{\theta}_i)}{\phi}, \end{aligned}

because the two copies of $c(y_i,\phi)$ cancel. Rearranging the numerator gives $y_i(\theta_{i,\mathrm{sat}}-\hat{\theta}_i)-b(\theta_{i,\mathrm{sat}})+b(\hat{\theta}_i)$. Summing over observations gives $\ell_{\mathrm{sat}}-\ell_M$; multiplying by two yields the stated formula. $\square$

Some presentations separate the factor $1/\phi$ and call the sum before scaling the unscaled deviance, especially when $\phi$ is not one. Here deviance always means the twice-log-likelihood gap as defined above, so any dispersion factor present in the likelihood is included. When the dispersion is estimated, the defining likelihood gap still applies, but the fixed-$\phi$ expression above need not describe the result after profiling or maximizing over dispersion.

Worked Examples

Worked Example: Poisson Counts With a Common Mean

Suppose two independent counts are $y_1=1$ and $y_2=3$, and the fitted model is an intercept-only Poisson GLM. Calculate its deviance relative to the saturated Poisson model.

The fitted common mean is the sample average: $\hat{\lambda}=(1+3)/2=2$. In the saturated model, the separate Poisson means maximize their likelihoods at $\lambda_{1,\mathrm{sat}}=1$ and $\lambda_{2,\mathrm{sat}}=3$. For a Poisson observation, the log likelihood is $y\log\lambda-\lambda-\log(y!)$. The factorial terms cancel in the likelihood gap, and the fitted and saturated means sum to the same total, so the linear mean terms cancel too. Therefore

\begin{aligned} D &=2\left[1\log\left(\frac{1}{2}\right) +3\log\left(\frac{3}{2}\right)\right]\\ &=2\left[-0.693147+3(0.405465)\right]\\ &=2(0.523248)\\ &\approx 1.0465. \end{aligned}

The deviance is positive because the common-mean restriction loses likelihood compared with fitting each count’s mean separately. The value is twice that log-likelihood loss; it is not the unmultiplied bracket.

Worked Example: Grouped Binomial Counts

There are two groups, each with $m_i=4$ trials. Their success counts are $y_1=3$ and $y_2=1$. The fitted model imposes a common success probability, while the saturated model gives the groups separate probabilities. Find the deviance.

The fitted probability is the pooled proportion, $\hat p=(3+1)/(4+4)=1/2$. The saturated probabilities are $\hat p_{1,\mathrm{sat}}=3/4$ and $\hat p_{2,\mathrm{sat}}=1/4$. For a group with $y$ successes in $m$ trials, its log likelihood, apart from the binomial coefficient, is $y\log p+(m-y)\log(1-p)$. The binomial coefficients do not depend on $p$ and cancel from the gap. Thus

\begin{aligned} D=2\bigg[ &3\log\left(\frac{3/4}{1/2}\right) +\log\left(\frac{1/4}{1/2}\right)\\ &+\log\left(\frac{1/4}{1/2}\right) +3\log\left(\frac{3/4}{1/2}\right) \bigg]. \end{aligned}

Each group contributes $3\log(1.5)+\log(0.5)=1.216395-0.693147=0.523248$. There are two such contributions, so

D=2(0.523248+0.523248) =2(1.046496) \approx 2.0930.

The group contributions match because the success and failure counts are exchanged between groups, and the saturated probabilities are exchanged in the same way. This is a grouped-binomial calculation: the two groups, not eight individually modelled Bernoulli means, define the observations being compared.

Worked Example: Gaussian Responses With a Common Mean

Let $Y_1,Y_2$ be independent normal responses with known variance $\sigma^2=2$, observed as $y_1=1$ and $y_2=3$. The fitted model has one common mean, while the saturated model has one mean per observation. Calculate the deviance.

The fitted mean is $\hat\mu=(1+3)/2=2$. The fitted residual sum of squares is $(1-2)^2+(3-2)^2=2$. The saturated means equal the observed values, so their residual sum of squares is zero. With known variance, the Gaussian log likelihood is a constant minus the residual sum of squares divided by $2\sigma^2$. Hence

\ell_{\mathrm{sat}}-\ell_M =\frac{2-0}{2(2)} =\frac{1}{2}, \qquad D=2\left(\frac{1}{2}\right)=1.

Equivalently, $D=\mathrm{SSE}/\sigma^2=2/2=1$. This shows directly how the likelihood-gap definition incorporates the known variance: the same residual sum of squares would yield a different deviance under a different fixed variance.

Interpreting Deviance Carefully

A smaller deviance means that the fitted model’s maximized likelihood is closer to the saturated-model maximum, on the scale of twice the log-likelihood gap. This supports a relative interpretation: among models with the same response data and likelihood convention, the one with lower deviance has lost less likelihood relative to the saturated reference.

The word “distance” is a useful metaphor, but deviance is not a metric. It is not a distance between parameter values, and it need not obey geometric distance properties. Its value depends on the observed data, response distribution, and likelihood specification. A small value does not by itself certify that a model is scientifically adequate, while a large value does not have a universal interpretation without an appropriate reference or calibration.

Common Mistakes

Reversing the gap. Using $2(\ell_M-\ell_{\mathrm{sat}})$ gives the wrong sign. The saturated likelihood is at least as large, so deviance is $2(\ell_{\mathrm{sat}}-\ell_M)$.
Forgetting the factor of two. First compute the log-likelihood gap, then multiply the entire gap by two. If a bracket evaluates to $0.523248$, the deviance is $1.046496$, not $0.523248$.
Assuming the saturated model always means a perfect prediction. It means maximizing the likelihood separately for each observation within the response family. Boundary estimates may be involved, and the fitted distribution need not assign probability one to the observed data.
Treating a deviance value as a standalone test conclusion. Deviance is defined by a likelihood comparison. Its inferential calibration depends on additional conditions and a specified reference; the value alone does not supply those.

Key takeaway: GLM deviance is twice the log-likelihood loss incurred by fitting a restricted model instead of the saturated reference. It is nonnegative and useful for describing relative likelihood fit, but it is not, on its own, a universal measure of adequacy or a calibrated significance test.

Check Your Understanding

Try these questions to check your understanding of deviance as a saturated-model likelihood gap.

In one sentence, define the saturated model used to define GLM deviance.
Why must deviance be nonnegative when the fitted model’s allowed distributions are contained in the saturated model’s allowed distributions?
For the exponential-family likelihood in this tutorial, identify which terms cancel when forming the saturated-minus-fitted log-likelihood gap.
In the Poisson example, what is the log-likelihood gap before multiplication by two, and how does it give the reported deviance?
Explain why deviance is not a distance between regression coefficient vectors.
Does a deviance value by itself establish that a model fits adequately? Explain briefly.