Deviance From the Saturated Model

Why Compare a Model With a Saturated Fit?

A generalized linear model summarizes the response data using a structured set of parameters, such as regression coefficients. To measure the loss in likelihood caused by that structure, compare the fitted model with a reference model that gives each independent observation its own parameter. This reference is the saturated model: it fits each observed response as well as the sampling model allows.

Deviance records the resulting likelihood gap. It is a measure of how much log-likelihood the fitted model gives up relative to that observation-by-observation reference. Its definition is simple, but the exact numerical value depends on the likelihood convention, particularly when the model has a dispersion parameter.

Definition: Let $\ell(\hat{\theta})$ be the maximized log-likelihood under a fitted model, and let $\ell_{\mathrm{sat}}$ be the saturated log-likelihood: the maximum, or supremum if a maximum is not attained, obtained by fitting each independent likelihood contribution separately. The deviance from the saturated model is $$ D=2\{\ell_{\mathrm{sat}}-\ell(\hat{\theta})\}. $$ The saturated model must use the same response likelihood and data as the fitted model. It is a reference fit, not a claim that the data-generating process truly has one free parameter for every observation.

The relevant observation is the unit contributing an independent factor to the likelihood. For individual Bernoulli data, that unit is one binary response. For grouped binomial data, it is one group’s count and number of trials. A saturated fit assigns separate parameters to those likelihood contributions; it does not necessarily assign one parameter to every person represented by a grouped record.

Exponential-Family Form of the Likelihood Gap

Suppose the independent responses $Y_1,\ldots,Y_n$ have log-likelihood contributions of the exponential-dispersion form $$ \ell_i(\theta_i;\phi)=\frac{y_i\theta_i-b(\theta_i)}{a(\phi)}+c(y_i,\phi), $$ where $b$ and $c$ are specified functions, $\phi$ is a common fixed dispersion parameter, and $a(\phi)>0$. The fitted model supplies natural parameters $\hat{\theta}_i$. A saturated contribution uses a natural parameter $\theta_{i,\mathrm{sat}}$ that maximizes the contribution for response $y_i$, or approaches its supremum when the maximum is on the boundary of the parameter space.

Result: For this likelihood, the deviance is $$ D=\frac{2}{a(\phi)}\sum_{i=1}^{n} \left[ y_i(\theta_{i,\mathrm{sat}}-\hat{\theta}_i) -b(\theta_{i,\mathrm{sat}})+b(\hat{\theta}_i) \right]. $$ The terms $c(y_i,\phi)$ cancel because the saturated and fitted likelihoods use the same observed data and dispersion parameter.

Here is the derivation. Write the saturated log-likelihood as the sum of its independently maximized contributions, and subtract the fitted log-likelihood. For each observation, $$ \ell_i(\theta_{i,\mathrm{sat}};\phi)-\ell_i(\hat{\theta}_i;\phi) = \frac{y_i(\theta_{i,\mathrm{sat}}-\hat{\theta}_i)-b(\theta_{i,\mathrm{sat}})+b(\hat{\theta}_i)}{a(\phi)}, $$ because the two occurrences of $c(y_i,\phi)$ are identical and cancel. Summing over $i$ and multiplying by $2$ gives the displayed expression. If the maximum occurs only as a boundary limit, the same subtraction applies to the supremum by taking that limit in the likelihood contribution.

For an interior saturated fit in a regular one-parameter exponential family, the natural parameter satisfies $b'(\theta_{i,\mathrm{sat}})=y_i$, since differentiating $y_i\theta-b(\theta)$ with respect to $\theta$ gives $y_i-b'(\theta)$. The expression above does not require that an interior solution exist: boundary cases are interpreted through the saturated supremum.

Terminology: The expression $$ d_i=2\left[ y_i(\theta_{i,\mathrm{sat}}-\hat{\theta}_i) -b(\theta_{i,\mathrm{sat}})+b(\hat{\theta}_i) \right] $$ is commonly called the unit deviance contribution. Under the likelihood written above, the deviance defined as twice the actual log-likelihood difference is $D=\sum_i d_i/a(\phi)$. When $a(\phi)=1$, these are the same. Some conventions call $\sum_i d_i$ the deviance and treat division by $\phi$ separately; always check which likelihood-based quantity is being reported.

Two Basic Properties

The saturated fit gives each independent contribution at least as much freedom as the fitted model. The following property makes that comparison precise.

Theorem: If every parameter value allowed by the fitted model is also allowed by the saturated model, then $D\geq 0$. Moreover, $D=0$ if and only if the fitted model attains the saturated log-likelihood (or, when using suprema, if its log-likelihood equals the saturated supremum).

Proof. By definition, $\ell_{\mathrm{sat}}$ is the maximum or supremum over the saturated parameter space. Because that space includes the fitted model’s allowable likelihood values, $\ell_{\mathrm{sat}}\geq \ell(\hat{\theta})$. Multiplication of this nonnegative difference by $2$ proves $D\geq0$. The difference is zero exactly when the fitted log-likelihood equals the saturated maximum or supremum. $\square$

This result concerns the likelihood gap, not the closeness of individual fitted values to observed values on every possible scale. A large deviance indicates a larger loss in log-likelihood relative to the saturated benchmark, but its practical interpretation depends on the response distribution and sampling design.

Worked Examples

Worked Example: Poisson Counts With a Common Fitted Mean

Suppose three independent Poisson counts are $y=(0,2,3)$, and a model constrains them to have a common mean $\mu$. Find the fitted mean and the deviance from the saturated model.

For a Poisson observation, the log-likelihood, apart from the term $-\log(y!)$, is $y\log\mu-\mu$. The common-mean maximum-likelihood estimate is the sample mean: $$ \hat{\mu}=\frac{0+2+3}{3}=\frac{5}{3}. $$ In the saturated fit, each observation has its own Poisson mean, equal to its observed count when that count is positive. For $y=0$, the saturated mean is $0$, understood as a boundary value. Subtracting the fitted log-likelihood from the saturated one cancels the $-\log(y!)$ terms and gives $$ D=2\sum_{i=1}^{3}\left\{y_i\log\left(\frac{y_i}{\hat{\mu}}\right)-(y_i-\hat{\mu})\right\}, $$ with $0\log(0/\hat{\mu})=0$, as defined by its limit. The linear terms sum to $5-3(5/3)=0$. Therefore $$ D=2\left\{2\log\left(\frac{2}{5/3}\right)+3\log\left(\frac{3}{5/3}\right)\right\} =2\{2\log(1.2)+3\log(1.8)\} \approx 4.256. $$

Worked Example: Grouped Binomial Counts

Two independent groups each have 10 trials. Their observed success counts are 3 and 8. Compare a model with a common success probability to the saturated model with a separate probability for each group.

Under the common-probability model, the maximum-likelihood estimate pools the successes: $$ \hat{p}=\frac{3+8}{10+10}=\frac{11}{20}=0.55. $$ The saturated estimates are $0.3$ and $0.8$. The binomial coefficients are the same under both fits, so they cancel in the likelihood difference. The deviance is $$ D=2\left[ 3\log\left(\frac{0.3}{0.55}\right)+7\log\left(\frac{0.7}{0.45}\right) +8\log\left(\frac{0.8}{0.55}\right)+2\log\left(\frac{0.2}{0.45}\right) \right]. $$ The four terms inside the brackets are approximately $-1.8184,\ 3.0928,\ 2.9975,$ and $-1.6219$. Their sum is approximately $2.6500$, so $D\approx 5.300$. The negative terms do not imply a negative deviance: the complete likelihood gap, including all contributions, is positive.

Worked Example: Normal Responses With Known Variance

Let $Y_1,Y_2,Y_3$ be independent $N(\mu,2)$ observations with values $1,2,4$. The fitted model has one common mean, while the saturated model has a separate mean for each observation. Find the likelihood deviance.

The common-mean estimate is $\hat{\mu}=(1+2+4)/3=7/3$. In the saturated model, each mean equals its observation. Since the variance is fixed at $2$, the normal log-likelihood difference gives $$ D=\frac{1}{2}\sum_{i=1}^{3}(y_i-\hat{\mu})^2 =\frac{1}{2}\left[ \left(1-\frac{7}{3}\right)^2+ \left(2-\frac{7}{3}\right)^2+ \left(4-\frac{7}{3}\right)^2 \right]. $$ The squared differences are $16/9,\ 1/9,$ and $25/9$, which sum to $42/9=14/3$. Thus $D=(1/2)(14/3)=7/3$. The corresponding unscaled sum of squared errors is $14/3$; the likelihood deviance is $14/3$ divided by the variance $2$.

Interpreting the Scale and Avoiding a Common Error

These examples show why the likelihood definition should be kept in view. For Poisson and binomial models with their usual fixed dispersion, the likelihood deviance is the familiar sum of unit contributions. For a normal model with known variance $\sigma^2$, twice the log-likelihood gap is the residual sum of squares divided by $\sigma^2$. Calling the unscaled sum of squares “the deviance” is common when using the unit-deviance convention, but it is not the same numerical quantity as twice the log-likelihood difference unless the scale factor is one.

Common mistake: “The saturated model has one parameter for every data point, so its log-likelihood must be infinite; therefore deviance is infinite.” This confuses flexibility with an unbounded likelihood. In regular models such as Poisson and binomial sampling, the saturated likelihood is finite for valid observations, although a maximizing parameter may lie on the boundary. The correction is to maximize each likelihood contribution over its allowed parameter values and, when necessary, use the supremum. For example, a saturated Poisson mean of zero for an observed zero count gives a finite likelihood contribution; it does not make the likelihood infinite.

Deviance is useful because it puts a structured fit beside a clear likelihood benchmark and yields a nonnegative measure of the likelihood lost through model restrictions. Its definition alone does not supply a universal threshold for deciding whether a value is large: that requires additional distributional and inferential arguments. It is also not a raw residual sum of squares in every model, nor does a single deviance value identify which observations account for the discrepancy.

Key takeaway: Deviance from the saturated model is exactly twice the log-likelihood difference, $2(\ell_{\mathrm{sat}}-\ell(\hat{\theta}))$. Derive its contribution from the likelihood being used, and keep any dispersion scaling explicit.

Check Your Understanding

Answer each question using the likelihood definition of deviance.

What makes a model saturated, and what likelihood units receive separate parameters?
Why do the terms $c(y_i,\phi)$ cancel when deriving the exponential-family deviance?
For a Poisson observation with $y_i=0$, how is the term $y_i\log(y_i/\hat{\mu}_i)$ interpreted?
In the normal example, why is the likelihood deviance not equal to the unscaled residual sum of squares?
What condition on the saturated and fitted likelihood spaces guarantees that deviance is nonnegative?