Unit Deviance from Saturated-Model Likelihoods

Introduction

Pearson residuals measure a response’s discrepancy from its fitted mean after scaling by a variance estimate. Deviance measures discrepancy another way: it compares the fitted likelihood with the likelihood attainable when each observation is given its own mean. This comparison connects a familiar model fit to a precise likelihood calculation.

The key quantity for one observation is its unit deviance. It is nonnegative because the saturated model is allowed to fit that observation at least as well as a model that constrains its mean. The contribution to the model’s total deviance also accounts for the observation’s weight.

From the Saturated Likelihood to Unit Deviance

Consider a one-parameter exponential-dispersion family with density or mass function

f(y;\theta,\phi,w) = a(y,\phi,w)\exp\left\{\frac{w}{\phi}\bigl(y\theta-b(\theta)\bigr)\right\}.

Here \(\theta\) is the natural parameter, \(\phi>0\) is the dispersion parameter, and \(w>0\) is an observation weight. The function \(a\) contains terms that do not depend on \(\theta\). Assume \(b\) is strictly convex and differentiable on the natural-parameter space. The mean is \(\mu=b'(\theta)\). When the observed value \(y\) is an attainable mean, write \(\theta_y\) for a natural parameter satisfying \(b'(\theta_y)=y\).

The saturated model gives this observation its own mean, so its natural parameter is \(\theta_y\) when that parameter exists. For an interior response, the log-likelihood difference between this saturated fit and a fit with mean \(\mu\) is

\ell_{\mathrm{sat}}-\ell(\theta) = \frac{w}{\phi} \left\{ y(\theta_y-\theta)-b(\theta_y)+b(\theta) \right\}.

Definition: For an interior response \(y\), the unit deviance is \(d(y,\mu)=2\{y(\theta_y-\theta)-b(\theta_y)+b(\theta)\}\), where \(b'(\theta_y)=y\) and \(b'(\theta)=\mu\). Thus, twice the log-likelihood gap for this observation is \(w d(y,\mu)/\phi\). At a boundary response, define the saturated log-likelihood as the supremum over the natural-parameter space and define the deviance by the corresponding limit when that limit exists. A finite \(\theta_y\) is not required at a boundary.

The factor \(w/\phi\) matters. The unit deviance describes the family’s discrepancy on its unit scale; the observation’s contribution to the model deviance is \(w d(y,\mu)\). With this convention, the model deviance is \(D=\sum_i w_i d(y_i,\mu_i)\), and twice the total log-likelihood gap is \(D/\phi\). In particular, when \(d\ne 0\), twice the gap for one observation equals its unit deviance when \(w=\phi\); \(w=\phi=1\) is one such case. If \(d=0\), they are equal for any positive \(w\) and \(\phi\).

Why the Contribution Cannot Be Negative

The saturated fit maximizes the likelihood for the observation over its possible mean values. For an interior response, this likelihood comparison can also be checked directly by convexity.

Theorem: Suppose \(y\) is an attainable interior mean and \(b\) is strictly convex and differentiable on the natural-parameter space. Then \(d(y,\mu)\ge 0\) for every fitted mean \(\mu\) in the family. If \(b\) is strictly convex, equality holds only when \(\mu=y\).

Apply the supporting-line inequality.
Convexity at \(\theta_y\) gives \(b(\theta)\ge b(\theta_y)+b'(\theta_y)(\theta-\theta_y)\).

Substitute the saturated mean.
Since \(b'(\theta_y)=y\), rearranging yields \(y(\theta_y-\theta)-b(\theta_y)+b(\theta)\ge 0\).

Multiply by two.
The resulting expression is \(d(y,\mu)\), so the unit deviance is nonnegative. Strict convexity makes the inequality strict when \(\theta\ne\theta_y\); strict monotonicity of \(b'\) then gives \(\mu\ne y\).

This proves the result for interior responses. For a boundary response, the saturated likelihood is a supremum or limit rather than necessarily a maximum at a finite natural parameter. Since each interior likelihood gap is nonnegative, any finite boundary limit of those gaps is also nonnegative.

Worked Calculations

Worked Example: A Poisson Observation at the Boundary

Let \(Y\) have a Poisson distribution with fitted mean \(\mu=2.5\), and suppose the observed count is \(y=0\). For a Poisson model, \(\theta=\log\mu\), \(b(\theta)=e^\theta\), and, apart from terms constant in \(\theta\), the log-likelihood for this observation is \(y\theta-e^\theta\).

At \(y=0\), there is no finite natural parameter satisfying \(b'(\theta_y)=0\), because \(b'(\theta)=e^\theta>0\) for every finite \(\theta\). But as \(\theta\) tends to \(-\infty\), the fitted mean tends to zero and \(y\theta-e^\theta=-e^\theta\) tends to zero. Thus the saturated log-likelihood, omitting the common constant, is the supremum \(0\).

At the fitted mean, the log-likelihood is \(-2.5\). The log-likelihood gap is therefore \(0-(-2.5)=2.5\), and twice that gap is \(5\). Equivalently, the Poisson unit deviance at this boundary response is \(d(0,2.5)=2(2.5)=5\). This boundary value is obtained by a limit, not by substituting a nonexistent finite \(\theta_y\).

Worked Example: Grouped Binomial Data and Observation Weight

Suppose 7 of 10 independent trials are successes, and the fitted success probability is \(p=0.5\). Write the observed proportion as \(y=7/10=0.7\). In the grouped-binomial representation, the trial count \(w=10\) is the weight, \(\phi=1\), and \(b(\theta)=\log(1+e^\theta)\), with \(\theta=\log\{p/(1-p)\}\).

The saturated probability is \(0.7\). The unit deviance per trial is

d(0.7,0.5) = 2\left\{ 0.7\log\left(\frac{0.7}{0.5}\right) + 0.3\log\left(\frac{0.3}{0.5}\right) \right\}.

Using \(\log(1.4)\approx0.3364722366\) and \(\log(0.6)\approx-0.5108256238\), the bracket is approximately \(0.7(0.3364722366)+0.3(-0.5108256238)=0.0822828785\). Hence \(d(0.7,0.5)\approx0.164565757\). The weighted contribution is \(w d\approx10(0.164565757)=1.64565757\), or \(1.64566\) to five decimal places. Because \(\phi=1\), this is also twice the log-likelihood gap for the grouped observation.

The distinction is important: \(0.164565757\) is the unit deviance per trial, while \(1.64566\) is the contribution from all 10 trials represented by this grouped observation. They are not competing calculations; the weight connects them.

Worked Example: Gaussian Data with Weight and Dispersion

Let \(Y\) be Gaussian with mean \(\mu\) and variance \(\phi/w\), where \(w=3\) and \(\phi=2\). Suppose the observed value is \(y=4\), while the fitted mean is \(\mu=2.5\). For a Gaussian family, the unit deviance is \(d(y,\mu)=(y-\mu)^2\). Therefore,

$$ d(4,2.5)=(4-2.5)^2=2.25. $$

The weighted contribution to model deviance is \(w d=3(2.25)=6.75\). The log-likelihood gap is \(w d/(2\phi)=6.75/4=1.6875\), so twice the gap is \(3.375\). This also equals \(w d/\phi=6.75/2=3.375\), as required. The unit deviance, weighted contribution, and twice the likelihood gap differ because the observation has weight 3 and dispersion 2.

What the Likelihood Comparison Tells Us

The saturated model is a reference point: it asks how much likelihood is lost by imposing the fitted mean instead of allowing this observation its own best-fitting mean. A zero contribution means the fitted mean matches the saturated mean in the interior case. A larger contribution means a larger likelihood loss, on the scale specified by the family. The contribution is not itself a signed measure of whether the observation is above or below the fitted mean.

Deviance is useful because the same likelihood-based construction applies across several response distributions. The resulting numerical scales are family-specific, however, and weights and dispersion must be kept explicit when relating deviance to a log-likelihood difference.

Common Mistakes

Assuming every saturated fit has a finite natural parameter. For Poisson data with \(y=0\), no finite \(\theta_y\) gives mean zero. Use the supremum or appropriate boundary limit instead.
Dropping the weight or dispersion factor. Twice the observation’s log-likelihood gap is \(w d/\phi\), not automatically \(d\). The weighted contribution to model deviance is \(w d\).
Calling the unit deviance negative when a response is below its fitted mean. Unit deviance is a likelihood gap and is nonnegative. Direction is not encoded by its sign.
Confusing a grouped contribution with a per-unit deviance. For 10 binomial trials, the grouped contribution is 10 times the unit deviance per trial.

Key takeaway: Unit deviance is twice the per-unit likelihood loss relative to the saturated fit. For an observation with weight \(w\) and dispersion \(\phi\), twice its log-likelihood gap is \(w d(y,\mu)/\phi\). The contribution is nonnegative, including at a boundary when defined by a finite likelihood limit.

Check Your Understanding

Use the likelihood-gap definition and keep weights and dispersion explicit.

For an interior response, which two natural parameters appear in the unit-deviance formula, and what means do they represent?
Why can the Poisson saturated fit for an observed zero be defined even though there is no finite natural parameter with mean zero?
If an observation has unit deviance \(d=1.2\), weight \(w=4\), and dispersion \(\phi=2\), what is its weighted deviance contribution and what is twice its log-likelihood gap?
In the grouped-binomial example, why is the contribution for all 10 trials larger than the unit deviance per trial?
What property of \(b\) gives the nonnegativity of the interior unit deviance?