Differentiating Exponential-Dispersion Likelihoods

Why the Dispersion Score Needs Two Contributions

In a generalized linear model, the mean describes the location of the response distribution, while the dispersion controls its spread. When differentiating a likelihood with respect to dispersion while holding the mean parameters fixed, the mean-dependent canonical parameter does not change. The likelihood can still change through two distinct routes: dispersion rescales the part involving the response and the cumulant function, and the normalizing term can itself depend on dispersion.

The second route is easy to overlook. The cumulant function \(b\) is part of the exponential-family kernel, whereas the term \(c(y,\phi)\) completes the density or mass function and often contains important dispersion-dependent factors. A correct score keeps both contributions.

Definition: An exponential-dispersion density or mass function has the form \(f(y;\theta,\phi)=\exp\{[y\theta-b(\theta)]/a(\phi)+c(y,\phi)\}\), where \(\theta\) is the canonical parameter, \(\phi>0\) is the dispersion parameter, \(a(\phi)>0\) is differentiable, and \(c\) is the remaining log-normalizing term. The dispersion score at fixed \(\theta\) is the partial derivative of the log-likelihood with respect to \(\phi\), treating \(y\) and \(\theta\) as constant.

For this differentiation, assume the observed value \(y\) is in the support of the model and that the support does not change with \(\phi\). Assume also that \(a\) and \(c(y,\cdot)\) are differentiable at the value of \(\phi\) under consideration. These conditions allow an ordinary pointwise derivative of the log-likelihood. No differentiation under an integral is needed for the score formula below.

The Fixed-Mean Dispersion Score

For a single observation, write \(A(y,\theta)=y\theta-b(\theta)\). Once \(\theta\) is held fixed, \(A\) is constant with respect to \(\phi\). The log-likelihood is \(A/a(\phi)+c(y,\phi)\), so the chain rule gives the score.

Theorem: Under the differentiability conditions above, the dispersion score for one observation, holding \(\theta\) fixed, is \(U_\phi(y;\theta,\phi)=-\dfrac{a'(\phi)}{a(\phi)^2}[y\theta-b(\theta)]+\dfrac{\partial c(y,\phi)}{\partial\phi}\). The first term is the contribution from dispersion scaling; the second is the contribution from the log-normalizing term.

Proof. Taking the logarithm of the density gives \(\ell(\phi)=[y\theta-b(\theta)]/a(\phi)+c(y,\phi)\). Since \(y\) and \(\theta\) are fixed, the numerator \(y\theta-b(\theta)\) is constant. Differentiating \(a(\phi)^{-1}\) gives \(-a'(\phi)/a(\phi)^2\), and differentiating the remaining term gives \(\partial c(y,\phi)/\partial\phi\). Adding the two derivatives proves the result. \(\square\)

A common parameterization is \(a(\phi)=\phi\). Then the formula becomes \[ U_\phi=-\frac{y\theta-b(\theta)}{\phi^2}+\frac{\partial c(y,\phi)}{\partial\phi}. \] The negative sign in the first term comes from differentiating \(1/\phi\). It does not mean that the entire score must be negative: the normalizing-term derivative can reinforce or offset it.

\ell(\phi)=\frac{A}{\phi}+c(y,\phi) \quad\Longrightarrow\quad U_\phi=-\frac{A}{\phi^2}+c_\phi(y,\phi), \qquad A=y\theta-b(\theta).

For independent observations with fixed canonical parameters \(\theta_i\), let the likelihood be the product of the observation-specific densities. If observation \(i\) has scaling \(a_i(\phi)\) and log-normalizing term \(c_i(y_i,\phi)\), the log-likelihood is a sum. Differentiating each summand gives the total score.

Corollary: For independent observations with log-likelihood \(\ell(\phi)=\sum_{i=1}^n\{[y_i\theta_i-b(\theta_i)]/a_i(\phi)+c_i(y_i,\phi)\}\), where each \(\theta_i\) is held fixed, the dispersion score is \(U_\phi=\sum_{i=1}^n\{-a_i'(\phi)[y_i\theta_i-b(\theta_i)]/a_i(\phi)^2+\partial c_i(y_i,\phi)/\partial\phi\}\).

Proof. The logarithm of the product likelihood is the sum of its log-likelihood contributions. The derivative of a finite sum is the sum of its derivatives. Apply the single-observation result to each contribution, using its own \(a_i\) and \(c_i\), and sum over \(i=1,\ldots,n\). \(\square\)

In the frequently used weighted form \(a_i(\phi)=\phi/w_i\), with fixed positive weights \(w_i\), differentiation of the scaling gives \(-w_i[y_i\theta_i-b(\theta_i)]/\phi^2\). The derivative of \(c_i(y_i,\phi)\) remains in the score; weights may also appear in that term. One should differentiate the actual log-likelihood rather than infer the full score from its kernel alone.

U_\phi=\sum_{i=1}^n\left\{ -\frac{w_i[y_i\theta_i-b(\theta_i)]}{\phi^2} +\frac{\partial c_i(y_i,\phi)}{\partial\phi} \right\}, \qquad a_i(\phi)=\frac{\phi}{w_i}.

Worked Calculations

Worked Example: Gaussian Observation With Known Mean

Suppose \(Y\sim N(\mu,\phi)\), with \(\mu=2\) held fixed, and observe \(y=3\). Find the dispersion score at \(\phi=2\). The Gaussian log-density is \(\ell(\phi)=-\tfrac12\log(2\pi\phi)-(y-\mu)^2/(2\phi)\).

Differentiate the logarithmic term and the residual term separately: \[ \frac{\partial}{\partial\phi}\left[-\frac12\log(2\pi\phi)\right]=-\frac{1}{2\phi}, \qquad \frac{\partial}{\partial\phi}\left[-\frac{(y-\mu)^2}{2\phi}\right]=\frac{(y-\mu)^2}{2\phi^2}. \]

Here \(y-\mu=1\), so at \(\phi=2\), \[ U_\phi=-\frac14+\frac{1}{8}=-\frac18. \] The normalizing term contributes \(-1/4\); the dispersion-scaled residual term contributes \(1/8\). Omitting the log-normalizing term would give the wrong score.

Worked Example: Two Weighted Gaussian Observations

Let \(Y_i\sim N(\mu_i,\phi/w_i)\) independently, with fixed means \(\mu_1=0,\mu_2=2\), weights \(w_1=2,w_2=1\), and observed values \(y_1=1,y_2=3\). Calculate the total score at \(\phi=1\).

For a Gaussian observation with variance \(\phi/w_i\), its log-density is \[ \ell_i=-\frac12\log(2\pi\phi/w_i)-\frac{w_i(y_i-\mu_i)^2}{2\phi}. \] The derivative is therefore \[ U_{\phi,i}=-\frac{1}{2\phi}+\frac{w_i(y_i-\mu_i)^2}{2\phi^2}. \]

Both residuals equal \(1\). For observation 1, \(U_{\phi,1}=-1/2+2/2=1/2\) at \(\phi=1\). For observation 2, \(U_{\phi,2}=-1/2+1/2=0\). Thus the total score is \(U_\phi=1/2+0=1/2\). The normalizing contributions sum to \(-1\), and the residual contributions sum to \(3/2\); together they give \(1/2\).

Worked Example: Gamma Observation and a Dispersion-Dependent Normalizer

Let \(Y\) have a gamma distribution with mean \(\mu\) and variance \(\phi\mu^2\), where \(\mu=1.5\) is fixed. Its shape is \(1/\phi\) and its scale is \(\mu\phi\). For an observed \(y=2\), calculate the dispersion score at \(\phi=0.5\).

In exponential-dispersion form, \(\theta=-1/\mu\), \(b(\theta)=-\log(-\theta)=\log\mu\), and \(a(\phi)=\phi\). The remaining term is \[ c(y,\phi)=\left(\frac1\phi-1\right)\log y-\frac{\log\phi}{\phi}-\log\Gamma(1/\phi). \] Differentiating this term gives \[ c_\phi(y,\phi)=\frac{-\log y+\log\phi-1+\psi(1/\phi)}{\phi^2}, \] where \(\psi(x)=\Gamma'(x)/\Gamma(x)\) is the digamma function. The scaling contribution is \[ -\frac{y\theta-b(\theta)}{\phi^2} =\frac{y/\mu+\log\mu}{\phi^2}. \]

Combining the two contributions, \[ U_\phi= \frac{y/\mu+\log\mu-\log y+\log\phi-1+\psi(1/\phi)}{\phi^2}. \] At the stated values, \(y/\mu=4/3\), \(\log\mu\approx0.405465\), \(\log y\approx0.693147\), \(\log\phi\approx-0.693147\), and \(\psi(2)=1-\gamma\approx0.422784\). The numerator is approximately \(1.333333+0.405465-0.693147-0.693147-1+0.422784=-0.224712\). Since \(\phi^2=0.25\), the score is approximately \(-0.89885\). In particular, differentiating only the scaling term would miss the gamma normalizer's contribution.

When There Is No Free Dispersion Score

Not every GLM likelihood has an estimable continuous dispersion parameter. For example, the ordinary Poisson model has \(Y\sim\operatorname{Poisson}(\mu)\), with its dispersion fixed by the distribution rather than represented by a free \(\phi\). In that model there is no ordinary likelihood derivative with respect to a free dispersion parameter. A dispersion score is meaningful only when the specified likelihood actually includes one.

For models that do include \(\phi\), keep the differentiation path clear. Holding the mean parameters fixed means that \(\mu_i\), and therefore \(\theta_i\) when determined by \(\mu_i\), are constants in this partial derivative. It does not mean that all terms involving \(b(\theta_i)\) disappear: \(y_i\theta_i-b(\theta_i)\) is multiplied by a dispersion-dependent factor, so that factor still contributes to the score.

Common Mistakes and a Reliable Check

Differentiating only the exponential-family kernel. This drops \(\partial c(y,\phi)/\partial\phi\), which can be substantial. Differentiate the complete log-density.
Treating the kernel numerator as zero because the mean is fixed. The numerator is constant in \(\phi\), not generally zero. Its product with \(1/a(\phi)\) has a nonzero derivative.
Forgetting the negative sign. When \(a(\phi)=\phi\), the derivative of \(1/\phi\) is \(-1/\phi^2\).
Allowing the mean to vary during a fixed-mean derivative. If \(\theta\) is held fixed, do not add a chain-rule term involving \(d\theta/d\phi\). Such terms belong to differentiation along a path where the mean changes.

Key takeaway: At fixed mean parameters, differentiate the log-likelihood as written. The dispersion score has a scaling contribution from \(1/a(\phi)\) and a normalizer contribution from \(c(y,\phi)\); neither can be discarded without checking the model's actual formula.

Check Your Understanding

Use the full log-likelihood in each calculation, and keep fixed the parameters specified as fixed.

For \(a(\phi)=\phi\), write the single-observation dispersion score in terms of \(y\theta-b(\theta)\) and \(c_\phi(y,\phi)\).
In the weighted parameterization \(a_i(\phi)=\phi/w_i\), what is the scaling contribution for observation \(i\)?
Why is the derivative of \(c(y,\phi)\) needed even when \(\theta\) is held fixed?
For a Gaussian observation with known mean, identify which part of the score comes from the log-variance term.
Why does an ordinary Poisson likelihood not have a free dispersion score?