Common Errors In GLM Score And Hessian Derivations

Why Small Derivative Errors Matter

A GLM score calculation can look nearly correct and still lead to a wrong estimating equation or curvature matrix. Common causes include reversing $y_i-\mu_i$, dropping the derivative of the inverse link, omitting a dispersion factor, or removing a likelihood term that depends on the parameter being estimated. A separate conceptual error is to treat the observed Hessian as though it were the score equation.

The safest check is to identify which quantity is being differentiated, write the dependence on the regression coefficients explicitly, and apply the chain rule before simplifying. The formulas below provide a compact audit for that process.

A Score and Observed-Hessian Audit

Suppose $Y_1,\ldots,Y_n$ are independent responses in an exponential-dispersion GLM. For observation $i$, let the mean be $\mu_i=E[Y_i]$, the variance be $\operatorname{Var}(Y_i)=\phi V(\mu_i)$, and the linear predictor be $\eta_i=x_i^\mathsf{T}\beta+o_i$, where $x_i$ is a fixed column vector, $o_i$ is a known offset, and $\phi>0$ is held fixed while differentiating with respect to $\beta$. Assume the inverse link $\mu_i=h(\eta_i)$ is twice differentiable and $V$ is differentiable at the relevant means. The usual exponential-dispersion log-likelihood contribution, up to terms not depending on $\beta$, is $\{y_i\theta_i-b(\theta_i)\}/\phi$, where $d\mu_i/d\theta_i=V(\mu_i)$.

Definition: The score vector for $\beta$ is $U(\beta)=\partial\ell(\beta)/\partial\beta$, the gradient of the log-likelihood. The observed Hessian is $H(\beta)=\partial^2\ell(\beta)/\partial\beta\,\partial\beta^\mathsf{T}$, the matrix of second derivatives evaluated at the observed data. A score equation is $U(\beta)=0$; it is not an equation setting $H(\beta)$ to zero.

Write $h'(\eta)=d\mu/d\eta$, $h''(\eta)=d^2\mu/d\eta^2$, and $V'(\mu)=dV(\mu)/d\mu$. Differentiating the log-likelihood first with respect to the mean gives $\partial\ell_i/\partial\mu_i=(y_i-\mu_i)/\{\phi V(\mu_i)\}$. Applying the chain rule to $\mu_i=h(\eta_i)$, and then to $\eta_i=x_i^\mathsf{T}\beta+o_i$, gives the score.

Result: Under the conditions above, the score and observed Hessian are $$ U(\beta)=\sum_{i=1}^n x_i\frac{(y_i-\mu_i)h'(\eta_i)}{\phi V(\mu_i)} $$ and $$ H(\beta)=\sum_{i=1}^n x_i x_i^\mathsf{T}\frac{1}{\phi} \left[ -\frac{\{h'(\eta_i)\}^2}{V(\mu_i)} +(y_i-\mu_i)\left\{ \frac{h''(\eta_i)}{V(\mu_i)} -\frac{\{h'(\eta_i)\}^2 V'(\mu_i)}{\{V(\mu_i)\}^2} \right\} \right]. $$

To verify the Hessian, define the scalar score multiplier $q_i=(y_i-\mu_i)h'(\eta_i)/\{\phi V(\mu_i)\}$. Since $x_i$ and the offset are fixed, $\partial(x_iq_i)/\partial\beta^\mathsf{T}=x_i x_i^\mathsf{T}(dq_i/d\eta_i)$. Differentiating the numerator and denominator of $q_i$ gives

\frac{dq_i}{d\eta_i} =\frac{1}{\phi}\left[ -\frac{\{h'(\eta_i)\}^2}{V(\mu_i)} +(y_i-\mu_i)\left\{ \frac{h''(\eta_i)}{V(\mu_i)} -\frac{\{h'(\eta_i)\}^2V'(\mu_i)}{\{V(\mu_i)\}^2} \right\} \right].

The first term arises from differentiating $y_i-\mu_i$. The other two arise from differentiating $h'(\eta_i)/V(\mu_i)$. In particular, the observed Hessian can depend on $y_i$; replacing it by a simplified expression without checking the link and variance function can lose real terms.

Worked Examples: Finding the Error

Worked Example: A Sign Check for Bernoulli Logit

Take one Bernoulli observation with $y=1$, scalar predictor $x=2$, no offset, and logit link. At $\beta=0$, calculate the score and observed Hessian with respect to $\beta$.

For the logit link, $\eta=x\beta$, $\mu=\exp(\eta)/\{1+\exp(\eta)\}$, and $h'(\eta)=\mu(1-\mu)$. Bernoulli variance has $V(\mu)=\mu(1-\mu)$ and $\phi=1$. At $\beta=0$, $\eta=0$ and $\mu=1/2$, so the score is

U(\beta)=x\frac{(y-\mu)h'(\eta)}{V(\mu)} =2\left(1-\frac12\right)=1.

The positive score means that, at this parameter value, a small increase in $\beta$ increases the log-likelihood. For the canonical logit, the score simplifies to $x(y-\mu)$, and its derivative is $-x^2\mu(1-\mu)$. Thus

H(\beta)=-2^2\left(\frac12\right)\left(\frac12\right)=-1.

Writing $x(\mu-y)$ instead would give $-1$ for the score, the wrong sign. A useful check is the likelihood itself: increasing $\beta$ increases $\eta$, hence increases the fitted success probability from $1/2$; for an observed success, that raises the Bernoulli log-likelihood.

Worked Example: The Link Derivative in a Probit Score

For one Bernoulli observation, use the probit link $\mu=\Phi(\eta)$, where $\Phi$ is the standard normal distribution function. Let $x=1$, $y=1$, and evaluate the score and Hessian at $\eta=0$. Here $\phi=1$, $V(\mu)=\mu(1-\mu)$, and $h'(\eta)=\varphi(\eta)$, the standard normal density.

At zero, $\mu=1/2$, $\varphi(0)=1/\sqrt{2\pi}$, and $h''(0)=0$. The score is therefore

U=\frac{(1-\frac12)\frac{1}{\sqrt{2\pi}}}{\frac14} =\frac{2}{\sqrt{2\pi}} \approx 0.7979.

If the factor $h'(\eta)$ is omitted, the resulting expression gives $(1-\frac12)/(\frac14)=2$, which is not the probit score. For the Hessian, $V'(\mu)=1-2\mu=0$ at $\mu=1/2$. Substitution into the general expression gives

H=-\frac{\{1/\sqrt{2\pi}\}^2}{1/4} =-\frac{2}{\pi} \approx -0.6366.

This is also the second derivative of $\log\Phi(\eta)$ at zero: its first derivative is $\varphi(\eta)/\Phi(\eta)$, whose derivative at zero is $-\{\varphi(0)/\Phi(0)\}^2=-2/\pi$. The independent check confirms both the derivative and its sign.

Worked Example: A Dispersion Term That Cannot Be Dropped

Let $Y\sim N(\mu,\phi)$, with known $\mu=2$, and observe $y=3$. Derive the likelihood score for $\phi$. The normal log-likelihood is

\ell(\phi)=-\frac12\log(2\pi\phi)-\frac{(y-\mu)^2}{2\phi} =-\frac12\log(2\pi\phi)-\frac{1}{2\phi}.

Differentiating both terms gives

\frac{d\ell}{d\phi}=-\frac{1}{2\phi}+\frac{1}{2\phi^2}.

Setting the score to zero yields $\phi=1$. The second derivative at that value is $1/(2\phi^2)-1/\phi^3=-1/2$, so this stationary point is a local maximum. If $-\tfrac12\log(2\pi\phi)$ is discarded as a “constant,” the remaining derivative is $1/(2\phi^2)>0$ for every $\phi>0$. The resulting calculation has no finite stationary point. That term is constant with respect to $\mu$ when estimating a mean with fixed dispersion, but it is not constant with respect to $\phi$.

Canonical Links: A Useful Check, Not a Universal Shortcut

A canonical link sets the linear predictor equal to the natural parameter: $\eta_i=\theta_i$. In that case, the score and Hessian simplify. The simplification is useful for checking a derivation, but it should not be carried over automatically to a noncanonical link.

Result: For a canonical link, with fixed $\phi$ and fixed design, $$ U(\beta)=\frac{1}{\phi}\sum_{i=1}^n x_i(y_i-\mu_i), \qquad H(\beta)=-\frac{1}{\phi}\sum_{i=1}^n x_i x_i^\mathsf{T}V(\mu_i). $$

To derive these expressions, write the parameter-dependent part of the $i$th log-likelihood as $\{y_i\theta_i-b(\theta_i)\}/\phi$. Since $d b(\theta_i)/d\theta_i=\mu_i$ and $\partial\theta_i/\partial\beta=x_i$, the first derivative is $x_i(y_i-\mu_i)/\phi$. Differentiating again, $d\mu_i/d\theta_i=V(\mu_i)$, so the second derivative is $-x_i x_i^\mathsf{T}V(\mu_i)/\phi$. Summing over independent observations gives the result. $\square$

This derivation also reveals a common sign trap: the Hessian contains a minus sign because increasing the natural parameter increases the mean. For a noncanonical link, the score generally includes $h'(\eta_i)/V(\mu_i)$, and the observed Hessian has additional response-dependent terms. The canonical formulas cannot simply be reused.

Score Equations Are Not Hessian Equations

The score is a vector of first derivatives. Its roots are candidate stationary points of the log-likelihood, subject to the parameter space and regularity of the model. The observed Hessian is a matrix of second derivatives that describes local curvature at a specified parameter value. At an interior local maximum with a twice-differentiable likelihood, the score is zero and the Hessian is negative semidefinite. The converse does not follow from either condition alone: a zero score may be a minimum or saddle point, and a negative Hessian at an arbitrary point does not make that point a stationary point.

For a scalar parameter, the distinction is direct: solve $\ell'(\beta)=0$, then use $\ell''(\beta)$ to assess local curvature at a solution. For multiple regression coefficients, solve $U(\beta)=0$; inspect the Hessian separately. A Hessian is not obtained by setting the score equal to zero, nor is a score equation obtained by setting each second derivative to zero.

Fix the target parameter.
Mark which quantities are held fixed. A term independent of $\beta$ may still depend on $\phi$.

Differentiate in layers.
Track the path from $\beta$ to $\eta_i$ to $\mu_i$, including the inverse-link derivative.

Check scale and sign.
Keep the dispersion factor and test the score direction against whether the observed response is above or below the fitted mean.

Label the result.
Identify a first derivative as a score and a second derivative as an observed Hessian before solving or interpreting it.

Common Mistakes

Reversing the residual. Replacing $y_i-\mu_i$ by $\mu_i-y_i$ reverses the score unless another sign change compensates. Check a simple case where the response exceeds the fitted mean.
Dropping the inverse-link derivative. The derivative of $\mu_i$ with respect to $\eta_i$ is $h'(\eta_i)$, not generally one. It equals one for the identity link, not for every link.
Omitting dispersion. In an exponential-dispersion model, regression scores carry the factor $1/\phi$. It may not change a zero of the score when $\phi$ is a positive fixed scalar, but it changes the score’s scale and the Hessian’s scale; it cannot be discarded when differentiating with respect to dispersion.
Calling a parameter-dependent term a constant. A likelihood term can be dropped only for a derivative with respect to a parameter on which it does not depend. Check this separately for each parameter.
Confusing score and curvature. The score equation is $U(\beta)=0$; the Hessian is evaluated to describe curvature. They are different derivatives with different roles.

Key takeaway: Write the likelihood’s dependence on the parameter being estimated, apply the chain rule explicitly, retain every factor that depends on that parameter, and keep first-derivative equations separate from second-derivative curvature.

Check Your Understanding

Use the formulas and checks above to diagnose each situation.

For a Bernoulli logit model, what is the sign of the score contribution when $y_i=0$ and $0<\mu_i<1$, assuming $x_i$ is a positive scalar?
In a probit GLM, which derivative factor appears when differentiating the mean with respect to the linear predictor?
Why can a term independent of regression coefficients still matter when estimating dispersion?
What does a score equation identify, and what does the observed Hessian describe?
For a canonical-link model with fixed dispersion, state the sign and variance-function factor in the observed Hessian contribution.