Deriving Generalized Linear Model Score Equations

From the response likelihood to the coefficient score

In a generalized linear model, the likelihood is built from the distribution of each response, while the regression coefficients determine that response's mean through a sequence of transformations. The score for the coefficients therefore follows a chain: first differentiate the likelihood with respect to the mean, then account for the link, and finally account for the linear predictor. Making each step explicit reveals exactly how the design matrix, observed-minus-fitted residuals, variance function, and link derivative combine.

Assume \(Y_1,\ldots,Y_n\) are independent, with fixed-dispersion exponential-dispersion-family densities or mass functions

Here \(\phi>0\) is held fixed when differentiating with respect to the regression coefficients. Assume the natural parameter is in the interior of its parameter space, \(b\) is twice differentiable, and \(b''(\theta_i)>0\). The standard exponential-family identities give \(E[Y_i]=b'(\theta_i)\) and \(\operatorname{Var}(Y_i)=\phi b''(\theta_i)\). Write \(\mu_i=E[Y_i]\) and define the variance function \(V\) by \(\operatorname{Var}(Y_i)=\phi V(\mu_i)\). Thus \(V(\mu_i)=b''(\theta_i)\), expressed as a function of the mean.

For a generalized linear model, the mean is related to the linear predictor by a link \(g\):

The vector \(x_i\) is the \(i\)th row of the \(n\times p\) design matrix \(X\), and \(\beta\) is the \(p\)-dimensional coefficient vector. For the derivation below, assume \(g\) is continuously differentiable on a neighborhood of each \(\mu_i\), and \(g'(\mu_i)\ne0\). The inverse function theorem then supplies a differentiable local inverse near each fitted mean. This condition justifies the derivative of the mean with respect to the linear predictor; differentiability and a nonzero derivative at a single point alone would not suffice.

First derivative: likelihood with respect to the mean

Let \(\ell_i\) denote the log-likelihood contribution from observation \(i\). The derivative with respect to the mean is the first link in the chain rule.

To verify this, differentiate the log density first with respect to the natural parameter:

Since \(d\mu_i/d\theta_i=b''(\theta_i)=V(\mu_i)>0\), the inverse function theorem gives \(d\theta_i/d\mu_i=1/V(\mu_i)\) locally. Applying the chain rule now yields

The numerator is the raw residual. The variance function and dispersion scale determine how strongly that residual changes the log-likelihood for a change in the mean.

Chain rule through the link and design matrix

The inverse link determines how a change in \(\eta_i\) changes \(\mu_i\). Because the link has a differentiable local inverse with nonzero derivative at \(\mu_i\),

Also, \(\partial\eta_i/\partial\beta_j=x_{ij}\). Combining these derivatives with the mean score gives the coefficient derivative for one observation.

For completeness, the \(j\)th component follows by differentiating the full log-likelihood \(\ell=\sum_{i=1}^n\ell_i\). Independence makes the log-likelihood a sum, and the chain rule gives

Stacking these \(p\) components gives the displayed matrix expression: multiplication by the diagonal matrix applies the observation-specific factor to each residual, and multiplication by \(X^{\mathsf T}\) sums those adjusted residuals against each column of the design matrix. This proves the result. \(\square\)

When an interior finite maximum likelihood estimate exists, its score must be zero. Thus the coefficient score equations are

These equations are necessary at an interior maximum, not a general promise that a solution exists or that every solution is a maximum. The earlier material on existence and uniqueness of maximum likelihood estimates addresses those separate questions.

Worked examples

Reading and using the score equations

The general formula separates three roles. The residual \(y_i-\mu_i\) records the discrepancy between observation and fitted mean. The variance function \(V(\mu_i)\) scales that discrepancy according to the response distribution, and the link derivative \(g'(\mu_i)\) accounts for how the link changes with the mean. Finally, \(X^{\mathsf T}\) aggregates these observation-level contributions into one equation per coefficient.

For some familiar links, the variance and link derivative cancel: this happens for the Poisson log link and the Bernoulli logit link in the examples. That simplification is not universal. For another link or response family, retain both factors unless their cancellation has been established algebraically. The matrix diagonal factor need not be positive for an arbitrary monotone link, so it is safest here to call it a multiplier rather than presume it is a positive weight.

Common mistake: differentiating the inverse link without justification

A tempting argument is: “\(g\) is differentiable at \(\mu_i\) and \(g'(\mu_i)\ne0\), therefore \(d\mu_i/d\eta_i=1/g'(\mu_i)\).” Those pointwise conditions alone do not guarantee a differentiable local inverse. The inverse function theorem requires a neighborhood condition; in this derivation, taking \(g\) continuously differentiable on a neighborhood of \(\mu_i\), with \(g'(\mu_i)\ne0\), supplies it. Alternatively, one can directly assume a differentiable local inverse exists there. Under either valid condition, the derivative formula follows, and only then is the coefficient score derivation justified.

A second practical error is to set the score equal to zero and conclude automatically that a finite maximum likelihood estimate exists. The score equation describes stationary points in the interior of the parameter space. Existence, boundary behavior, and whether a stationary point is a maximum are distinct issues.