The Fisher Scoring Equation for GLM Coefficients

From the Mean Model to a Coefficient Update

In a generalized linear model, coefficients determine a linear predictor, the link turns that predictor into a mean, and the mean determines both the likelihood’s slope and its curvature. Fisher scoring uses the expected curvature to turn the current coefficient score into a step. The resulting equation applies across GLM families, although its numerical ingredients depend on the link and variance function.

This derivation holds the dispersion parameter fixed while updating the regression coefficients. The response observations are independent, and the design matrix is treated as fixed. We will first obtain the score from the likelihood contribution for one observation, then calculate the expected negative Hessian and assemble the scoring equation.

Definition: Let $Y_1,\ldots,Y_n$ be independent responses with observed values $y_1,\ldots,y_n$. Let $X$ be a fixed $n \times p$ design matrix, with row $x_i^T$, and let $\beta \in \mathbb{R}^p$. The linear predictor and mean for observation $i$ are $$ \eta_i=x_i^T\beta,\qquad \mu_i=h(\eta_i), $$ where $h=g^{-1}$ is a differentiable inverse link. Write $\dot{\mu}_i=d h(\eta_i)/d\eta_i$. Assume the responses have an exponential-dispersion likelihood with differentiable log likelihood, mean $\mu_i$, variance $\phi V(\mu_i)$, and $\phi>0$ fixed. Assume $V(\mu_i)>0$, and that the likelihood support does not depend on $\beta$.

For the likelihood calculation, take the usual exponential-dispersion form for each response: $$ \ell_i(\beta)=\frac{y_i\theta_i-b(\theta_i)}{\phi}+c(y_i,\phi), \qquad \mu_i=b'(\theta_i),\qquad V(\mu_i)=b''(\theta_i). $$ Here $\theta_i$ is the natural parameter. The mean and variance functions are assumed differentiable where evaluated. Since $d\mu_i/d\theta_i=V(\mu_i)$, the chain rule gives $d\theta_i/d\mu_i=1/V(\mu_i)$. This is the likelihood identity that connects the mean and variance functions to the coefficient score.

Deriving the Coefficient Score

Theorem: Under the stated model and differentiability conditions, the score for $\beta$, with $\phi$ fixed, is $$ U(\beta)=\frac{\partial \ell(\beta)}{\partial\beta} =X^T\operatorname{diag}\!\left(\frac{\dot{\mu}_i}{\phi V(\mu_i)}\right)(y-\mu), $$ where $y=(y_1,\ldots,y_n)^T$, $\mu=(\mu_1,\ldots,\mu_n)^T$, and $\ell(\beta)=\sum_i\ell_i(\beta)$.

To prove this, first differentiate one likelihood contribution with respect to its mean. The derivative with respect to $\theta_i$ is $(y_i-b'(\theta_i))/\phi=(y_i-\mu_i)/\phi$. Applying the chain rule through $\theta_i$ gives $$ \frac{\partial \ell_i}{\partial \mu_i} =\frac{y_i-\mu_i}{\phi}\frac{d\theta_i}{d\mu_i} =\frac{y_i-\mu_i}{\phi V(\mu_i)}. $$ The derivative of $\mu_i=h(x_i^T\beta)$ with respect to the coefficient vector is $\dot{\mu}_i x_i$. Thus $$ \frac{\partial \ell_i}{\partial\beta} =x_i\frac{\dot{\mu}_i(y_i-\mu_i)}{\phi V(\mu_i)}. $$ Summing over the independent observations yields the stated matrix expression. $\square$

The score is a weighted sum of design rows multiplied by response residuals. The coefficient multiplying each residual depends on the local slope of the inverse link, $\dot{\mu}_i$, and the response variance, $\phi V(\mu_i)$. In particular, a noncanonical link does not change the form of the score, but it changes $\dot{\mu}_i$ and therefore changes the coefficient multiplying each residual.

Worked Example: A Gaussian Identity-Link Update

Suppose $Y_i$ are independent Gaussian observations with mean $\mu_i=\beta$, variance $\phi=2$, and $V(\mu)=1$. The model has an intercept only, so $X$ is a column of ones. The observed values are $2,4,6$, and the current coefficient is $\beta^{(0)}=0$. Compute one Fisher scoring step.

For the identity link, $\dot{\mu}_i=1$, and each current mean is $0$. The score is $$ U(\beta^{(0)})=\sum_{i=1}^3\frac{y_i-0}{2} =\frac{2+4+6}{2}=6. $$ The expected information for this intercept is $3/2$, since each observation contributes $1/(\phi V)=1/2$. Therefore the scoring equation gives $$ \frac{3}{2}\Delta\beta=6, \qquad \Delta\beta=4, \qquad \beta^{(1)}=\beta^{(0)}+\Delta\beta=4. $$ The updated fitted mean is $4$, the sample average. For this model the score is linear in $\beta$, so this step reaches the likelihood maximizer.

Expected Curvature and the Scoring Equation

The observed Hessian is the matrix of second derivatives of the log likelihood at the realized data. Fisher scoring instead uses the expected negative Hessian, called the expected Fisher information. These matrices need not be equal for a given data set. To see what Fisher scoring uses here, differentiate the score before taking its expectation.

Theorem: Suppose, in addition to the conditions above, that the required expectations exist and $E[Y_i]=\mu_i$. Define $$ \mathcal{I}(\beta)=-E\!\left[\frac{\partial^2\ell(\beta)}{\partial\beta\,\partial\beta^T}\right]. $$ Then $$ \mathcal{I}(\beta) =X^T\operatorname{diag}\!\left(\frac{\dot{\mu}_i^2}{\phi V(\mu_i)}\right)X. $$ If $\mathcal{I}(\beta)$ is nonsingular, the Fisher scoring increment $\Delta\beta$ and update are $$ \mathcal{I}(\beta)\Delta\beta=U(\beta), \qquad \beta_{\mathrm{new}}=\beta+\Delta\beta. $$

For the derivation, set $q_i(\eta_i)=\dot{\mu}_i/(\phi V(\mu_i))$. The score contribution is $x_i q_i(\eta_i)(y_i-\mu_i)$. Differentiating with respect to $\beta$ gives the Hessian contribution $$ x_i x_i^T\left[ \frac{d q_i}{d\eta_i}(y_i-\mu_i) -q_i(\eta_i)\dot{\mu}_i \right]. $$ Because $E[Y_i-\mu_i]=0$, the expectation of the first term in brackets is zero. The second term is $-\dot{\mu}_i^2/(\phi V(\mu_i))$. Summing and negating the expected Hessian gives the displayed information matrix. Solving the linear system for the increment gives the scoring update. $\square$

This argument obtains the expected curvature directly from the differentiated score: the term involving the derivative of $q_i$ vanishes in expectation because it multiplies a mean-zero residual. It does not say that the observed Hessian itself is just the negative information matrix. That observed Hessian retains a residual-dependent term.

Fisher scoring equation: At iteration $t$, evaluate the score and expected information at $\beta^{(t)}$, then solve $$ \mathcal{I}(\beta^{(t)})\Delta\beta^{(t)}=U(\beta^{(t)}), \qquad \beta^{(t+1)}=\beta^{(t)}+\Delta\beta^{(t)}. $$

When $V(\mu_i)>0$, $\phi>0$, and $\dot{\mu}_i\ne 0$ for every observation, the diagonal entries in the information matrix are positive. In that case, full column rank of $X$ ensures that $\mathcal{I}(\beta)$ is positive definite and hence invertible. Without nonsingularity, the usual Fisher scoring step is not uniquely defined by this equation; a generalized inverse or a modified algorithm would require an additional choice.

Worked Example: A Bernoulli Logit Intercept Update

Let $Y_i$ be independent Bernoulli responses, with an intercept-only logit model and observed outcomes $1,0,1$. At the current value $\beta^{(0)}=0$, calculate one Fisher scoring update.

The inverse logit gives $\mu_i=1/(1+\exp(-\beta))$, so at $\beta=0$, $\mu_i=1/2$. For a Bernoulli response, $V(\mu)=\mu(1-\mu)$, and the inverse-link derivative is $\dot{\mu}=\mu(1-\mu)$. Thus $\dot{\mu}_i=1/4$ and $\phi=1$. Each score contribution is $$ \frac{\dot{\mu}_i}{V(\mu_i)}(y_i-\mu_i) =\frac{1/4}{1/4}(y_i-1/2)=y_i-1/2. $$ The total score is $1/2-1/2+1/2=1/2$. Each observation contributes $\dot{\mu}_i^2/V(\mu_i)=(1/4)^2/(1/4)=1/4$ to the information, so the total information is $3/4$. Therefore $$ \Delta\beta=\frac{1/2}{3/4}=\frac{2}{3}, \qquad \beta^{(1)}=\frac{2}{3}. $$ This is one scoring step from the specified starting value; it is not asserted to be the final maximum-likelihood estimate.

Worked Example: A Poisson Log-Link Intercept Update

Suppose the independent counts are $1,2,4$, with an intercept-only Poisson log-link model. Use $\phi=1$, $V(\mu)=\mu$, $\mu=\exp(\beta)$, and start at $\beta^{(0)}=0$.

At this starting value, $\mu_i=1$ and $\dot{\mu}_i=1$ for all three observations. Each score contribution is $$ \frac{\dot{\mu}_i}{V(\mu_i)}(y_i-\mu_i)=y_i-1. $$ The total score is $0+1+3=4$. Each observation contributes $\dot{\mu}_i^2/V(\mu_i)=1$ to the expected information, so $\mathcal{I}=3$. The increment and updated coefficient are $$ \Delta\beta=\frac{4}{3}, \qquad \beta^{(1)}=\frac{4}{3}. $$ The corresponding fitted mean is $\exp(4/3)$. The update is an additive step on the coefficient, or equivalently on the log mean in this intercept-only model.

What the Equation Does—and Does Not—Guarantee

The scoring equation combines the current score with the expected local curvature. If the score is zero and the information is nonsingular, the increment is zero. Otherwise, the step adjusts the coefficients in a direction determined by both the residual pattern and the model’s expected information. The matrix formula is useful because it separates these ingredients and makes the dependence on the inverse link and variance function explicit.

A Fisher scoring step is an iterative update, not a guarantee that the likelihood increases at every iteration or that an algorithm will converge from every starting value. Even with a valid score and information matrix, practical computations may require checks on the step and on whether fitted means remain in the model’s valid range.

Common Mistake

Using the observed Hessian as though it were always the Fisher information. The observed Hessian contains the term involving $d q_i/d\eta_i$ multiplied by $y_i-\mu_i$. It generally varies with the realized residuals. The Fisher scoring equation uses the negative expected Hessian; the residual-dependent term disappears only after taking expectation, since $E[Y_i-\mu_i]=0$.
Dropping the inverse-link derivative. Replacing $\dot{\mu}_i$ by $1$ is justified for an identity link, not for a general link. The chain rule contributes this derivative to the score, and its square appears in the expected information.
Treating a scoring increment as the fitted coefficient. The update is $\beta_{\mathrm{new}}=\beta+\Delta\beta$, not $\beta_{\mathrm{new}}=\Delta\beta$, unless the current coefficient happens to be zero.

Key takeaway: With fixed dispersion, the GLM coefficient score is $X^T\operatorname{diag}(\dot{\mu}_i/(\phi V(\mu_i)))(y-\mu)$, while the expected information is $X^T\operatorname{diag}(\dot{\mu}_i^2/(\phi V(\mu_i)))X$. Fisher scoring solves the information-weighted score equation for a coefficient increment.

Check Your Understanding

Use the definitions and equations above to answer these questions.

Which chain-rule derivative produces the factor $\dot{\mu}_i$ in the coefficient score?
Why does the residual-dependent term in the observed Hessian vanish when computing the expected information?
For an intercept-only model, explain why full column rank of $X$ holds when there is at least one observation and the intercept column is nonzero.
If $\dot{\mu}_i=0$ for one observation, what does that imply about that observation’s contribution to the expected information?
Why is a Fisher scoring increment not, in general, itself the next coefficient value?