Why Fisher Scoring Becomes Weighted Least Squares
A generalized linear model (GLM) is usually estimated by an iterative method because its likelihood equations are not generally linear in the coefficients. Fisher scoring turns each iteration into a familiar linear-algebra problem: fit a temporary response to the model matrix using weights determined by the current fitted means. The temporary response and weights change from one iteration to the next; they are not new observed data or permanent features of the model.
This equivalence is useful both computationally and conceptually. It shows how the likelihood score determines the direction of an update, while the expected information determines the scale of that update. The result is a weighted least-squares system that can be solved with standard numerical methods.
Here \(x_i\) is the row of predictor values for observation \(i\), and \(\beta\) is the vector of coefficients. The derivation assumes independent observations, a differentiable link, positive variances, and a design matrix with enough rank to identify the coefficients. A model with correlated responses, a variance structure not represented by the stated GLM, or a singular information matrix needs additional treatment; the simple weighted least-squares update below does not automatically apply.
From the Score to the Fisher-Scoring Step
For an exponential-dispersion model, write the contribution to the log likelihood, apart from terms not involving the parameters, as \(\{y_i\theta_i-b(\theta_i)\}/(\phi a_i)\). The mean is \(\mu_i=b'(\theta_i)\), and the variance function satisfies \(V(\mu_i)=b''(\theta_i)\). Differentiating the log likelihood with respect to \(\eta_i\), then applying the chain rule, gives \(\partial\ell_i/\partial\eta_i=(y_i-\mu_i)(d\mu_i/d\eta_i)/(\phi a_i V(\mu_i))\). Since \(\eta_i=x_i^\mathsf{T}\beta\), differentiating with respect to \(\beta\) multiplies this expression by \(x_i\).
The Fisher information is the expected negative Hessian of the log likelihood. For a GLM in this parameterization, its contribution from observation \(i\) is \(x_i x_i^\mathsf{T}(d\mu_i/d\eta_i)^2/(\phi a_i V(\mu_i))\). Thus the information matrix can be written \(X^\mathsf{T}WX\), where \(X\) has rows \(x_i^\mathsf{T}\) and \(W\) is diagonal with the corresponding nonnegative weights.
Fisher scoring takes the current estimate \(\beta\), computes its score and expected information, and adds the solution of the information-weighted score equation. If \(X^\mathsf{T}WX\) is nonsingular, the increment is \((X^\mathsf{T}WX)^{-1}U(\beta)\). To rewrite that step as least squares, define the working response \(z_i\) by adding a first-order adjustment to the current linear predictor:
The adjustment is chosen so that \(W_{ii}(z_i-\eta_i)\) equals observation \(i\)’s contribution to the score before multiplication by \(x_i\). Indeed, substituting the definitions gives \(W_{ii}(z_i-\eta_i)=(y_i-\mu_i)(d\mu_i/d\eta_i)/(\phi a_iV(\mu_i))\). Therefore \(X^\mathsf{T}W(z-\eta)=U(\beta)\). Since the current predictor vector is \(\eta=X\beta\), the Fisher-scoring equation becomes a weighted least-squares normal equation.
This identity is the central result: one Fisher-scoring update is obtained by fitting the working responses \(z_i\) to the original model matrix \(X\), with weights \(W_{ii}\). It is a local calculation at the current estimate, not a claim that the original GLM is itself an ordinary least-squares model.
Derivation as an Iterative Procedure
Compute \(\eta_i=x_i^\mathsf{T}\beta\), then \(\mu_i=g^{-1}(\eta_i)\).
Use the current mean and link derivative to form \(z_i=\eta_i+(y_i-\mu_i)/(d\mu_i/d\eta_i)\).
Set \(W_{ii}=(d\mu_i/d\eta_i)^2/(\phi a_iV(\mu_i))\), using the same current fitted means.
Find \(\beta_{\mathrm{new}}\) satisfying \(X^\mathsf{T}WX\beta_{\mathrm{new}}=X^\mathsf{T}Wz\). Then recompute the means, responses, and weights at the new estimate.
If all observations share a common dispersion \(\phi\), the factor \(1/\phi\) multiplies every weight. It appears in both sides of the normal equation and therefore cancels from the coefficient update. It does not follow that dispersion is irrelevant to uncertainty estimates: the covariance calculation still depends on the dispersion specification.
Worked Examples
Worked Example: Normal Model With an Identity Link
Suppose \(Y_1=2\) and \(Y_2=4\) are independent observations from normal distributions with common known variance \(4\). Fit an intercept-only model with identity link, so \(\mu_i=\eta_i=\beta\). Starting from \(\beta=1\), find one Fisher-scoring update and confirm it by weighted least squares.
Method and assumptions. This is a normal GLM with \(V(\mu)=1\), \(\phi=4\), and \(a_i=1\). The identity link gives \(d\mu_i/d\eta_i=1\). There are two observations, and the design matrix has one column of ones.
The score is \(U(\beta)=\{(2-1)+(4-1)\}/4=4/4=1\). Each information weight is \(1^2/4=0.25\), so the information is \(0.25+0.25=0.5\). The coefficient increment is \(1/0.5=2\), giving \(\beta_{\mathrm{new}}=1+2=3\).
For the weighted least-squares check, the working response is \(z_i=\eta_i+(y_i-\mu_i)=y_i\), because the link derivative is one. Thus \(z_1=2\), \(z_2=4\), and both weights are \(0.25\). The fitted intercept is \((0.25(2)+0.25(4))/(0.25+0.25)=1.5/0.5=3\). The two methods agree. Common mistake avoided: using the variance \(4\) as the weight; least-squares weights are inverse variances here, \(1/4\), not \(4\).
Worked Example: One Poisson Log-Link Update
Two independent hypothetical rating cells have Poisson claim counts \(Y_1=1\) and \(Y_2=4\). Fit a common count mean with a log link, no offset, and current coefficient \(\beta=\ln 2\), so both current means are \(2\). Calculate one Fisher-scoring update, first from the score and information and then from the working-response fit.
Method and assumptions. For a Poisson response, \(\phi=1\), \(a_i=1\), and \(V(\mu_i)=\mu_i\). With the log link, \(\eta_i=\ln\mu_i\) and \(d\mu_i/d\eta_i=\mu_i\). Each weight is therefore \(W_{ii}=\mu_i=2\).
The score is \((1-2)+(4-2)=-1+2=1\), and the total information is \(2+2=4\). The coefficient increment is \(1/4=0.25\); hence \(\beta_{\mathrm{new}}=\ln 2+0.25=0.943147\), or \(0.9431\) to four decimal places.
For the least-squares check, \(\eta_i=\ln 2\) and the working responses are \(z_1=\ln 2+(1-2)/2=0.193147\) and \(z_2=\ln 2+(4-2)/2=1.693147\). Both weights are \(2\), so the fitted intercept is their weighted mean: \(\{2(0.193147)+2(1.693147)\}/4=3.772588/4=0.943147\). The update agrees with the score calculation. The resulting means are approximately \(e^{0.943147}=2.568\); they need not equal either observed count after just one iteration. Common mistake avoided: dropping \(\ln 2\) from the working responses or treating the residual adjustments as the responses themselves.
Worked Example: Binomial Logit Update With a Group Indicator
A hypothetical study observes two groups, each with \(n_i=4\) independent trials. Group 0 has \(y_0=1\) success and group 1 has \(y_1=3\) successes. Model each group’s success proportion \(r_i=y_i/n_i\) with a logit link and design rows \((1,0)\) and \((1,1)\). Starting from \(\beta=(0,0)^\mathsf{T}\), calculate one Fisher-scoring update.
Method and assumptions. The responses are independent binomial proportions, with \(a_i=1/n_i=1/4\), \(\phi=1\), \(V(p)=p(1-p)\), and \(\eta=\log\{p/(1-p)\}\). At the starting coefficients both fitted probabilities are \(p=0.5\), so \(dp/d\eta=p(1-p)=0.25\). The weight for each proportion is \((0.25)^2/\{(1/4)(0.25)\}=1\).
The observed proportions are \(r_0=1/4=0.25\) and \(r_1=3/4=0.75\). Since both current linear predictors are zero, the working responses are \(z_0=0+(0.25-0.5)/0.25=-1\) and \(z_1=0+(0.75-0.5)/0.25=1\).
With unit weights, the information matrix and score are \(X^\mathsf{T}WX=\begin{pmatrix}2&1\\1&1\end{pmatrix}\) and \(X^\mathsf{T}W(z-\eta)=(0,1)^\mathsf{T}\). Solving the normal equations gives \(2\Delta\beta_0+\Delta\beta_1=0\) and \(\Delta\beta_0+\Delta\beta_1=1\), hence \(\Delta\beta_0=-1\) and \(\Delta\beta_1=2\). The updated coefficients are \((-1,2)^\mathsf{T}\). The updated group logits are \(-1\) and \(-1+2=1\), respectively.
As a check, the group-specific coefficient increments are the fitted working responses \(-1\) and \(1\), exactly as the update implies. Common mistake avoided: treating the group-1 coefficient increment as its fitted logit. The group-1 logit is the sum of the intercept and group effect, \(1\), not the group effect \(2\) alone.
Interpretation, Conditions, and Common Traps
Fisher scoring uses the expected information, whereas Newton–Raphson uses the observed negative Hessian evaluated at the current data and parameter values. In a GLM, the expected information has the especially convenient form \(X^\mathsf{T}WX\). Replacing it with the observed Hessian without justification changes the algorithm and may change the update.
The weighted least-squares representation also has practical limits. If some weights are zero, or if the weighted design matrix is rank deficient, the normal equations may not have a unique solution. A factor with no observed variation, for example, cannot have its separate coefficient estimated from the data. Numerical implementations generally solve the weighted least-squares system using stable matrix factorization rather than explicitly calculating \((X^\mathsf{T}WX)^{-1}\).
- Use the current iterate. Both \(z_i\) and \(W_{ii}\) depend on the current \(\eta_i\) and \(\mu_i\); do not calculate one at the old iterate and the other at the new one within the same step.
- Keep the link derivative straight. The working-response adjustment divides by \(d\mu_i/d\eta_i\), while the weight contains its square.
- Do not confuse the response with the working response. The data remain \(y_i\). The temporary \(z_i\) is constructed to make the Fisher-scoring normal equations match the score equation.
Check Your Understanding
For each problem, identify the model quantities needed for the update before calculating.
- An intercept-only Poisson log-link model has two independent counts \(2\) and \(6\), with current mean \(4\) for each. Find the score, information, coefficient increment, and updated intercept. (Answer: score \(0\); information \(8\); increment \(0\); updated intercept \(\ln 4\).)
- For a normal identity-link model with known variance \(9\), observations \(1,7,10\), and current intercept \(4\), find one Fisher-scoring update. (Answer: updated intercept \(6\), increment \(2\).)
- A Poisson log-link model has one observation \(y=3\) and current mean \(\mu=2\). Find its Fisher-scoring increment for an intercept-only model. (Answer: \(0.5\).)
- Two binomial groups each have \(n=8\) trials, with 2 and 6 successes. For the intercept-only logit model at current probability \(0.5\), find the coefficient increment. (Answer: \(0\).)
- For an independent Poisson log-link model with current means \((1,3)\), counts \((2,1)\), and intercept-only design, calculate the score, information, and coefficient increment. (Answer: score \(-1\); information \(4\); increment \(-0.25\).)