Newton Versus Fisher Scoring for Logit and Poisson Fits

Why Compare the Two Updates?

Newton–Raphson and Fisher scoring use curvature to decide how to change GLM coefficients. Newton–Raphson uses the observed curvature of the log-likelihood at the current estimate; Fisher scoring replaces it with its expectation under the fitted model. That replacement can matter: the matrices need not be the same, and different curvature estimates can yield different step sizes and directions.

For canonical-link logistic and Poisson regression, however, the two curvature matrices coincide. Consequently, when both methods start at the same coefficient vector and their information matrix is invertible, they take the same steps. Worked calculations make that fact concrete and show what it does—and does not—tell us about likelihood progress and convergence.

The Two Iterations

Let \(\ell(\beta)\) be the log-likelihood for a coefficient vector \(\beta\), and let \(U(\beta)=\nabla\ell(\beta)\) be its score. The observed information is the negative Hessian, \(J(\beta)=-\nabla^2\ell(\beta)\); the expected Fisher information is \(I(\beta)=E[J(\beta)]\), with expectation taken under the model at \(\beta\). Provided the relevant matrix is invertible, the two updates are

\begin{aligned} \text{Newton–Raphson:}\qquad &\beta_{\mathrm{new}}=\beta+J(\beta)^{-1}U(\beta),\\ \text{Fisher scoring:}\qquad &\beta_{\mathrm{new}}=\beta+I(\beta)^{-1}U(\beta). \end{aligned}

The plus signs follow from defining information as negative curvature: the Newton step solves \(J(\beta)\,\Delta=U(\beta)\). If a method uses a damped or otherwise modified step, it is no longer simply this full update.

Definition: An iteration is a full Newton step or full Fisher-scoring step when it uses the update above without shortening or modifying \(\Delta\). The update size can be reported as the coefficient change \(\Delta=\beta_{\mathrm{new}}-\beta\), or, for a vector, its Euclidean norm \(\|\Delta\|_2\). Always give the coefficient components as well: a single norm does not show which coefficient moved.

When Canonical Logit and Poisson Updates Agree

For a canonical exponential-family GLM with fixed design matrix \(X\), fixed dispersion \(\phi\), and linear predictor \(\eta_i=x_i^\top\beta\) (an offset can be added without changing the argument), the log-likelihood has the form

\ell(\beta)=\sum_i\frac{y_i\eta_i-b(\eta_i)}{\phi}+C(y,\phi).

Here \(\mu_i=b'(\eta_i)\), and \(b''(\eta_i)>0\) for the usual nondegenerate observations. The canonical link means that the natural parameter \(\eta_i\) is the linear predictor. Logistic regression with the logit link and Poisson regression with the log link have this property.

Theorem: In a canonical-link exponential-family GLM with fixed design and dispersion, the observed information and expected Fisher information for \(\beta\) are both \(X^\top W X\), where \(W\) is diagonal with entries \(b''(\eta_i)/\phi\). Thus, whenever this matrix is invertible, Newton–Raphson and Fisher scoring give the same full coefficient update.

Proof. Since \(\partial\eta_i/\partial\beta=x_i\), differentiating the displayed log-likelihood gives

U(\beta)=\sum_i x_i\frac{y_i-b'(\eta_i)}{\phi} =X^\top\left(\frac{y-\mu}{\phi}\right).

Differentiating once more, using \(\partial b'(\eta_i)/\partial\beta=b''(\eta_i)x_i\), gives

\nabla^2\ell(\beta) =-\sum_i x_i x_i^\top\frac{b''(\eta_i)}{\phi} =-X^\top W X.

This Hessian depends on the data only through the fixed design and the current \(\beta\), not through the observed responses \(y_i\). Therefore the observed information is \(J(\beta)=X^\top W X\). Its expectation under the model is the same matrix, so \(I(\beta)=J(\beta)\). Substituting equal matrices into the two update equations proves the claim when the matrix is invertible. \(\square\)

One immediate consequence is that, in exact arithmetic, the whole sequences—not just the first updates—match if the methods have the same start and use the same full-step rule. Indeed, equal starting vectors give equal score and information matrices and hence equal next vectors. Applying the same argument at each subsequent equal vector proves the claim by induction, for as long as the updates are defined.

Worked Logistic Regression Iterations

Worked Example: A Binomial Intercept-Only Fit

Suppose \(y=2\) successes are observed in \(m=10\) trials, with one common probability \(p\) and logit \(p=\beta\). Omitting the constant binomial coefficient, the log-likelihood, score, and information are

\ell(\beta)=2\beta-10\log(1+e^\beta),\qquad U(\beta)=2-10p,\qquad J(\beta)=I(\beta)=10p(1-p), \quad p=\frac{1}{1+e^{-\beta}}.

Start at \(\beta_0=-1.2\). Then \(p_0\approx0.231475\), so \(U(\beta_0)\approx-0.314752\) and \(J(\beta_0)=I(\beta_0)\approx1.77894\). The common increment is

\Delta_0=\frac{-0.314752}{1.77894}\approx-0.1769, \qquad \beta_1\approx-1.3769.

The log-likelihood changes from \(\ell(\beta_0)\approx-5.0328\) to \(\ell(\beta_1)\approx-5.0041\), an increase of about \(0.0287\). At \(\beta_1=-1.3769\), \(p_1\approx0.201507\), \(U(\beta_1)\approx-0.01507\), and \(J(\beta_1)=I(\beta_1)\approx1.60902\). Thus

\Delta_1\approx\frac{-0.01507}{1.60902}\approx-0.009368, \qquad \beta_2\approx-1.3863.

The exact maximum occurs at \(p=2/10=0.2\), or \(\hat\beta=\log(0.2/0.8)\approx-1.386294\). The second step is much smaller than the first because the score is now close to zero. Both algorithms have the same likelihood progress here because each coefficient vector is identical; this agreement is not a general guarantee that a full step increases likelihood from an arbitrary start.

Worked Poisson Regression Iterations

Worked Example: A Poisson Intercept With Exposure

Suppose two Poisson counts have exposures \(1\) and \(3\), with observed counts summing to \(6\). Use the log link and common intercept \(\beta\), so the means are the exposures multiplied by \(e^\beta\). Omitting terms that do not depend on \(\beta\), the log-likelihood and information are

\ell(\beta)=6\beta-4e^\beta,\qquad U(\beta)=6-4e^\beta,\qquad J(\beta)=I(\beta)=4e^\beta.

At \(\beta_0=0\), the score is \(2\), the information is \(4\), and both methods take \(\Delta_0=2/4=0.5\). Hence \(\beta_1=0.5\). The log-likelihood rises from \(-4\) to

\ell(0.5)=3-4e^{0.5} \approx 3-6.594885=-3.594885.

At \(\beta_1=0.5\), \(U(\beta_1)=6-6.594885\approx-0.594885\) and \(J(\beta_1)=I(\beta_1)\approx6.594885\). The next update is

\Delta_1\approx\frac{-0.594885}{6.594885}\approx-0.09021, \qquad \beta_2\approx0.40979.

The likelihood at \(\beta_2\) is approximately \(-3.5672\), higher than at \(\beta_1\) and close to its maximum. Setting the score to zero gives \(4e^{\hat\beta}=6\), so \(\hat\beta=\log(1.5)\approx0.40547\). The example shows the common trajectory and shrinking steps near the solution. It does not imply that every full Newton or scoring step in every GLM must raise the likelihood.

Worked Example: A Two-Coefficient Logistic Update

Worked Example: Comparing the Full Update Vectors

Consider two binomial groups, each with four trials. Their covariate values are \(x=0\) and \(x=1\), and their success counts are \(1\) and \(3\), respectively. Fit \(\eta=\beta_0+\beta_1x\), starting from \(\beta=(0,0)^\top\). Both fitted probabilities are \(0.5\). The group score contributions \(y-mp\) are \(-1\) and \(1\), so

U(\beta)= \begin{pmatrix} -1+1\\ 0(-1)+1(1) \end{pmatrix} = \begin{pmatrix}0\\1\end{pmatrix}.

Each group's information weight is \(mp(1-p)=4(0.5)(0.5)=1\). With design rows \((1,0)\) and \((1,1)\), the information matrix and its inverse are

J(\beta)=I(\beta)= \begin{pmatrix}2&1\\1&1\end{pmatrix}, \qquad J(\beta)^{-1}=I(\beta)^{-1}= \begin{pmatrix}1&-1\\-1&2\end{pmatrix}.

Multiplying by the score gives \(\Delta=(-1,2)^\top\), with norm \(\|\Delta\|_2=\sqrt{5}\approx2.236\). Thus both methods move to \((-1,2)^\top\). The initial log-likelihood, omitting binomial constants, is \(-8\log 2\approx-5.5452\). At the updated coefficients, the two linear predictors are \(-1\) and \(1\), and

\ell(-1,2) =\{-1-4\log(1+e^{-1})\} +\{3-4\log(1+e)\} \approx-2.2530-2.2530=-4.5061.

The full step increases the likelihood in this example. The fitted probabilities at the finite maximum are \(1/4\) and \(3/4\), corresponding to \(\hat\beta_0=\log(1/3)\approx-1.0986\) and \(\hat\beta_1=\log(3)-\log(1/3)\approx2.1972\). The starting update is sizable, but it moves toward that solution. Its size alone cannot establish whether a step is safe in a different data set.

Interpreting Agreement and Convergence

For the canonical logit and Poisson models above, observed and expected information agree at every finite coefficient vector. So, when the matrix is invertible and the same full-step algorithm is used, differences in update size or likelihood progress cannot be attributed to choosing Newton–Raphson rather than Fisher scoring: the updates are the same. In actual software, small numerical differences can still arise from rounding, linear-system solvers, or additional safeguards.

Equality of updates is not a convergence theorem. The methods can share a poor starting value, encounter an ill-conditioned or singular information matrix, take a step that reduces likelihood, or fail to approach a finite maximum in a model with a boundary fit. Nor does the observed increase in a worked example establish that every later full step will increase likelihood. Convergence should be assessed using the score, step size, and likelihood behavior together, rather than inferred from the algorithm’s name.

Common Mistake

Wrong reasoning: “Fisher scoring uses expected information, so it must take smaller, safer steps than Newton–Raphson in logistic regression.”
Correction: For the canonical logit model, the observed and expected information matrices are equal. With matching starts and full updates, the methods therefore take exactly the same steps. A smaller or safeguarded update would require an additional mechanism, not merely Fisher scoring.

Key takeaway: Compare the actual score and information matrices before attributing different behavior to Newton–Raphson and Fisher scoring. In canonical logistic and Poisson regression they coincide; identical steps still do not guarantee convergence from every start.

Check Your Understanding

Use the update equations and examples to answer these questions.

What definitions of observed and expected information make the Newton and Fisher-scoring updates have the same plus-sign form?
For the binomial intercept-only example, why do both methods take the same second step as well as the same first step?
In the Poisson example, calculate the score and information at \(\beta=0.5\). What is the resulting increment?
In the two-coefficient logistic example, what does the norm \(\sqrt{5}\) tell you, and what does it leave out?
Why does equality of the two information matrices not prove that a full step always increases the likelihood or that the iterations always converge?