Why Canonical Links Can Make Newton Equal Fisher Scoring

Why the Link Changes the Curvature

Newton–Raphson uses the curvature of the log-likelihood at the observed data, while Fisher scoring uses its expectation under the model. These matrices need not generally agree. In a canonical-link generalized linear model with fixed dispersion, however, the regression-coefficient Hessian has no data-dependent residual term. The observed and expected information therefore coincide at every coefficient value where the model is defined—not only at the maximum-likelihood estimate.

This is a statement about the curvature for the regression coefficients. It does not say that the response data do not matter: the data still determine the score and hence the update direction. Nor does it assert the same equality for dispersion parameters or for every noncanonical link.

Here \(x_i\) is the fixed predictor vector for observation \(i\), and \(\beta\) is the coefficient vector. The form above allows different fixed positive scale factors \(a_i(\phi)\); for example, known prior weights can be incorporated into them. We assume the model is regular enough that its stated likelihood derivatives exist. The exponential-family mean and variance identities are \(E_\beta[Y_i]=b'(\theta_i)\) and \(\operatorname{Var}_\beta(Y_i)=a_i(\phi)b''(\theta_i)\).

Deriving the Score and Hessian

Under the canonical link, \(\theta_i=x_i^\mathsf{T}\beta\). Since \(x_i\) does not depend on \(\beta\), differentiating the contribution \(\ell_i\) gives

Differentiating once more gives the Hessian contribution:

The response \(Y_i\) appears in the score but not in this Hessian. That cancellation is the key fact. Summing over independent observations yields the score vector \(U(\beta)=\nabla_\beta\ell(\beta)\) and the information matrix

Proof. For each observation, the second derivative derived above is \(-b''(\theta_i)x_i x_i^\mathsf{T}/a_i(\phi)\), which is a function of the fixed design, \(\beta\), and fixed dispersion only. Thus \(J(\beta)\) is exactly the displayed sum for every possible response vector. Taking its expectation under the model leaves it unchanged: \(I(\beta)=E_\beta[J(\beta)]=J(\beta)\). This proves the claim. \(\square\)

The same result can be written using the design matrix \(X\). Let \(W(\beta)\) be diagonal with entries \(b''(x_i^\mathsf{T}\beta)/a_i(\phi)\). Then \(J(\beta)=I(\beta)=X^\mathsf{T}W(\beta)X\). This matrix may be singular if the design does not identify all coefficients; equality still holds, although an update requiring its inverse would then need further attention.

Three Worked Examples

When Newton and Fisher Scoring Give the Same Step

For a nonsingular observed information matrix, a Newton–Raphson coefficient update is \(\beta_{\mathrm{new}}=\beta+J(\beta)^{-1}U(\beta)\). Fisher scoring replaces \(J(\beta)\) by \(I(\beta)\). Under the theorem’s assumptions those matrices are equal, so the two updates agree exactly, provided the inverse exists.

Proof. The Newton increment is \(\Delta_N=J(\beta)^{-1}U(\beta)\), and the Fisher-scoring increment is \(\Delta_F=I(\beta)^{-1}U(\beta)\). By the theorem, \(I(\beta)=J(\beta)\). Since this common matrix is nonsingular, the two inverses are equal, and therefore \(\Delta_N=\Delta_F\). \(\square\)

This conclusion concerns the proposed full coefficient increment. It does not guarantee that every full step increases the likelihood, that the likelihood has a finite maximizer, or that the algorithm converges from every starting value. Those are separate questions about optimization and the likelihood surface.

Why the Canonical Link Matters

The cancellation can be seen by comparing a general link with the canonical one. Write the canonical parameter as a twice-differentiable function \(h(\eta_i)\) of the linear predictor, so \(\theta_i=h(\eta_i)\). Differentiating the likelihood contribution with respect to \(\eta_i\) gives \((Y_i-b'(\theta_i))h'(\eta_i)/a_i(\phi)\). Its second derivative is

Consequently, the observed information contribution for \(\beta\) is

The second term depends on the observed residual. Its expectation is zero because \(E_\beta[Y_i]=b'(\theta_i)\), but it need not be zero for the observed data. With a canonical link, \(h(\eta)=\eta\), so \(h'=1\) and \(h''=0\); the residual-dependent curvature term vanishes identically. This pinpoints the mechanism without claiming that every noncanonical model must have unequal information in every special data set.

Common Mistake and Practical Scope

Common mistake: “The observed information uses the data, so it must differ from expected information; therefore Newton and Fisher scoring always take different steps.” The first statement is not a valid general argument. In a canonical-link GLM the likelihood Hessian for \(\beta\) is independent of the responses, so averaging it over the model changes nothing. The score still uses the observed responses, and both methods use that same score.

The fixed-dispersion qualification also matters. The result established here concerns the regression-coefficient block with \(\phi\) held fixed. If dispersion is estimated jointly, derivatives involving that additional parameter must be considered separately; the coefficient-block equality alone does not establish equality of the full information matrices. Similarly, a non-linear predictor in \(\beta\) can introduce additional second-derivative terms, so the canonical-link result for a linear predictor does not automatically apply.