Fisher Scoring as a Curvature-Stabilized Update

From Observed Curvature to Expected Curvature

At a current coefficient value, Newton–Raphson uses the curvature of the log-likelihood for the observed data. Fisher scoring changes that calculation: it uses the expected information under the model at the current coefficient value instead. The score remains the same, so the substitution changes the scale and direction of the proposed coefficient update, not the likelihood being optimized.

This replacement can make progress more stable when the observed curvature is strongly affected by the particular residuals in the sample. Expected information averages out those residual-dependent terms. But it is not a universal guarantee of stable or successful full steps: the expected-information matrix must be invertible to define the usual update, and even a well-defined update need not improve the likelihood when taken in full.

Definition: Let $\ell(\beta)$ be a twice differentiable log-likelihood, $U(\beta)=\nabla \ell(\beta)$ its score, and $J(\beta)=-\nabla^2\ell(\beta)$ its observed information. The expected information at $\beta$ is $I(\beta)=E_\beta[J(\beta)]$, with the expectation taken under the model at that coefficient value. When $I(\beta)$ is nonsingular, the Fisher-scoring update is $$ \beta_{\mathrm{new}}=\beta+I(\beta)^{-1}U(\beta). $$

The corresponding Newton–Raphson update uses $J(\beta)^{-1}U(\beta)$ in place of $I(\beta)^{-1}U(\beta)$, when the observed information is nonsingular. Thus both updates use the same score, but generally apply different curvature matrices to it. The expectation defining $I(\beta)$ is conditional on the fixed design and uses the current model parameter; it is not an average over possible design matrices.

Expected Information in a GLM

Consider independent responses $Y_i$ in a GLM with fixed design vectors $x_i$, linear predictors $\eta_i=x_i^\mathsf{T}\beta$, means $\mu_i$, and variance function $\operatorname{Var}(Y_i)=a(\phi)V(\mu_i)$. Assume the likelihood is differentiable in the mean, the link gives a differentiable $\mu_i$ as a function of $\eta_i$, and the dispersion $\phi$ is fixed for this coefficient update. Write $\mu_i'=d\mu_i/d\eta_i$. The score contribution for observation $i$ is $$ U_i(\beta)=x_i (Y_i-\mu_i)q_i,\qquad q_i=\frac{\mu_i'}{a(\phi)V(\mu_i)}. $$

Let $q_i'$ denote the derivative of $q_i$ with respect to $\eta_i$. Differentiating the score contribution with respect to $\beta$ gives an observed-information contribution of $\{\mu_i'q_i-(Y_i-\mu_i)q_i'\}x_ix_i^\mathsf{T}$. Since $E_\beta[Y_i-\mu_i]=0$, its expectation has weight $\mu_i'q_i=(\mu_i')^2/[a(\phi)V(\mu_i)]$.

Theorem: Under the conditions just stated, the expected information for the regression coefficients is $$ I(\beta)=\sum_{i=1}^n w_i x_ix_i^\mathsf{T}, \qquad w_i=\frac{(\mu_i')^2}{a(\phi)V(\mu_i)}. $$ In matrix form, $I(\beta)=X^\mathsf{T}WX$, where $X$ has rows $x_i^\mathsf{T}$ and $W$ is diagonal with entries $w_i$. It is positive semidefinite. If every $w_i>0$ and $X$ has full column rank, it is positive definite.

To verify the curvature calculation, differentiating $(Y_i-\mu_i)q_i$ with respect to $\eta_i$ gives $-\mu_i'q_i+(Y_i-\mu_i)q_i'$. The negative Hessian contribution is therefore the expression above. Taking its expectation removes the residual term because the conditional mean of $Y_i-\mu_i$ is zero. This calculation requires the displayed derivatives and finite expectations; it does not require an interchange of differentiation and integration, because the GLM score and Hessian contributions have been differentiated directly.

Form the score.
Evaluate $U(\beta)$ from the observed responses at the current coefficients.

Form expected information.
Compute the GLM weights $w_i$ at the current fitted means and assemble $X^\mathsf{T}WX$.

Solve for the update.
When $I(\beta)$ is nonsingular, solve $I(\beta)\Delta=U(\beta)$ and set $\beta_{\mathrm{new}}=\beta+\Delta$.

The positive-semidefinite claim follows directly: for any vector $v$, $$ v^\mathsf{T}I(\beta)v =\sum_{i=1}^n w_i(x_i^\mathsf{T}v)^2\geq 0. $$ If every weight is positive and $X$ has full column rank, then $Xv\neq 0$ for every nonzero $v$, so the sum is strictly positive. These conditions give positive definiteness and a unique Fisher update. They do not, by themselves, say that a full update will improve the likelihood.

Worked Updates and Curvature Comparisons

Worked Example: A Probit Intercept Update

Suppose one Bernoulli observation is $y=1$, and the model is probit with intercept $\beta$, so $p=P(Y=1)=\Phi(\beta)$. At the current value $\beta=1$, use $\Phi(1)\approx0.8413447$ and the standard normal density $\varphi(1)\approx0.2419707$. For this model, the score is $U=(y-p)q$, where $q=\varphi(\beta)/[p(1-p)]$.

At $\beta=1$, $p(1-p)\approx0.133484$, so $q\approx0.2419707/0.133484=1.8127$. Thus $U\approx(1-0.8413447)(1.8128)=0.2876$. The expected information is $I=\varphi(1)^2/[p(1-p)]\approx0.4386$, giving the Fisher increment $$ \Delta_F=\frac{U}{I}\approx\frac{0.2876}{0.4386}=0.656. $$

For comparison, $q'/q=-\beta-q(1-2p)$, so at this point $q'\approx0.431$. The observed information is $J=I-(y-p)q'\approx0.4386-(0.1586553)(0.431)=0.3702$. The Newton increment is therefore approximately $0.2876/0.3702=0.777$. The two updates differ because the observed curvature contains the residual-dependent term, while expected information does not. This calculation illustrates the substitution; it does not establish that either full step is preferable in every data set.

Worked Example: Expected Curvature Can Avoid a Severe Newton Step

Let $Y$ have an exponential distribution with mean $\mu=e^\beta$, which is a Gamma GLM with log link and dispersion $1$. Observe $y=0.25$ and start at $\beta=\log 2$, so $\mu=2$. Ignoring terms constant in $\beta$, the log-likelihood is $\ell(\beta)=-\beta-y e^{-\beta}$. Its score is $U=-1+y/\mu=-0.875$, and its observed information is $J=y/\mu=0.125$. The expected information is $I=1$.

The Newton increment is $U/J=-7$, while the Fisher increment is $U/I=-0.875$. The current log-likelihood, omitting constants, is $-\log 2-0.25/2\approx-0.8181$. A full Newton step gives $\beta_{\mathrm{new}}=\log 2-7$, hence $\mu_{\mathrm{new}}=2e^{-7}\approx0.001824$, and log-likelihood approximately $6.3069-0.25/0.001824\approx-130.8$: a large decrease.

The full Fisher step instead gives $\beta_{\mathrm{new}}=\log 2-0.875\approx-0.1819$, or $\mu_{\mathrm{new}}\approx0.8337$. Its log-likelihood is approximately $0.1819-0.25/0.8337=-0.1180$, an increase from $-0.8181$. Here expected curvature gives a much less extreme and improving step. This is an example of stabilization, not a guarantee that every Fisher step improves the likelihood.

Worked Example: Solving a Two-Coefficient Fisher Update

Take two independent exponential responses with means $\mu_i=e^{x_i^\mathsf{T}\beta}$, dispersion $1$, and observations $y_1=2$, $y_2=0.5$. Let $x_1=(1,0)^\mathsf{T}$, $x_2=(1,1)^\mathsf{T}$, and start at $\beta=(0,0)^\mathsf{T}$. Both fitted means are $1$. The score contribution is $x_i(y_i/\mu_i-1)$, so $$ U=(1,0)^\mathsf{T}+(-0.5)(1,1)^\mathsf{T}=(0.5,-0.5)^\mathsf{T}. $$

For this model and link, each expected-information weight is $1$. Therefore $$ I=x_1x_1^\mathsf{T}+x_2x_2^\mathsf{T} =\begin{pmatrix}2&1\\1&1\end{pmatrix}. $$ Solving $I\Delta=U$ gives $\Delta=(1,-1.5)^\mathsf{T}$, since $(2-1,1-1.5)^\mathsf{T}=(1,-0.5)^\mathsf{T}$ is not the right check; directly multiplying gives $I\Delta=(2-1.5,1-1.5)^\mathsf{T}=(0.5,-0.5)^\mathsf{T}$, as required. Thus the updated coefficients are $(1,-1.5)^\mathsf{T}$.

The initial log-likelihood, omitting constants, is $-2-0.5=-2.5$. After updating, the two linear predictors are $1$ and $-0.5$; the log-likelihood is $$ (-1-2e^{-1})-\bigl(-0.5+0.5e^{0.5}\bigr) =-0.5-\frac{2}{e}-0.5e^{0.5} \approx-2.0601. $$ This full Fisher step improves the likelihood by about $0.4399$. The matrix calculation is also a reminder that a usable update depends on the information matrix and the design together.

When the Direction Is an Ascent Direction

There is a precise local guarantee, provided expected information is positive definite. It concerns sufficiently short movement in the Fisher direction, not necessarily the full step.

Theorem: Suppose $\ell$ is differentiable at $\beta$, $I(\beta)$ is symmetric positive definite, and $U(\beta)\neq0$. For $\Delta=I(\beta)^{-1}U(\beta)$, sufficiently small positive $t$ satisfies $\ell(\beta+t\Delta)>\ell(\beta)$. If $U(\beta)=0$, then $\Delta=0$ and the directional derivative along this update is zero.

For the proof, positive definiteness of $I$ implies positive definiteness of $I^{-1}$. Consequently, when $U\neq0$, $$ \left.\frac{d}{dt}\ell(\beta+t\Delta)\right|_{t=0} =U^\mathsf{T}\Delta =U^\mathsf{T}I^{-1}U>0. $$ Differentiability gives $\ell(\beta+t\Delta)=\ell(\beta)+tU^\mathsf{T}\Delta+o(t)$ as $t$ tends to zero. Since the coefficient of $t$ is strictly positive, the likelihood increases for all sufficiently small positive $t$. If $U=0$, then $\Delta=I^{-1}0=0$, so the update direction has derivative zero. $\square$

Why the Substitution Matters—and Its Limits

The expected-information weights in a GLM are nonnegative, so the weighted design matrix has a transparent structure. When the design has full column rank and all weights are positive, the matrix is positive definite and the Fisher update is uniquely defined. In settings where observed curvature varies sharply with the realized residuals, its expectation can yield a smoother curvature calculation and avoid an erratic Newton increment, as the exponential example demonstrated.

The guarantee is limited. Positive definiteness and a nonzero score imply a unique update and a strict local ascent direction, but not an improving full step. A full update may travel far beyond the neighborhood in which the directional argument applies. Nor does positive semidefiniteness guarantee an invertible matrix: rank-deficient designs or zero weights can make expected information singular. Those cases require additional computational choices rather than the ordinary inverse in the Fisher-scoring formula.

Mistake: “Expected information is positive, so the full update must improve the likelihood.” Positive definiteness gives a strict ascent direction only when the score is nonzero, and the proof applies to sufficiently short moves. It does not certify the full step.
Mistake: “A zero score gives a strict ascent direction.” If $U=0$ and $I$ is invertible, the Fisher increment is zero; its directional derivative is zero.
Mistake: “Expected information is always positive definite.” It is positive semidefinite in the GLM weighted-design form. Full rank and positive weights are sufficient for positive definiteness, but may not hold.

Key takeaway: Fisher scoring replaces observed information with its model expectation, producing a GLM weighted-design curvature matrix. When that matrix is positive definite and the score is nonzero, the update is uniquely defined and points in a strict local ascent direction—but that does not guarantee an improving full step.

Check Your Understanding

Use the definitions and results above to answer the following questions.

In the Fisher-scoring formula, which quantity is evaluated from the observed responses, and which quantity is averaged under the model?
Why does the residual-dependent term in the GLM observed information disappear when taking its expectation?
State sufficient conditions for $X^\mathsf{T}WX$ to be positive definite.
If $I$ is positive definite but $U=0$, what is the Fisher increment, and what is the directional derivative?
Why does a strict ascent direction not prove that the full Fisher update improves the likelihood?