Fisher Scoring And Expected-Information Updates

Why Replace Observed Information?

Maximum-likelihood algorithms use the likelihood’s local curvature to decide how far and in what direction to change parameter estimates. Newton–Raphson uses the curvature observed in the current sample. Fisher scoring instead uses the curvature expected under the model at the current parameter value. This replacement can make the update easier to calculate and, in some problems, less erratic. It does not guarantee that every proposed parameter value is admissible or that every step improves the likelihood.

Let $\ell(\theta)$ be the log-likelihood for a parameter vector $\theta$. The score $U(\theta)$ is its gradient, and the observed information $J(\theta)$ is the negative Hessian of the log-likelihood. The expected information $I(\theta)$ is the expectation of the observed information under the model indexed by $\theta$. These matrices describe curvature, but they are not generally equal for a particular data set.

Definition: A Fisher-scoring update from the current value $\theta^{(t)}$ is $$ \theta^{(t+1)}=\theta^{(t)}+I\!\left(\theta^{(t)}\right)^{-1}U\!\left(\theta^{(t)}\right), $$ provided the expected-information matrix is invertible. Here $U(\theta)=\partial\ell(\theta)/\partial\theta$, and $I(\theta)=E_\theta[J(\theta)]$, where $J(\theta)=-\partial^2\ell(\theta)/\partial\theta\partial\theta'$. The expectation is taken over data generated under the model at the stated parameter value.

The update adds a parameter change to the current value. It is not a direct formula for the maximum-likelihood estimate: unless the procedure reaches a stationary point, the result is only the next iterate. For a scalar parameter, the update is $\theta^{(t+1)}=\theta^{(t)}+U(\theta^{(t)})/I(\theta^{(t)})$. In several dimensions, the information matrix scales and can rotate the score vector.

From Newton–Raphson to Fisher Scoring

Newton–Raphson solves the local quadratic approximation to the log-likelihood. With the negative-Hessian convention for observed information, its update solves $J(\theta^{(t)})\Delta^{(t)}=U(\theta^{(t)})$, then sets $\theta^{(t+1)}=\theta^{(t)}+\Delta^{(t)}$. Fisher scoring makes one substitution: it replaces the sample-specific matrix $J(\theta^{(t)})$ with its expectation $I(\theta^{(t)})$.

J(\theta^{(t)})\Delta_{\mathrm{NR}}^{(t)}=U(\theta^{(t)}), \qquad I(\theta^{(t)})\Delta_{\mathrm{FS}}^{(t)}=U(\theta^{(t)}).

Under the usual regularity conditions for a correctly specified likelihood, the score has mean zero and the information equality gives $I(\theta)=E_\theta[U(\theta)U(\theta)']=-E_\theta[\partial^2\ell(\theta)/\partial\theta\partial\theta']$. The information equality explains why expected information is a meaningful curvature measure; it does not say the realized observed information equals expected information at every parameter value.

Evaluate the score.
At the current parameter value, calculate $U(\theta^{(t)})$, which points toward locally increasing log-likelihood to first order.

Calculate expected information.
Find $I(\theta^{(t)})$ from the model’s expectation, not by substituting the observed Hessian.

Solve for the change.
Solve $I(\theta^{(t)})\Delta^{(t)}=U(\theta^{(t)})$, then add the change to the current parameter vector.

Check the result.
Confirm the proposal lies in the parameter space and assess whether the likelihood and score indicate progress. A valid-looking formula alone does not establish convergence.

Worked Examples

Worked Example: One-Step Scoring for a Normal Mean

Suppose three independent hypothetical observations are $1,3,5$, from $Y_i\sim N(\mu,4)$, with known variance $4$. Starting at $\mu^{(0)}=0$, calculate the Fisher-scoring update. The model assumes independent normal observations with the stated common mean and known variance.

Ignoring terms that do not depend on $\mu$, the log-likelihood is $\ell(\mu)=-\sum_i(y_i-\mu)^2/8+\text{constant}$. Therefore the score is $U(\mu)=\sum_i(y_i-\mu)/4$, and the observed and expected information are both $n/4$, since the second derivative is the nonrandom value $-n/4$.

At $\mu=0$, the score is $U(0)=(1+3+5)/4=9/4=2.25$, and the expected information is $I(0)=3/4=0.75$. Thus

\Delta_{\mathrm{FS}}=\frac{U(0)}{I(0)} =\frac{2.25}{0.75}=3, \qquad \mu^{(1)}=0+3=3.

The update reaches the sample mean, $\bar{x}=3$, which is the maximum-likelihood estimate of $\mu$ when the variance is known. Here Fisher scoring and Newton–Raphson coincide because the observed and expected information are identical at every $\mu$.

Worked Example: A Two-Parameter Normal Likelihood

Consider independent hypothetical observations $y_1=0$ and $y_2=2$ from $N(\mu,v)$, where both the mean $\mu$ and variance $v>0$ are unknown. Compare Newton–Raphson and Fisher scoring from $(\mu,v)=(0,1)$. The observations are independent and normally distributed with a common mean and strictly positive common variance.

Let $S=\sum_i(y_i-\mu)^2$. The log-likelihood, score, and observed information are

\ell(\mu,v)=-\frac{n}{2}\log(2\pi v)-\frac{S}{2v}, \qquad U_\mu=\frac{\sum_i(y_i-\mu)}{v}, \qquad U_v=-\frac{n}{2v}+\frac{S}{2v^2},

J(\mu,v)= \begin{pmatrix} n/v & \sum_i(y_i-\mu)/v^2\\ \sum_i(y_i-\mu)/v^2 & -n/(2v^2)+S/v^3 \end{pmatrix}.

At the starting point, $n=2$, $\sum_i(y_i-\mu)=2$, and $S=0^2+2^2=4$. Hence the score is $(2,1)'$, and the observed information is

J(0,1)= \begin{pmatrix}2&2\\2&3\end{pmatrix}.

In particular, the lower-right entry is $-n/(2v^2)+S/v^3=-1+4=3$. Newton–Raphson solves

\begin{pmatrix}2&2\\2&3\end{pmatrix} \begin{pmatrix}\Delta_\mu\\\Delta_v\end{pmatrix} = \begin{pmatrix}2\\1\end{pmatrix}.

The equations are $2\Delta_\mu+2\Delta_v=2$ and $2\Delta_\mu+3\Delta_v=1$. Subtracting the first equation from the second gives $\Delta_v=-1$, and then $\Delta_\mu=2$. The Newton proposal is therefore $(2,0)$, which is inadmissible because the variance must be strictly positive. Even though the observed-information matrix at the starting point is positive definite (its determinant is $2$), its full Newton step leaves the parameter space.

For Fisher scoring, the expected information for this normal model is

I(\mu,v)= \begin{pmatrix} n/v&0\\ 0&n/(2v^2) \end{pmatrix}.

At $(0,1)$, this is $\begin{pmatrix}2&0\\0&1\end{pmatrix}$. Solving $I\Delta=U$ gives $\Delta_\mu=2/2=1$ and $\Delta_v=1/1=1$, so the Fisher-scoring proposal is $(1,2)$. It is admissible. The update directions differ: Newton–Raphson proposes $(2,-1)$, whereas Fisher scoring proposes $(1,1)$. The latter does not decrease the variance at this step because the expected-information matrix has no cross term.

One more Fisher-scoring calculation shows the role of iteration. At $(1,2)$, the residuals are $-1,1$, so their sum is zero and $S=2$. The score is $U_\mu=0$ and $U_v=-2/(2\cdot2)+2/(2\cdot2^2)=-0.25$. The expected information is diagonal with entries $2/2=1$ and $2/(2\cdot2^2)=0.25$. Thus the next changes are $\Delta_\mu=0$ and $\Delta_v=-0.25/0.25=-1$, giving $(1,1)$. This is the MLE for these data: the fitted mean is $1$, and the MLE variance is $\{(-1)^2+1^2\}/2=1$. This example illustrates a potentially more manageable Fisher step, not a universal guarantee that Fisher scoring stays admissible or converges.

Worked Example: Fisher Scoring for a Poisson Intercept

Suppose hypothetical independent counts are $0,1,3$, with $Y_i\sim\operatorname{Poisson}(\lambda)$, parameterized by $\eta=\log\lambda$, so $\lambda=e^\eta$. Calculate one Fisher-scoring step from $\eta^{(0)}=0$. The model assumes independent Poisson observations sharing the same positive mean.

Writing $T=\sum_i y_i=4$ and $n=3$, the log-likelihood is $\ell(\eta)=T\eta-ne^\eta-\sum_i\log(y_i!)$. The score is $U(\eta)=T-ne^\eta$, and the negative second derivative, hence both observed and expected information, is $I(\eta)=ne^\eta$.

At $\eta=0$, the score is $4-3e^0=1$, while expected information is $3e^0=3$. Therefore $\Delta_{\mathrm{FS}}=1/3$, and the next iterate is $\eta^{(1)}=1/3$, corresponding to $\lambda^{(1)}=e^{1/3}\approx1.396$. The MLE is $\hat{\lambda}=T/n=4/3$, or $\hat{\eta}=\log(4/3)\approx0.2877$, so this first step moves toward but does not reach the estimate. Because observed and expected information coincide in this example, Newton–Raphson gives the same first step.

What the Update Does—and Does Not—Guarantee

Expected information is often positive definite in an identifiable, regular model, giving Fisher scoring a well-defined local scaling of the score. Replacing sample-specific curvature can make the matrix simpler and can avoid some problematic curvature patterns. In the two-parameter normal example, the expected information is diagonal and produces a positive variance proposal where the full Newton step proposed zero.

That example should not be read as a general ranking of the methods. Fisher scoring can still propose a negative variance in another setting, overshoot, or fail to converge. Positive definiteness of the update matrix alone does not ensure that the proposed parameter value is within the parameter space or that the log-likelihood increases. Newton–Raphson may converge rapidly near a well-behaved optimum, while Fisher scoring may be easier to compute or more stable in other regions. The appropriate comparison depends on the likelihood and the starting value.

In both procedures, the score and information must be evaluated at the current iterate. For Fisher scoring, expected information is computed under the model at that iterate; it is not a fixed matrix evaluated only once, and it is not the observed information with a different label. Convergence is assessed through the parameter changes and score, together with the likelihood and the validity of the parameter values.

Common Mistakes

Using the wrong sign for observed information. Observed information is the negative Hessian. In the normal example, $J_{vv}=-n/(2v^2)+S/v^3$; reversing that sign changes the Newton step.
Confusing observed and expected information. Newton–Raphson uses $J(\theta)$; Fisher scoring uses $I(\theta)=E_\theta[J(\theta)]$. They may coincide in a particular model, but not in general.
Treating a proposed update as an estimate. An iteration is only a candidate next value. Check parameter restrictions and convergence rather than declaring the MLE after one step.
Assuming Fisher scoring always improves the likelihood. It does not. The update uses expected local curvature, so a proposed step can still overshoot or be inadmissible.

Key takeaway: Fisher scoring replaces the observed-information matrix in the Newton–Raphson system with expected information evaluated at the current parameter value. This changes the scaling and sometimes the direction of the update; it can improve numerical behavior, but every proposed step still requires validity and convergence checks.

Check Your Understanding

Try these questions using the definitions and calculations above.

State the Fisher-scoring update and define each of its two ingredients.
For the two-observation normal example, why is the Newton proposal $(2,0)$ inadmissible even though the observed-information matrix at the starting value is positive definite?
In the Poisson-intercept example, calculate the Fisher-scoring change if the starting value were $\eta=0$ but the total count were $T=3$, with $n=3$.
When do Newton–Raphson and Fisher scoring give the same update at a particular iteration?
Does positive expected information guarantee that the next iterate increases the log-likelihood? Explain.