Newton Updates and Their Failure Modes

Why Observed Curvature Matters

Likelihood maximization is often carried out by iteratively updating a parameter estimate. Newton–Raphson chooses each update using two quantities at the current parameter value: the score, which indicates the direction of local increase, and the observed information, which measures the local curvature of the log-likelihood. This curvature adjustment can make Newton updates fast near a maximum, but it can also make them dangerously large when the likelihood is nearly flat.

The central distinction is between an update that points uphill locally and an update that improves the likelihood when taken in full. A Newton direction can be an ascent direction while its full step lands far away, where the likelihood is much smaller. This is especially relevant in generalized linear models when fitted means or probabilities are extreme, or when the starting value gives very little observed curvature.

Deriving the Newton–Raphson Update

Definition: Let $\ell(\beta)$ be a twice differentiable log-likelihood for a parameter vector $\beta$. Its score is $U(\beta)=\nabla\ell(\beta)$, its Hessian is $H(\beta)=\nabla^2\ell(\beta)$, and its observed information is $J(\beta)=-H(\beta)$. When $J(\beta)$ is invertible, the Newton–Raphson update for maximizing $\ell$ is $$ \beta_{\mathrm{new}}=\beta+J(\beta)^{-1}U(\beta). $$

The sign in this expression matters. The Newton equation seeks a zero of the score. A first-order expansion gives $U(\beta+h)\approx U(\beta)+H(\beta)h$. Setting this approximation equal to zero gives $h=-H(\beta)^{-1}U(\beta)=J(\beta)^{-1}U(\beta)$.

The same update follows by maximizing a local quadratic approximation to the log-likelihood. For a small displacement $h$, the approximation is

\ell(\beta+h)\approx \ell(\beta)+U(\beta)^\mathsf{T}h-\frac{1}{2}h^\mathsf{T}J(\beta)h.

If $J(\beta)$ is positive definite, this quadratic has a unique maximizer. Differentiating it with respect to $h$ gives $U(\beta)-J(\beta)h=0$, so its maximizer is $h=J(\beta)^{-1}U(\beta)$. This is the Newton increment.

Observed Information in a Canonical Model

For independent observations in a canonical exponential-family model, write the linear predictor as $\eta_i=x_i^\mathsf{T}\beta$, and suppose the contribution to the log-likelihood, up to a term not depending on $\beta$, is $\{y_i\eta_i-b(\eta_i)\}/a(\phi)$. Here $a(\phi)>0$ is the dispersion factor, and the mean is $\mu_i=b'(\eta_i)$. Differentiating the log-likelihood gives

U(\beta)=\frac{1}{a(\phi)}\sum_{i=1}^n x_i(y_i-\mu_i), \qquad J(\beta)=\frac{1}{a(\phi)}\sum_{i=1}^n x_ix_i^\mathsf{T}b''(\eta_i).

Indeed, differentiating $y_i\eta_i-b(\eta_i)$ with respect to $\beta$ gives $x_i(y_i-b'(\eta_i))$; differentiating again gives $-x_ix_i^\mathsf{T}b''(\eta_i)$. Thus the observed information has the displayed sign. In this canonical model it depends on the current fitted values through $\eta_i$. If the design matrix has full column rank and $b''(\eta_i)>0$ for every observation, then $J(\beta)$ is positive definite: for any nonzero vector $v$,

v^\mathsf{T}J(\beta)v =\frac{1}{a(\phi)}\sum_{i=1}^n b''(\eta_i)(x_i^\mathsf{T}v)^2>0.

This condition makes the Newton direction well defined and locally uphill, but it does not guarantee that the full update increases the actual log-likelihood. Nor does it guarantee that the matrix is numerically well conditioned. In models where observed and expected information differ, replacing observed information with expected information gives Fisher scoring; that is a different update, even though the two can agree in canonical models.

Worked Example: One Poisson Newton Update

Suppose $Y\sim\operatorname{Poisson}(e^\beta)$ and the observed count is $y=2$. Ignoring the constant $-\log(2!)$, the log-likelihood, score, and observed information are

\ell(\beta)=2\beta-e^\beta,\qquad U(\beta)=2-e^\beta,\qquad J(\beta)=e^\beta.

Starting at $\beta=1$, the increment and next iterate are

\Delta=\frac{2-e}{e}=\frac{2}{e}-1, \qquad \beta_{\mathrm{new}}=1+\Delta=\frac{2}{e}\approx0.7357589.

At the new iterate, $e^{\beta_{\mathrm{new}}}\approx2.0870647$, so $\ell(\beta_{\mathrm{new}})\approx2(0.7357589)-2.0870647=-0.6155469$. At the starting value, $\ell(1)=2-e\approx-0.7182818$. The update increases the log-likelihood in this case. The check uses the likelihood itself; the Newton equation alone does not certify improvement.

What Newton Guarantees—and What It Does Not

Theorem: Suppose $\ell$ is continuously differentiable in a neighborhood of $\beta$, its score there is $U(\beta)$, and $J(\beta)$ is positive definite. If $U(\beta)\ne0$, then the Newton direction $d=J(\beta)^{-1}U(\beta)$ is a strict ascent direction. For all sufficiently small positive $t$, $\ell(\beta+td)>\ell(\beta)$. This result does not assert that $t=1$ improves the log-likelihood.

To prove the claim, first compute the directional derivative:

U(\beta)^\mathsf{T}d =U(\beta)^\mathsf{T}J(\beta)^{-1}U(\beta)>0.

The inequality holds because $J(\beta)^{-1}$ is positive definite and $U(\beta)\ne0$. Differentiability now gives $\ell(\beta+td)-\ell(\beta)=tU(\beta)^\mathsf{T}d+o(t)$ as $t\downarrow0$, where $o(t)/t\to0$. Since the coefficient of $t$ is positive, the difference is positive for all sufficiently small $t>0$. This proves local ascent, but says nothing about how far the full step $d$ travels. $\square$

This argument also explains why step-halving can help. For example, a line search tests $t=1,\tfrac12,\tfrac14,\ldots$ and accepts a step only when it satisfies a chosen improvement condition. Under the theorem’s assumptions, sufficiently small steps increase the likelihood, so this procedure can avoid accepting an excessively long full update.

Worked Example: Extreme Fitted Probability in a Logistic Model

Suppose there are $n=10$ independent Bernoulli observations with common success probability $p=e^\beta/(1+e^\beta)$, and exactly $y=1$ success is observed. Up to constants, the log-likelihood and its derivatives are

\ell(\beta)=\beta-10\log(1+e^\beta),\qquad U(\beta)=1-10p,\qquad J(\beta)=10p(1-p).

At the starting value $\beta=10$, $p\approx0.999954602$. Consequently, $U(10)\approx-8.99954602$ and $J(10)\approx0.00045395808$. The Newton increment is

\Delta=\frac{U(10)}{J(10)} \approx -19824.6192, \qquad \beta_{\mathrm{new}}\approx10-19824.6192=-19814.6192.

The tiny observed information makes a moderate-sized score produce an enormous step. The starting log-likelihood is approximately $10-10\log(1+e^{10})=-90.0004540$. At the new, very negative iterate, $\log(1+e^{\beta_{\mathrm{new}}})$ is essentially zero, so the log-likelihood is approximately $-19814.6192$: it has fallen dramatically. The score and information gave a locally uphill direction, but the full step went far beyond the region where that local approximation is useful.

Worked Example: Small Curvature in a Poisson Model

Return to the Poisson model with observed count $y=2$, but start at $\beta=-5$. The fitted mean is $e^{-5}\approx0.006737947$, so the observed information is very small. The Newton increment is exactly

\Delta=\frac{2-e^{-5}}{e^{-5}}=2e^5-1\approx295.826318, \qquad \beta_{\mathrm{new}}\approx290.826318.

At the starting value, $\ell(-5)=-10-e^{-5}\approx-10.0067379$. The full update creates a fitted mean $e^{290.826318}$, which is enormous; the new log-likelihood $2(290.826318)-e^{290.826318}$ is therefore far below its starting value. Yet a much shorter step in the same direction can help. Taking a step of length $5$ in $\beta$, which corresponds to a fraction $5/295.826318\approx0.01690$ of the Newton increment, reaches $\beta=0$, where $\ell(0)=-1$. This is an increase from $-10.0067379$. Step length, not just direction, is decisive.

Failure Modes and Practical Checks

Two common warning signs are a very small eigenvalue of the observed information and extreme fitted means or probabilities. Small curvature makes $J^{-1}$ large, so even a moderate score can produce a huge update. In multivariable models, near-collinearity among predictor directions can also make $J$ poorly conditioned: some combinations of coefficients then change the likelihood only slightly, and their update components can be highly sensitive to numerical error. A matrix can be positive definite in exact arithmetic and still be difficult to invert reliably.

A common mistake is to reason, “The score points uphill and the log-likelihood is concave, so a full Newton step must increase the likelihood.” The first statement establishes only a positive directional derivative at the current point. Concavity does not say that the quadratic approximation remains accurate along the entire step. The logistic example above has positive information and a concave log-likelihood at every finite $\beta$, yet its full Newton step sharply decreases the likelihood.

In practice, assess both the proposed step and the objective. Useful safeguards include starting from a plausible estimate, solving linear systems rather than explicitly forming $J^{-1}$, scaling predictors, monitoring the score and log-likelihood, and using step-halving or a trust-region method when a full step is too aggressive. If fitted means are so extreme that exponentials overflow or probabilities round to zero or one, stable numerical evaluations are essential. A failed or stalled iteration is a warning about the computation or the model’s geometry; it is not by itself evidence that a maximum-likelihood estimate exists or does not exist.

Key takeaway: Newton–Raphson uses observed curvature to scale the score, and a positive-definite observed information matrix makes the resulting direction locally uphill. It does not guarantee that the full update improves the likelihood. Check curvature, fitted values, and the actual objective, and control the step when needed.

Check Your Understanding

Use the definitions and examples above to explain each answer, rather than relying only on a formula.

Starting from a local quadratic approximation to $\ell(\beta)$, derive the Newton increment in terms of the score and observed information.
Why does positive definiteness of $J(\beta)$ make the Newton direction an ascent direction, and why does it not guarantee improvement for a full step?
In the logistic example, what role does the very small observed information play in the size of the update?
For the Poisson example started at $\beta=-5$, explain why a step to $\beta=0$ improves the log-likelihood even though the full Newton step does not.
Name two practical checks or safeguards that can make Newton iterations more reliable, and explain what failure mode each addresses.