Newton-Raphson For Maximum Likelihood

Why Newton–Raphson Uses Local Curvature

Maximum likelihood estimation identifies parameter values that make the observed data relatively likely under a specified model. In nonlinear likelihoods, the score equations often cannot be solved directly, so numerical optimization is used. Newton–Raphson uses both the current slope of the log likelihood and its local curvature to propose a parameter update.

The curvature matters. The update is obtained from the stationary point of a local quadratic approximation. It is the maximum of that approximation only when the Hessian is negative definite. If the curvature has the wrong sign or is indefinite, the same algebra does not justify treating the update as a maximizing step.

Definition: Let \(\ell(\theta)\) be the log likelihood for a parameter vector \(\theta\), and suppose it is twice differentiable near the current value. The score is \(s(\theta)=\nabla \ell(\theta)\), the column vector of first derivatives. The Hessian is \(H(\theta)=\nabla^2\ell(\theta)\), the matrix of second derivatives. The observed information is \(J(\theta)=-H(\theta)\).

For a scalar parameter, the score is the first derivative and the Hessian is the second derivative. For a vector of \(k\) parameters, the score has \(k\) entries and the Hessian is a \(k\times k\) matrix. All derivatives below are evaluated at the current iterate unless indicated otherwise.

Deriving the Newton–Raphson Update

Write the proposed change in the parameter as \(d\). A second-order Taylor expansion of the log likelihood around the current value \(\theta\) gives the local quadratic approximation

q(d)=\ell(\theta)+s(\theta)^\top d+\frac{1}{2}d^\top H(\theta)d.

To find a stationary point of this quadratic, differentiate with respect to \(d\). Its gradient is \(s(\theta)+H(\theta)d\). If \(H(\theta)\) is invertible, setting that gradient to zero gives

s(\theta)+H(\theta)d=0 \quad\Longrightarrow\quad d=-H(\theta)^{-1}s(\theta).

Therefore the Newton–Raphson update is

\theta_{\text{new}} = \theta-H(\theta)^{-1}s(\theta) = \theta+J(\theta)^{-1}s(\theta).

Curvature condition: If \(H(\theta)\) is negative definite, then \(q(d)\) is a strictly concave quadratic and its stationary point is its unique maximum. If the Hessian is invertible but not negative definite, the formula still gives the quadratic’s stationary point, but that point need not be a maximum. If the Hessian is singular, this inverse-based update is not defined.

The update also follows by applying Newton’s method to the score equation \(s(\theta)=0\). Linearizing the score at the current iterate gives \(s(\theta+d)\approx s(\theta)+H(\theta)d\). Solving this linearized equation yields the same \(d\). This explains why Newton–Raphson can be used to find a likelihood stationary point even when the local quadratic does not have a maximum.

An update is a proposal, not a certificate that the likelihood has increased or that a maximum has been found. Even when the Hessian is negative definite at the current iterate, the quadratic approximation is local; the actual log likelihood may behave differently farther away. A successful calculation must be interpreted in light of the likelihood and the model’s parameter space.

Worked Scalar Update

Worked Example: Updating a Poisson Rate

Suppose three independent hypothetical counts are \(y_1=1\), \(y_2=2\), and \(y_3=3\), with each count modeled as Poisson with rate \(\lambda>0\). Use \(\eta=\log\lambda\), so \(\lambda=e^\eta\), and start at \(\lambda_0=1.5\). Compute one Newton–Raphson update for \(\eta\).

For these data, \(\sum_i y_i=6\). Apart from the data-only constant \(-\sum_i\log(y_i!)\), the log likelihood is

\ell(\eta)=6\eta-3e^\eta+\text{constant}.

Its score and Hessian are

s(\eta)=6-3e^\eta, \qquad H(\eta)=-3e^\eta.

At \(\eta_0=\log(1.5)\approx0.4055\), \(e^{\eta_0}=1.5\), so \(s(\eta_0)=6-3(1.5)=1.5\) and \(H(\eta_0)=-3(1.5)=-4.5\). The Hessian is negative, so the local quadratic is concave. The scalar Newton step is

d_0=-\frac{s(\eta_0)}{H(\eta_0)} =-\frac{1.5}{-4.5} =0.3333.

Thus \(\eta_1=0.4055+0.3333=0.7388\), corresponding to \(\lambda_1=e^{0.7388}\approx2.093\). This is one iteration, not a claim that the update is already the maximum. In fact, the likelihood’s maximizing rate is \(\bar y=6/3=2\), or \(\eta=\log 2\approx0.6931\). The update is close but not exact.

In this example the Hessian is negative for every finite \(\eta\), so each local quadratic is concave. That condition makes each stationary point of the quadratic its maximum; it does not make every full Newton step land at the global likelihood maximum in one iteration. In general nonlinear models, further iterations may be needed.

Worked Multivariate Update

Worked Example: Updating an Intercept and Slope

Consider three hypothetical observations from a normal linear model with known error variance \(1\): \(y_i=\beta_0+\beta_1x_i+\varepsilon_i\), where the errors are independent \(N(0,1)\). The observations are \((x_i,y_i)=(-1,1),(0,2),(1,4)\). Compute one update from \(\beta=(\beta_0,\beta_1)^\top=(1,1)^\top\).

Let \(X\) have rows \((1,x_i)\), and let \(y\) be the vector of outcomes. Ignoring a constant, the log likelihood is \(\ell(\beta)=-\tfrac12(y-X\beta)^\top(y-X\beta)\). Differentiating gives \(s(\beta)=X^\top(y-X\beta)\) and \(H(\beta)=-X^\top X\).

At the starting value, the fitted values are \(0,1,2\), so the residual vector is \(r=y-X\beta=(1,1,2)^\top\). Hence

s(\beta)=X^\top r = \begin{pmatrix} 1&1&1\\ -1&0&1 \end{pmatrix} \begin{pmatrix}1\\1\\2\end{pmatrix} = \begin{pmatrix}4\\1\end{pmatrix}.

For these design points, \(X^\top X=\begin{pmatrix}3&0\\0&2\end{pmatrix}\), so \(H=\begin{pmatrix}-3&0\\0&-2\end{pmatrix}\). This Hessian is negative definite: for every nonzero vector \(v\), \(v^\top Hv=-3v_1^2-2v_2^2<0\). The quadratic approximation therefore has a unique maximum. The step is

d=-H^{-1}s = \begin{pmatrix}1/3&0\\0&1/2\end{pmatrix} \begin{pmatrix}4\\1\end{pmatrix} = \begin{pmatrix}4/3\\1/2\end{pmatrix}.

The updated coefficients are \(\beta_{\text{new}}=(1+4/3,\,1+1/2)^\top=(7/3,\,3/2)^\top\). Here the log likelihood is exactly quadratic in \(\beta\), so its quadratic approximation is the log likelihood itself and this update reaches its maximum. The exactness follows from this model’s quadratic likelihood, not from a general guarantee of Newton–Raphson.

When Curvature Fails

A Hessian with positive curvature in some direction does not describe a local maximum in that direction. If the Hessian is indefinite, its quadratic has both upward- and downward-curving directions, and its stationary point is not a maximum. A positive definite Hessian makes the stationary point a minimum of the quadratic; the quadratic is then unbounded above. These cases matter because the update equation alone only solves the stationary-point equation.

Worked Example: A Stationary Point That Is Not a Maximum

Suppose one hypothetical observation \(x=1\) comes from \(X\sim N(\theta^2,1)\), where \(\theta\) is real. Up to a constant, the log likelihood is \(\ell(\theta)=-\tfrac12(1-\theta^2)^2\). Examine a Newton–Raphson update from \(\theta_0=0\).

Differentiating gives \(s(\theta)=2\theta(1-\theta^2)\) and \(H(\theta)=2-6\theta^2\). At zero, \(s(0)=0\) and \(H(0)=2>0\). The Hessian is invertible, so the formula gives \(d_0=-s(0)/H(0)=0\). But the local quadratic is \(\ell(0)+\tfrac12H(0)d^2=-\tfrac12+d^2\), which curves upward and is unbounded above. Its stationary point is a minimum, not a maximum.

The full log likelihood also makes the failure clear: \(\ell(0)=-1/2\), while \(\ell(1)=\ell(-1)=0\), so \(\theta=0\) is not a likelihood maximum. A zero score by itself does not establish maximization; curvature must be examined as well.

Practical Interpretation and Common Pitfalls

Newton–Raphson can converge rapidly near a well-behaved maximum because the Hessian accounts for how quickly the score changes. Its main limitation at this stage is equally direct: an inverse Hessian provides a direction and distance based on local curvature, but neither the existence of the inverse nor a small score proves that the proposed point maximizes the likelihood.

Calling every Newton step a maximizing step. The update solves the stationary-point equation for the quadratic approximation. It is that approximation’s maximum only when the Hessian is negative definite.
Checking the score but ignoring curvature. A zero score can occur at a maximum, a minimum, or a saddle point. Examine the Hessian at a candidate stationary point.
Treating an invertible Hessian as sufficient. Invertibility permits the algebraic update, but it does not ensure negative curvature or a maximizing direction.
Assuming one step finds the answer. One update generally moves toward a solution; it reaches the maximum exactly in special cases, such as a likelihood that is exactly quadratic in the parameter.

The update is most straightforward when the parameter is interior, the log likelihood is twice differentiable nearby, and the Hessian is nonsingular. The negative-definite condition is the additional local curvature check that turns the stationary point of the quadratic approximation into its maximum. None of these local facts alone establishes that the full likelihood has a unique global maximum.

Key takeaway: Newton–Raphson uses the score and Hessian to find the stationary point of a local quadratic approximation to the log likelihood. That point is the quadratic’s unique maximum when the Hessian is negative definite; without that curvature condition, the update is not necessarily a maximizing step.

Check Your Understanding

Answer each question using the score, Hessian, and curvature distinctions developed above.

Write the Newton–Raphson update in terms of the Hessian and score, and state the condition under which the local quadratic’s stationary point is its unique maximum.
If the Hessian is invertible but positive definite, what does the Newton update identify for the quadratic approximation, and why is that not a maximum?
In the Poisson example, why is the curvature condition satisfied at the starting value \(\lambda_0=1.5\)?
Why did the multivariate normal example reach the maximum in one update, and why should that not be expected for every likelihood?
In the curvature-failure example, the score is zero at \(\theta=0\). Give two pieces of evidence showing that this point is not a likelihood maximum.