Local Convergence Rates and the Role of Starting Values

Why Local Convergence Is Not a Global Guarantee

Newton’s method is attractive for likelihood optimization because, near a well-behaved maximum, its error can shrink roughly as the square of the previous error. Once an iterate is sufficiently close, this can mean a rapid gain in accuracy. But the word “near” matters: the local result does not guarantee that an arbitrary starting value will reach that region. The likelihood’s curvature can be weak far from the maximum, making a Newton step very large or sending it to a location where the method behaves poorly.

We formulate the method for a GLM log-likelihood \(\ell(\beta)\), where \(\beta\in\mathbb{R}^p\) is the coefficient vector. Write \(g(\beta)=\nabla\ell(\beta)\) for the score vector and \(H(\beta)=\nabla^2\ell(\beta)\) for the Hessian. When \(H(\beta)\) is nonsingular, a full Newton update for maximizing the log-likelihood is

The update seeks a zero of the score. If \(\beta^*\) is a stationary point, then \(g(\beta^*)=0\). The convergence result below concerns iterates close to such a point when it is a strict local maximum, so \(H(\beta^*)\) is negative definite.

Local Quadratic Convergence

Here \(\|\cdot\|\) is the Euclidean norm for vectors and its induced operator norm for matrices. The Lipschitz condition holds, for example, if the third derivatives of \(\ell\) are bounded on the ball. We prove the error bound directly. Set \(e=\beta-\beta^*\). The fundamental theorem of calculus, applied to the score along the line segment from \(\beta^*\) to \(\beta\), gives

The segment is in \(B\), which is convex, and \(g(\beta^*)=0\). Substituting this expression into the Newton update and subtracting \(\beta^*\) yields

Because \(H(\beta^*)\) is nonsingular and \(H\) is continuous, there is a sufficiently small closed ball around \(\beta^*\) on which \(H(\beta)\) remains nonsingular and \(\|H(\beta)^{-1}\|\le M\) for some finite \(M\). Shrink the ball to lie within \(B\). The Lipschitz condition then gives

Thus the theorem holds with \(C=ML/2\). Negative definiteness of the Hessian at a stationary point implies a strict local maximum by the second-derivative test. \(\square\)

To verify the claim, let \(e_k=\|\beta_k-\beta^*\|\). As long as \(\beta_k\) is in the ball, the theorem gives \(e_{k+1}\le Ce_k^2\le Cr e_k\). Since \(Cr<1\), this implies \(e_{k+1}\le r\), so the next iterate also lies in the ball. Induction shows every update is defined and stays there. Moreover, \(e_k\le (Cr)^k e_0\), which tends to zero. The stronger quadratic bound \(e_{k+1}\le Ce_k^2\) holds at every iteration. \(\square\)

This is a local guarantee: it provides a region where the method works, not a promise that an initial value outside that region will enter it. The radius depends on the likelihood’s local curvature and its variation. It may be small, and it is not generally known in advance.

Worked Newton Updates in GLM Intercept Models

What Starting Values Change

The examples separate two ideas that are easy to conflate. First, a nonsingular Hessian and smooth likelihood near a strict maximum support a local quadratic rate. Second, a Newton update from elsewhere depends on the score and Hessian at that starting point. If the Hessian is nearly singular there, its inverse can magnify the score and create a very large step. The local error bound says nothing about the quality of that step.

This distinction matters in GLM computation because the curvature can vary substantially with the coefficients. For instance, in the binomial intercept example, probabilities near zero make the magnitude of the Hessian \(4p(1-p)\) small. The resulting update from \(-4\) was large even though the model had a finite maximum. A finite maximum and local convergence near it do not imply safe behavior from every initial value.

Practical safeguards such as likelihood-based step acceptance or step-halving can prevent some harmful full steps. They address global movement; they do not change the local theorem’s assumptions. When accepted updates eventually become full Newton steps in the neighborhood where the theorem applies, the quadratic rate can then appear. Assessing an algorithm’s stopping conditions is a separate question from establishing this convergence rate.

Common Mistake

• Wrong reasoning: “Newton’s method converges quadratically, so starting anywhere should produce very fast convergence.” This mistakes a local rate for a global guarantee.
• Correction: Quadratic convergence requires a starting point sufficiently close to a stationary point, along with smoothness and an invertible Hessian nearby. Far away, curvature may be weak or changing, and a full Newton step may be very large or reduce the likelihood. The theorem controls the error only inside a suitable local region.