Newton and Quasi-Newton Parameter Updates

Why Curvature Matters in Parameter Updates

An estimation method may define a population-model parameter estimate as a minimizer of an objective \(F(\theta)\), such as a negative log-likelihood. An iteration must decide not only which direction to move from the current estimate, but also how strongly the objective curves in that direction. Gradient information indicates local direction of increase; curvature information helps scale the step and account for dependence among parameters.

Newton’s method uses the objective’s Hessian, the matrix of second derivatives, at each iteration. A quasi-Newton method instead builds an approximation to curvature from successive parameter and gradient values. Both are local optimization methods: an update is based on information near the current parameter vector, so it does not by itself guarantee that the objective decreases or that the global minimum is found. The earlier material on selecting estimation methods by model and design concerns the broader choice of estimation approach; here the focus is on these parameter updates.

The objective may be defined in a particular parameterization and scale. In the equations below, vectors and matrices refer to that chosen coordinate system. For a maximization objective, such as a log-likelihood, one can equivalently minimize its negative; mixing the maximization and minimization sign conventions without changing the update is a common source of errors.

Deriving the Newton Update

Near \(\theta_k\), a second-order Taylor approximation to the objective at \(\theta_k+d\) is

where \(g_k=g(\theta_k)\) and \(H_k=H(\theta_k)\). If \(H_k\) is symmetric positive definite, this quadratic approximation has a unique minimizer. Differentiating the approximation with respect to \(d\) gives \(g_k+H_kd=0\), so the minimizing displacement solves \(H_kd_k=-g_k\). The implementation should solve this linear system; explicitly forming \(H_k^{-1}\) is usually less numerically desirable than solving the system directly.

Newton’s method is especially effective near a regular strict local minimum, where the Hessian is positive definite and the quadratic approximation is accurate. Under standard smoothness conditions, when started sufficiently close to such a minimum and using the exact Hessian, it has locally quadratic convergence: the error can shrink roughly as the square of its previous size. This is a local result, not a guarantee from an arbitrary starting point. Far from the solution, a Newton step can be too large, increase the objective, or leave the region where the objective is well behaved. Step controls and other safeguards are separate considerations.

Quasi-Newton Updates and the Secant Condition

For many parameters, computing and storing an exact Hessian can be costly, and numerical curvature can be unreliable. Quasi-Newton methods use gradient changes to update an approximation. Define the parameter displacement \(s_k=\theta_{k+1}-\theta_k\) and gradient change \(y_k=g_{k+1}-g_k\), where \(g_{k+1}=g(\theta_{k+1})\). If the objective is locally close to quadratic, its gradient changes approximately according to \(y_k\approx H_{k+1}s_k\). A Hessian approximation \(C_{k+1}\) can therefore be required to satisfy the secant condition \(C_{k+1}s_k=y_k\).

A common implementation stores an approximation \(B_k\) to the inverse Hessian. The inverse secant condition is \(B_{k+1}y_k=s_k\). The BFGS inverse update is

provided \(y_k^\mathsf{T}s_k\ne0\). This rank-two update uses the new displacement and gradient change rather than calculating the full Hessian anew. For minimization, the usual positive-curvature requirement is \(y_k^\mathsf{T}s_k>0\). With a symmetric positive-definite \(B_k\), that condition preserves positive definiteness in the BFGS update.

Choosing and Diagnosing an Update

In either method, inspect the quantities that define the update rather than treating a numerical step as automatically trustworthy. For Newton, check whether the Hessian is symmetric to numerical precision, whether it is nonsingular and well conditioned, and whether the computed direction solves \(H_kd_k=-g_k\) accurately. An indefinite Hessian near a proposed minimum, a very large displacement, or a direction that is not a descent direction signals that the local quadratic model may be unsuitable. For BFGS, check the sign and scale of \(y_k^\mathsf{T}s_k\), and verify that the stored approximation remains symmetric and numerically usable. A tiny or nonpositive curvature quantity can make \(\rho_k\) unstable or invalidate the positive-definiteness argument; implementations may need to modify or skip such an update.

These methods are not interchangeable in every setting. Newton uses current second-derivative information and can be highly efficient near a well-behaved optimum, but exact curvature may be expensive or difficult to obtain for a complex population-model objective. BFGS trades exact curvature for a history-based approximation and is often attractive when gradients are available and the parameter dimension is manageable. Limited-memory BFGS stores a small number of recent displacement and gradient-change pairs, reducing storage for large problems, at the cost of representing curvature less fully. Neither approach removes the need for a suitable objective, informative data, good starting values, or checks on convergence and model adequacy. Stochastic estimation also has gradient variability that can complicate curvature updates; quasi-Newton formulas do not automatically correct noisy gradient differences.

A common mistake is to assume that because Newton’s method uses second derivatives, every Newton step must move toward a minimum. That is false: an indefinite Hessian can produce a direction that is not a descent direction, and a step based on a local approximation can behave poorly away from the solution. Another error is to use the BFGS formula with the wrong sign convention or to ignore the curvature condition. For minimization, define \(y_k=g_{k+1}-g_k\) and \(s_k=\theta_{k+1}-\theta_k\) consistently, then check \(y_k^\mathsf{T}s_k>0\) before relying on the positive-definite update interpretation.