Oscillation From Oversized Newton Steps

When a Full Newton Step Crosses the Maximum

Newton’s method uses the score and local curvature to choose a coefficient update. Near a well-behaved maximum, this can be highly efficient. But a local curvature estimate need not describe the likelihood accurately across a large proposed step. A full update can therefore pass across the maximum to the other side. If subsequent updates keep crossing, the coefficient sequence oscillates; depending on the likelihood’s shape, those oscillations may shrink or grow.

For a scalar coefficient \(\theta\), write \(U(\theta)=\ell'(\theta)\) for the score and \(I(\theta)=-\ell''(\theta)\) for the observed information. When \(I(\theta)>0\), the full Newton update for maximizing the log-likelihood is

\theta_{\mathrm{new}}=\theta+\frac{U(\theta)}{I(\theta)}.

The update is defined wherever \(I(\theta)\ne0\); positive observed information makes the Newton direction an ascent direction when the score is nonzero. Strict concavity alone does not guarantee that it is positive at every point. In a vector problem, the corresponding update is \(\boldsymbol{\beta}_{\mathrm{new}}=\boldsymbol{\beta}+I(\boldsymbol{\beta})^{-1}U(\boldsymbol{\beta})\), when the observed information matrix is invertible. The scalar case makes crossing and alternation especially transparent.

Definition: Suppose \(\hat{\theta}\) is a finite likelihood maximum. A coefficient sequence \((\theta_k)\) oscillates across the maximum if successive iterates lie on opposite sides of it: \((\theta_k-\hat{\theta})(\theta_{k+1}-\hat{\theta})<0\). Oscillation describes the locations of iterates, not whether the log-likelihood increases or decreases.

A Curvature Condition for Crossing

The next result isolates the role of changing curvature. Assume that \(\ell\) is twice continuously differentiable on an open interval containing \(\theta\) and an interior maximizer \(\hat{\theta}\), that \(U(\hat{\theta})=0\), and that \(I(t)>0\) throughout the segment between \(\theta\) and \(\hat{\theta}\). These conditions make the full update well-defined at \(\theta\) and allow the integral identity below.

Result: Define the average observed information between \(\theta\) and \(\hat{\theta}\) by \[ \bar I(\theta,\hat{\theta})=\frac{1}{\theta-\hat{\theta}}\int_{\hat{\theta}}^\theta I(t)\,dt \] when \(\theta\ne\hat{\theta}\). The full Newton update satisfies \[ \theta_{\mathrm{new}}-\hat{\theta} =(\theta-\hat{\theta})\left(1-\frac{\bar I(\theta,\hat{\theta})}{I(\theta)}\right). \] Thus it crosses the maximum exactly when \(\bar I(\theta,\hat{\theta})/I(\theta)>1\).

Proof. Since \(U'=-I\) and \(U(\hat{\theta})=0\), the fundamental theorem of calculus gives \[ U(\theta)=-\int_{\hat{\theta}}^\theta I(t)\,dt =-(\theta-\hat{\theta})\bar I(\theta,\hat{\theta}). \] Substitute this identity into the Newton update and subtract \(\hat{\theta}\): \[ \theta_{\mathrm{new}}-\hat{\theta} =(\theta-\hat{\theta})+\frac{U(\theta)}{I(\theta)} =(\theta-\hat{\theta})\left(1-\frac{\bar I(\theta,\hat{\theta})}{I(\theta)}\right). \] Because \(\theta\ne\hat{\theta}\), the old and new errors have opposite signs precisely when the factor in parentheses is negative, which is equivalent to \(\bar I(\theta,\hat{\theta})/I(\theta)>1\). \(\square\)

This condition compares curvature at the current iterate with average curvature along the route to the maximum. If the current curvature is smaller than that average, the Newton displacement can be too large and cross the maximum. If \(I\) is constant, the ratio is one and the update lands exactly at the maximum. Crossing alone does not say whether the next iterate is closer: when the ratio exceeds two, the new error is larger in magnitude than the old one.

Worked Example: Constant Curvature Does Not Oscillate

Consider the concave log-likelihood \(\ell(\theta)=-\tfrac12(\theta-2)^2\), up to an additive constant. Its score is \(U(\theta)=-(\theta-2)\), and its observed information is \(I(\theta)=1>0\) everywhere. Starting at \(\theta_0=5\), \[ \theta_1=5+\frac{-(5-2)}{1}=2. \] The update reaches the maximum in one step. In fact, the ratio of average to current information is \(1/1=1\) at every starting point other than \(2\), matching the crossing result. The absence of oscillation here depends on the likelihood being exactly quadratic, not merely concave.

A Logistic Likelihood Can Alternate

For a concrete GLM example, observe two Bernoulli outcomes, one success and one failure, with a shared intercept \(\beta\) and no other covariates. The success probability is \(p=e^\beta/(1+e^\beta)\). Ignoring no parameter-dependent terms, the log-likelihood is \[ \ell(\beta)=\log(p)+\log(1-p)=-2\log\!\left(2\cosh(\beta/2)\right). \] It has a finite maximum at \(\hat{\beta}=0\). Differentiating gives \(U(\beta)=-\tanh(\beta/2)\) and \(I(\beta)=2p(1-p)=1/[2\cosh^2(\beta/2)]>0\). Since \(U(\beta)/I(\beta)=-\sinh(\beta)\), the full update is \(\beta_{\mathrm{new}}=\beta-\sinh(\beta)\).

Worked Example: Alternation That Converges

Start the logistic model at \(\beta_0=1\). The first full update is \[ \beta_1=1-\sinh(1)\approx 1-1.1752=-0.1752. \] It crosses the maximum. The next update is \[ \beta_2=-0.1752-\sinh(-0.1752) =-0.1752+\sinh(0.1752)\approx 0.0009. \] Thus the coefficient changes sign again, and its magnitude falls from \(1\) to about \(0.1752\), then to about \(0.0009\).

This is not just a short numerical pattern. For \(x>0\), \(\sinh(x)>x\), so \(x-\sinh(x)<0\); by symmetry, every nonzero iterate changes sign. If \(0<x\leq1\), then \(\sinh(x)<2x\): the ratio \(\sinh(x)/x\) is increasing for \(x>0\), and \(\sinh(1)<2\). Consequently \(0<\sinh(x)-x<x\). Starting from magnitude \(1\), the magnitudes therefore decrease and have a limit \(L\in[0,1]\). Continuity of the update implies \(L=\sinh(L)-L\), so \(\sinh(L)=2L\). The inequality \(\sinh(L)<2L\) for \(0<L\leq1\) rules out a positive limit; hence \(L=0\). The iterates alternate while converging to the maximum.

The log-likelihood increases at each of these updates. It depends on \(|\beta|\), and \(\log\cosh(x/2)\) strictly increases for \(x>0\); reducing \(|\beta|\) therefore increases \(\ell(\beta)\). For the first step, the increase is \[ \ell(\beta_1)-\ell(\beta_0) =2\{\log\cosh(0.5)-\log\cosh(0.0876)\}\approx 0.233. \] Alternation and likelihood decrease are different phenomena.

Worked Example: Alternation That Moves Away

Use the same logistic model, but start at \(\beta_0=3\). Since \(\sinh(3)\approx10.0179\), the full update gives \[ \beta_1=3-\sinh(3)\approx-7.0179. \] This crosses the maximum and increases the coefficient’s distance from it from \(3\) to about \(7.0179\). The likelihood falls: \[ \ell(\beta_1)-\ell(\beta_0) =-2\{\log\cosh(7.0179/2)-\log\cosh(1.5)\} \approx -3.92. \] At magnitude \(a\geq3\), the next magnitude is \(\sinh(a)-a\). Since \(\sinh(3)>2(3)\), and \(\sinh(a)/a\) increases for \(a>0\), this next magnitude exceeds \(a\); the same remains true at every subsequent, larger magnitude. The coefficients keep alternating while moving farther from zero, and their likelihood values keep falling. A finite, strictly concave likelihood and positive information do not guarantee that a full Newton step improves the likelihood.

Damping and Likelihood Improvement

A standard remedy for an oversized update is to take only a fraction of the Newton displacement. For \(0<\alpha\leq1\), the damped update is \(\theta_{\mathrm{new}}=\theta+\alpha U(\theta)/I(\theta)\). Step-halving and other likelihood-based line searches choose a step fraction by checking candidate updates; the key fact here is why reducing the step can help.

Result: Suppose \(\ell\) is continuously differentiable near \(\theta\), and \(I(\theta)>0\). If \(U(\theta)\ne0\), then there is an \(\varepsilon>0\) such that every \(0<\alpha\leq\varepsilon\) gives a strictly larger log-likelihood at \(\theta+\alpha U(\theta)/I(\theta)\) than at \(\theta\).

Proof. Let \(d=U(\theta)/I(\theta)\), which is nonzero, and define \(\phi(\alpha)=\ell(\theta+\alpha d)\). The chain rule gives \(\phi'(0)=\ell'(\theta)d=U(\theta)^2/I(\theta)>0\). By continuity of \(\phi'\), there is an \(\varepsilon>0\) for which \(\phi'(\alpha)>0\) on \([0,\varepsilon]\). Integrating this positive derivative shows \(\phi(\alpha)-\phi(0)>0\) for each \(0<\alpha\leq\varepsilon\). \(\square\)

This is a local existence guarantee, not a claim that any particular fixed fraction will work. A line search tests actual likelihood values because the size of an acceptable step depends on the likelihood away from the current coefficient. If the score is zero, the Newton displacement is zero and this argument does not apply; the iterate is already stationary. In multiple dimensions, the corresponding local argument requires a well-defined ascent direction, such as a Newton direction formed with positive-definite observed information.

Worked Example: Damping a Large Logistic Step

At \(\beta_0=3\) in the logistic example, the full Newton displacement is \(d=-\sinh(3)\approx-10.0179\). A trial fraction \(\alpha=0.25\) gives \[ \beta_1=3+0.25(-10.0179)\approx0.4955. \] This damped update does not cross zero. Its likelihood increase is \[ \ell(0.4955)-\ell(3) =2\{\log\cosh(1.5)-\log\cosh(0.2478)\} \approx1.65>0. \] The quarter step substantially improves the likelihood, whereas the full step decreased it by about \(3.92\). This example illustrates a useful remedy; it does not establish that a quarter step is universally adequate. A likelihood-based acceptance rule checks the candidate step in the model at hand.

Common Mistake and Practical Interpretation

Mistake: assuming concavity makes every full Newton update safe. Concavity constrains the shape of the log-likelihood, but it does not ensure that the local curvature at the current iterate predicts the function accurately over a large step. Check whether the likelihood actually increases.
Mistake: treating every sign change as solver failure. As the logistic example starting at \(1\) shows, coefficients can alternate and converge rapidly. Track both the coefficient sequence and the likelihood sequence.
Mistake: claiming observed information is positive everywhere just because the likelihood is strictly concave. For a Newton update, verify positive information at the points where the update is used; strict concavity by itself does not provide that pointwise guarantee.
Mistake: assuming damping always means taking half a step. A fixed fraction may help or fail depending on the local likelihood. Likelihood-based step selection is safer when full steps produce oscillation or a decrease in fit.

Oscillation is therefore a diagnostic, not a verdict. Alternating coefficients with improving likelihood and shrinking steps can be compatible with convergence. Alternation accompanied by growing coefficient magnitudes or falling likelihood indicates that the full steps are not behaving acceptably. Damping limits the distance traveled and, with a suitable likelihood-based acceptance check, can restore reliable progress toward a finite maximum.

Check Your Understanding

Use the score, information, and likelihood sequence together when answering.

For a scalar likelihood with positive observed information along the segment to its maximum, what comparison determines whether a full Newton update crosses that maximum?
Why does constant observed information make a full Newton update land at the maximum in the scalar setting described here?
In the logistic example, explain why starting at \(\beta=1\) produces alternating coefficients but increasing likelihood.
What happens to the coefficient magnitude and likelihood when the same logistic model starts at \(\beta=3\) and takes full Newton steps?
Why does a sufficiently small positive damping fraction improve the log-likelihood when the score is nonzero and the information is positive?