Score, Step, and Likelihood Stopping Criteria

Why Stopping Is a Statistical Decision

Newton and Fisher-scoring algorithms produce a sequence of coefficient vectors, but an algorithm needs a rule for deciding when to stop. A stopping rule is not just a computational convenience: it determines what evidence we have that the fitted coefficients are close to a useful solution. Three common signals are a small score, a small coefficient update, and a small change in log-likelihood. They describe different features of the iteration, so they should not be treated as interchangeable.

Write \(\beta\in\mathbb{R}^p\) for the GLM coefficient vector and \(\ell(\beta)\) for its log-likelihood. The score is the gradient of the log-likelihood, and the observed information is the negative Hessian. At iteration \(t\), let \(\beta^{(t)}\) be the current accepted coefficient vector and define the accepted step by \(s^{(t)}=\beta^{(t+1)}-\beta^{(t)}\). The log-likelihood increment is \(\Delta\ell^{(t)}=\ell(\beta^{(t+1)})-\ell(\beta^{(t)})\).

Definition: A score-norm criterion stops when the score \(U(\beta)=\nabla\ell(\beta)\) is sufficiently small. A coefficient-step criterion stops when the accepted update \(s^{(t)}\) is sufficiently small. A log-likelihood-increment criterion stops when \(\Delta\ell^{(t)}\) is sufficiently small, usually in absolute or relative terms. These are numerical rules; each needs a stated norm, scale, and tolerance to be fully specified.

What Each Criterion Measures

A small score is evidence of approximate stationarity: the log-likelihood has little first-order change in any direction measured by the chosen norm. For an interior maximum of a differentiable likelihood, the score is zero. Thus a score criterion is directly tied to a necessary first-order condition for an interior optimum. It does not, by itself, show that a stationary point is a maximum, or that the coefficients are close to a particular solution.

A small step says that the algorithm changed the coefficients very little on its last update. It does not say why the update was small. The update could be small because the score is small, but it could also be small because the method damped the proposed update, because a line search chose a small step fraction, or because of how the algorithm is parameterized. A step criterion reports on the computation, not directly on stationarity.

A small likelihood increment says that the accepted update changed the objective little. This may occur near a maximum, but it can also occur after a tiny update or where the likelihood changes slowly. A relative criterion, such as \(\lvert\Delta\ell^{(t)}\rvert/(1+\lvert\ell(\beta^{(t)})\rvert)\), additionally depends on the size of the current log-likelihood. The same absolute increment can look more or less important under that ratio.

Implementations often combine an absolute and a relative threshold. For example, a score rule might compare \(\lVert U(\beta)\rVert\) with \(\epsilon_{\mathrm{abs}}+\epsilon_{\mathrm{rel}}S\), where \(S\) is a stated scale. A step rule might use the maximum componentwise relative change, \(\max_j |s_j|/(1+|\beta_j|)\). The norm, scale, and tolerances matter: a raw Euclidean norm can grow with the number of coefficients, and a componentwise relative step can depend on the units used for a coefficient. A reported “converged” result is most useful when the rule and its tolerances are also reported.

Worked Example: A Score Check in Logistic Regression

Suppose 7 of 10 independent Bernoulli observations are successes, and the model has only an intercept \(\beta\), with success probability \(p(\beta)=e^\beta/(1+e^\beta)\). Ignoring terms not depending on \(\beta\), the log-likelihood and score are

\ell(\beta)=7\beta-10\log(1+e^\beta), \qquad U(\beta)=7-10p(\beta).

At \(\beta=0.8\), \(p(0.8)=e^{0.8}/(1+e^{0.8})\approx 0.689974\). Therefore,

U(0.8)=7-10(0.689974)\approx 0.10026.

A score tolerance of \(0.11\), using the absolute value of the scalar score, would pass at this coefficient. The exact stationary point solves \(p(\beta)=0.7\), so \(\hat\beta=\log(7/3)\approx 0.84730\). This example shows that the score can quantify residual first-order change; the threshold alone does not give a general error bound for the coefficients. That requires additional information about the curvature of the likelihood.

Two Useful Results About the Criteria

The following results clarify why the criteria can disagree. The first states what a small score can guarantee when the log-likelihood has adequate curvature. The second explains why a small step or likelihood increment can occur even when the score is not small.

Theorem: Let \(C\subset\mathbb{R}^p\) be convex, and let \(\ell\) be twice continuously differentiable on an open set containing \(C\). Suppose \(\beta^\star,\beta\in C\), \(U(\beta^\star)=0\), and, for some \(m>0\), every \(x\in C\) and \(v\in\mathbb{R}^p\) satisfy \(v^\mathsf{T}\nabla^2\ell(x)v\leq -m\lVert v\rVert^2\). Then

\lVert\beta-\beta^\star\rVert\leq \frac{\lVert U(\beta)\rVert}{m}.

Proof. Put \(d=\beta-\beta^\star\). Convexity of \(C\) ensures that \(\beta^\star+td\in C\) for \(0\leq t\leq1\). By the fundamental theorem of calculus applied to the score along this segment,

U(\beta)-U(\beta^\star) =\int_0^1 \nabla^2\ell(\beta^\star+td)d\,dt.

Since \(U(\beta^\star)=0\), take the inner product with \(d\). The assumed curvature bound gives

U(\beta)^\mathsf{T}d =\int_0^1 d^\mathsf{T}\nabla^2\ell(\beta^\star+td)d\,dt \leq -m\lVert d\rVert^2.

Consequently \(m\lVert d\rVert^2\leq -U(\beta)^\mathsf{T}d\leq |U(\beta)^\mathsf{T}d|\leq\lVert U(\beta)\rVert\lVert d\rVert\), where the last inequality is Cauchy–Schwarz. If \(d=0\), the claimed bound holds immediately. Otherwise, divide by \(m\lVert d\rVert\) to obtain the result. \(\square\)

This bound explains an important qualification: a small score controls distance to the stationary point only when a suitable curvature bound holds along the path. Without that condition, the score norm alone does not quantify coefficient error.

Result: If \(\ell\) is continuously differentiable on the line segment from \(\beta\) to \(\beta+s\), and \(\lVert U(x)\rVert\leq M\) everywhere on that segment, then

|\ell(\beta+s)-\ell(\beta)|\leq M\lVert s\rVert.

Proof. Define \(g(t)=\ell(\beta+ts)\) for \(0\leq t\leq1\). The chain rule gives \(g'(t)=U(\beta+ts)^\mathsf{T}s\). The fundamental theorem of calculus and Cauchy–Schwarz therefore yield

|\ell(\beta+s)-\ell(\beta)| =\left|\int_0^1 U(\beta+ts)^\mathsf{T}s\,dt\right| \leq \int_0^1 \lVert U(\beta+ts)\rVert\lVert s\rVert\,dt \leq M\lVert s\rVert.

Thus a short step can produce a small likelihood increment even when the score is appreciable. The result is an upper bound on the increment, not evidence that the score is small. \(\square\)

Worked Example: A Tiny Damped Step With a Nontrivial Score

Consider one Bernoulli observation \(y=1\) in an intercept-only logistic model. Its log-likelihood is \(\ell(\beta)=\beta-\log(1+e^\beta)\), and its score is \(U(\beta)=1-p(\beta)\). At \(\beta=0\), \(p(0)=1/2\), so \(U(0)=1/2\). The observed information is \(p(0)(1-p(0))=1/4\), making the full Newton step \(U(0)/(1/4)=2\).

Suppose an algorithm accepts only a fraction \(\alpha=10^{-6}\) of that step. The accepted update is \(s=2\times10^{-6}\). A relative step rule with threshold \(10^{-5}\), using \(|s|/(1+|\beta|)\) at \(\beta=0\), would pass because \(2\times10^{-6}<10^{-5}\). Yet the new score is

U(2\times10^{-6}) =\frac{1}{1+e^{2\times10^{-6}}} \approx 0.4999995.

The step criterion passes while the score remains about \(0.5\). A small accepted update is therefore not, on its own, a reliable substitute for checking the score.

Worked Example: A Small Relative Likelihood Increment

Take 1000 Bernoulli observations, of which 501 are successes, in an intercept-only logistic model. At \(\beta=0\), the score is \(501-1000(1/2)=1\), while the log-likelihood is \(\ell(0)=-1000\log 2\approx -693.147\). Consider an accepted step \(s=0.001\). Using \(\log(1+e^x)=\log 2+x/2+\log\cosh(x/2)\), the increment is

\Delta\ell =501(0.001)-1000\left(0.001/2+\log\cosh(0.0005)\right) =0.001-1000\log\cosh(0.0005) \approx 0.000875.

The relative increment \(|\Delta\ell|/(1+|\ell(0)|)\) is approximately \(0.000875/694.147\approx1.26\times10^{-6}\). A relative-increment threshold of \(10^{-5}\) would pass, although the score at the current coefficient is 1. The likelihood criterion is responding to a small relative objective change, not directly verifying stationarity.

Using Stopping Rules Responsibly

In practice, a solver may monitor all three criteria rather than treating them as competing definitions of convergence. A reasonable report distinguishes the criterion that triggered termination from the other diagnostics. For example, “stopped because the relative step was below tolerance” is more informative than “converged” if the score has not also been checked. When an accepted step is required to increase the log-likelihood, the increment is nonnegative apart from numerical rounding; a small increase may still indicate only a small move.

A score norm is often the most direct check of first-order stationarity, but its magnitude depends on the data scale and the chosen norm. A step check is useful for detecting negligible changes in the coefficients, but can be satisfied by damping. A likelihood check directly tracks the objective, but a small increment does not determine the gradient. Any criterion can also be affected by numerical precision and the chosen absolute or relative tolerance.

For a differentiable concave log-likelihood, an interior stationary point is a global maximizer: concavity implies \(\ell(\gamma)\leq\ell(\beta)+U(\beta)^\mathsf{T}(\gamma-\beta)\), and the right-hand side equals \(\ell(\beta)\) when \(U(\beta)=0\). This standard fact makes a small score especially meaningful when concavity applies, but it does not turn a merely small score into an exact zero. The curvature result above gives one way to make a quantitative statement about approximate stationarity.

Common Mistakes

Calling every small update convergence. A line search or damping can make an accepted step tiny. Check the score as well, rather than assuming the update is small because the optimum has been reached.
Treating a small likelihood increment as proof of a maximum. The increment describes change between two iterates; it is not a first-order condition. Inspect the score and, where relevant, the curvature.
Interpreting a small score as a universal coefficient-accuracy guarantee. Such a guarantee needs curvature assumptions, as in the proved bound. A tolerance without those assumptions certifies only a small score under the selected norm.
Reporting a tolerance without its scale. State whether the check is absolute or relative, which norm is used, and how the scale is defined. A bare statement that the algorithm “met tolerance” is difficult to interpret.

Key takeaway: Score norms assess approximate stationarity, coefficient steps assess the size of an update, and likelihood increments assess objective change. Each is useful, but none should be interpreted as certifying more than it actually measures.

Check Your Understanding

Use the definitions and results above to answer these questions.

Which stopping criterion is most directly related to the first-order condition for an interior maximum, and why?
Give two reasons an algorithm might accept a very small coefficient step even when its score is not small.
What additional assumption lets a score bound control the distance to a stationary coefficient vector?
In the relative-likelihood example, why can the increment criterion pass while the current score is nonzero?
What information should accompany a reported numerical stopping tolerance?