Diagnosing Flat Likelihoods and Tiny Updates

When a Small Update Is Not Enough

An optimizer may stop because its parameter change is tiny or because consecutive log-likelihood values barely differ. Neither observation, by itself, establishes that the fit is trustworthy. A flat likelihood can make substantial parameter movement produce almost no change in the objective; conversely, a small score can coexist with weak curvature and a large Newton step. The useful diagnosis comes from considering the score, curvature, parameter change, and likelihood increment together.

Write the parameter vector as $\beta$, the log-likelihood as $\ell(\beta)$, and the score as $U(\beta)=\nabla\ell(\beta)$. The observed information is $J(\beta)=-\nabla^2\ell(\beta)$. When $J(\beta)$ is nonsingular, the full Newton step for maximizing the log-likelihood is $d=J(\beta)^{-1}U(\beta)$. An algorithm may instead accept a fraction of that step, such as $\alpha d$, or use a different update.

Definition: A likelihood is locally flat in a parameter direction when its curvature in that direction is small in magnitude. If $J(\beta)$ is symmetric and positive definite, its eigenvalues describe the local curvature: a small eigenvalue corresponds to a direction in which the log-likelihood changes relatively little for a given parameter displacement.

Flatness is relative to the parameterization and its units. A coefficient measured in very small predictor units can have tiny score and information values even when changes in its linear predictor are consequential. Thus, an unscaled score norm or coefficient change should not be interpreted in isolation.

What the Newton Step and Curvature Reveal

Suppose $\ell$ is twice continuously differentiable near the current value $\beta$, and $J(\beta)$ is symmetric positive definite. Let $d=J(\beta)^{-1}U(\beta)$. The local quadratic approximation to the log-likelihood predicts a gain of $\tfrac12 U(\beta)^{\mathsf T}J(\beta)^{-1}U(\beta)$ from the full Newton step. This predicted gain is sometimes called half the squared Newton decrement.

Result: Under the conditions above, for a sufficiently small displacement $d$, Taylor expansion gives $$ \ell(\beta+d)-\ell(\beta) =U(\beta)^{\mathsf T}d-\frac{1}{2}d^{\mathsf T}J(\beta)d+o(\|d\|^2). $$ For the Newton direction $d=J(\beta)^{-1}U(\beta)$, the quadratic part of this change is $$ \frac{1}{2}U(\beta)^{\mathsf T}J(\beta)^{-1}U(\beta). $$

To derive the expansion, apply the second-order Taylor formula to $\ell$ at $\beta$. Its linear term is $U(\beta)^{\mathsf T}d$, and its quadratic term is $\tfrac12d^{\mathsf T}\nabla^2\ell(\beta)d=-\tfrac12d^{\mathsf T}J(\beta)d$. Continuity of the Hessian makes the remaining term $o(\|d\|^2)$ as $d$ tends to zero. Substituting $d=J^{-1}U$ gives $U^{\mathsf T}d=d^{\mathsf T}Jd=U^{\mathsf T}J^{-1}U$, so the quadratic part is half that quantity. This is a local prediction, not a guarantee that a finite step has that exact gain. In particular, if curvature is very small, a small remainder relative to $\|d\|^2$ need not be small relative to the predicted gain. $\square$

The eigendecomposition makes the role of curvature explicit. Let $\lambda_1,\ldots,\lambda_p$ be the positive eigenvalues of $J(\beta)$, with corresponding orthonormal eigenvectors $v_1,\ldots,v_p$. Write $u_i=v_i^{\mathsf T}U(\beta)$. The Newton step has components $u_i/\lambda_i$ in these directions. Consequently, a small score component can still produce a substantial step if its curvature eigenvalue is smaller still.

Result: If $J(\beta)$ is symmetric positive definite, then $$ \frac{\|U(\beta)\|}{\lambda_{\max}(J(\beta))} \leq \|J(\beta)^{-1}U(\beta)\| \leq \frac{\|U(\beta)\|}{\lambda_{\min}(J(\beta))}. $$

For the proof, expand $U=\sum_i u_i v_i$. The squared step norm is $\sum_i u_i^2/\lambda_i^2$, while the squared score norm is $\sum_i u_i^2$. Since $1/\lambda_{\max}^2\leq 1/\lambda_i^2\leq 1/\lambda_{\min}^2$, multiplying by $u_i^2$, summing, and taking square roots proves both bounds. $\square$

These inequalities are informative only relative to the chosen coordinates and norm. A tiny minimum eigenvalue can make the upper bound large, and a condition number $\lambda_{\max}/\lambda_{\min}$ far above one indicates that curvature differs greatly across directions. If the information matrix is singular, the ordinary Newton step is not defined; a generalized inverse or a modified method requires separate interpretation.

Worked Examples of Flatness and Apparent Convergence

Worked Example: A Weakly Scaled Logistic Coefficient

Consider a binomial logistic model with $n=100$ trials, all having predictor $x=10^{-6}$. Let $y=51$ successes and current coefficient $\beta=10{,}000$. The linear predictor is $\eta=x\beta=0.01$, so

p=\frac{1}{1+e^{-0.01}}\approx0.5024999792.

The score and observed information for this coefficient are

U(\beta)=x(y-np) =10^{-6}(51-50.24999792) \approx7.5000208\times10^{-7},

and

J(\beta)=nx^2p(1-p) =100(10^{-6})^2p(1-p) \approx2.4999375\times10^{-11}.

The information is tiny because the predictor changes very little when this coefficient changes by one unit. But the Newton step is not tiny:

d=\frac{U(\beta)}{J(\beta)} \approx\frac{7.5000208\times10^{-7}}{2.4999375\times10^{-11}} \approx30{,}000.8.

A small raw score here does not show that the coefficient is near a well-curved optimum. The coefficient scale is weakly connected to the linear predictor: the Newton step is large, while the predictor changes by $xd\approx0.0300008$. The quadratic approximation predicts a gain of approximately $\tfrac12U^2/J\approx0.01125$. Checking the step and curvature alongside the score prevents the misleading conclusion that the tiny score alone signals convergence.

Worked Example: A Tiny Update in a Weakly Informed Direction

Take a Gaussian GLM with identity link and known variance $1$, two observations, design matrix $X=\operatorname{diag}(1,10^{-4})$, response vector $y=(0,10^{-8})^{\mathsf T}$, and current coefficient vector $\beta=(0,0)^{\mathsf T}$. Ignoring a constant, its log-likelihood is $-\tfrac12\|y-X\beta\|^2$. Therefore $U=X^{\mathsf T}(y-X\beta)$ and $J=X^{\mathsf T}X$ at the current value, giving

U=(0,10^{-12})^{\mathsf T},\qquad J= \begin{pmatrix} 1&0\\ 0&10^{-8} \end{pmatrix}.

The Newton step and curvature eigenvalues are

d=J^{-1}U=(0,10^{-4})^{\mathsf T}, \qquad \lambda_{\min}=10^{-8},\quad \lambda_{\max}=1.

The coefficient change $10^{-4}$ and the objective improvement are both small. Indeed, the exact maximum is $\hat\beta=(0,10^{-4})^{\mathsf T}$, and moving there reduces the squared residual by $(10^{-8})^2=10^{-16}$. The log-likelihood increase is therefore $\tfrac12(10^{-16})=5\times10^{-17}$. Yet the second direction is very weakly informed: its information is $10^{-8}$, and its likelihood curvature is correspondingly flat. In this model, the standard error from the inverse information in that direction is $1/\sqrt{10^{-8}}=10^4$. A tiny update and tiny likelihood increment coexist with substantial uncertainty; they do not, by themselves, establish that the parameter is precisely determined.

Worked Example: A Tiny Score Near a Logistic Boundary

Suppose 20 Bernoulli observations are all successes, and fit an intercept-only logistic model at $\beta=20$. With $p=(1+e^{-20})^{-1}$, write $q=1-p\approx2.0611536\times10^{-9}$. The score and observed information are

U(\beta)=20(1-p)=20q\approx4.1223072\times10^{-8}, \qquad J(\beta)=20p(1-p)\approx4.1223072\times10^{-8}.

Both are numerically small, but their ratio gives a Newton step of $U/J=1/p\approx1.000000002$, not a tiny change. The log-likelihood, apart from a constant, is $20\log p$. A step from $\beta=20$ to $\beta=21$ increases it by

20\{\log(1+e^{-20})-\log(1+e^{-21})\} \approx2.6058\times10^{-8}.

That increment is small in absolute terms, but the score is not zero and the likelihood continues to increase as the intercept grows. There is no finite maximizing intercept in this all-success sample. This example shows why small objective changes can be especially deceptive near a boundary: check the score and direction of travel as well as the reported increment.

A Joint Diagnostic, Not a Single Threshold

For an accepted update $s$, record the score at the current parameter, the eigenvalues of the observed information, the size of $s$, and the actual increment $\ell(\beta+s)-\ell(\beta)$. If the optimizer uses a damped step $s=\alpha d$, record $\alpha$ too. A very small $\alpha$ can make the accepted change tiny even when the underlying Newton direction is not.

Diagnostic	Question it helps answer	Warning sign
Score norm	Is the log-likelihood locally close to stationary?	A small unscaled norm may reflect coefficient units.
Curvature eigenvalues	How strongly does the likelihood change in different directions?	A very small eigenvalue signals a weakly curved direction.
Parameter change	How far did the algorithm move in the chosen coordinates?	A tiny change may result from damping or scaling.
Likelihood increment	Did the accepted update improve the objective, and by how much?	A tiny increment can occur on a flat surface or near a boundary.

Whenever possible, interpret these quantities using sensible parameter scales. A scaled parameter change, a score adjusted for parameter scale, or diagnostics expressed in predictor coordinates can clarify whether the apparent smallness is a units issue. Compare the actual likelihood increment with the local quadratic prediction, while remembering that the prediction is local and can be inaccurate for a large step.

A practical reading is joint rather than mechanical. A small score together with adequate curvature, a small step, and a small increment is more consistent with local convergence than any one of those facts alone. A small score with a tiny curvature eigenvalue calls for caution. A tiny accepted step with a non-small score may indicate heavy damping, and a tiny increment near a boundary may reflect continuing movement toward an unattained limit. These are diagnostic interpretations, not universal stopping thresholds; tolerances depend on the model, parameterization, and numerical precision.

Common Mistake

Wrong reasoning: “The coefficients barely changed and the log-likelihood is unchanged to the printed digits, so the algorithm has converged.” This ignores the information matrix and the possibility that a solver accepted only a heavily damped step. It also treats printed precision as if it were a mathematical stopping criterion.

Correction: Inspect the score and curvature at the reported estimate, quantify the accepted parameter change on an appropriate scale, and compute the likelihood increment at sufficient precision. If a curvature eigenvalue is tiny, determine which combination of coefficients it corresponds to and whether the data actually identify that direction. If the score remains appreciable or the fit is moving toward a boundary, a tiny update or increment is not enough to certify convergence.

Key takeaway: Diagnose apparent convergence by reading score norms, curvature eigenvalues, parameter changes, and likelihood increments together. Tiny numerical changes can indicate a well-converged fit, weak identification, poor scaling, damping, or boundary behavior; the surrounding diagnostics distinguish these cases.

Check Your Understanding

Use the score, curvature, update, and likelihood change together in each answer.

For a positive-definite observed information matrix, why can a small score component still produce a large Newton-step component?
In the Gaussian example, what does the eigenvalue $10^{-8}$ say about the second coefficient direction?
Why is the tiny score in the all-success logistic example not evidence of a finite maximum?
What additional information would you want before interpreting a tiny accepted update from a damped Newton method?
State the quadratic approximation to the log-likelihood gain for a full Newton step, and explain why it need not equal the actual gain for a finite step.