Introduction
A failed GLM fit is not a diagnosis. An optimizer may stop because its likelihood stopped changing, because its score is small, because its steps became tiny, or because its curvature calculation failed. These patterns can look similar in a software message but point to different underlying problems. The useful question is not simply whether the algorithm stopped, but what the likelihood trace, score, steps, and information matrix jointly say about the fit.
This tutorial develops a decision workflow for interpreting those diagnostics. The goal is to choose a response that fits the evidence: damp an overly aggressive step, use Fisher scoring when observed curvature is troublesome, revise a starting value when the path is poor, or investigate whether the likelihood has only a boundary supremum rather than a finite maximizer.
What to Record During a Fit
Let \(\ell(\beta)\) be the log-likelihood for a GLM parameter vector \(\beta\). At iteration \(t\), record the log-likelihood \(\ell(\beta^{(t)})\), the score \(U(\beta^{(t)})=\nabla\ell(\beta^{(t)})\), the parameter displacement \(d^{(t)}=\beta^{(t+1)}-\beta^{(t)}\), and the observed information \(J(\beta^{(t)})=-\nabla^2\ell(\beta^{(t)})\). When available, also record the expected information \(I(\beta^{(t)})=E[J(\beta^{(t)})]\), with the expectation taken under the model at the current parameter value.
The likelihood trace shows whether accepted iterates are improving the objective. The score measures local first-order evidence that the current point is not stationary. The step records how far the parameter moved. The information describes local curvature. None is a sufficient diagnostic by itself: a tiny step can occur far from a stationary point, and a small score can occur while the likelihood is approaching a boundary without attaining a finite maximum.
A Decision Workflow
For a maximization algorithm, accepted steps should not decrease the log-likelihood when the procedure requires likelihood-based acceptance. Decreases or repeated backtracking call for a closer look at the proposed direction and step length.
A small score and small step are compatible with a finite stationary fit, but are not proof on their own. A small step with a substantial score suggests stagnation, poor scaling, or a badly conditioned update. A small score with persistent parameter drift can signal a boundary approach.
Check whether the observed information is positive definite and whether its eigenvalues indicate near-singularity. An indefinite observed Hessian can yield a non-ascent Newton direction. Near-singularity can produce unreliable or very large updates even when a linear system is technically solvable.
Use damping when a sensible direction is too aggressive; consider Fisher scoring when observed curvature is unstable but expected information is usable; revise the start when the path is poor; investigate boundary behavior when coefficients keep moving as the score declines.
Why Damping Can Rescue a Good Direction
A Newton direction is \(d=-H^{-1}U\), where \(H=\nabla^2\ell(\beta)\), provided \(H\) is nonsingular. A direction is an ascent direction when \(U^\mathsf{T}d>0\). This check helps separate a bad direction from a step that is merely too long.
Proof. Since \(H\) is symmetric negative definite, its inverse is also negative definite. Therefore
for every nonzero \(U\), which proves the first claim. If \(H\) is indefinite, \(H^{-1}\) is also indefinite, so the quadratic form \(U^\mathsf{T}H^{-1}U\) can have either sign. Thus the same expression for \(U^\mathsf{T}d\) can be negative, and the Newton direction need not ascend. \(\square\)
Even when \(d\) is an ascent direction, a full step can be too large. If \(\ell\) is differentiable at \(\beta\), then for \(U^\mathsf{T}d>0\),
Dividing by \(\alpha>0\) and taking the limit gives a positive limit, so the difference is positive for all sufficiently small positive \(\alpha\). This establishes why reducing the step length can help when the direction is sound. It does not guarantee that every halved step will be acceptable under a particular algorithm, nor does it repair a non-ascent direction. \(\square\)
Worked Example: Separating a Good Direction From an Oversized Step
Consider an intercept-only binomial model with 8 successes in 10 trials and a logit link. Its full log-likelihood, including the data-only binomial coefficient, is
At \(\beta=0\), \(\ell(0)=\log 45-10\log 2\approx 3.806662-6.931472=-3.124810\). The score is \(U(\beta)=8-10p\), where \(p=\exp(\beta)/(1+\exp(\beta))\), and the observed information is \(J(\beta)=10p(1-p)\). Thus \(U(0)=3\), \(J(0)=2.5\), and the Newton displacement is \(U(0)/J(0)=1.2\). The proposed point is \(\beta=1.2\), with
The likelihood increases, so this Newton step is acceptable. Now suppose a separate oversized proposal from \(\beta=0\) is \(\beta=3\). Its likelihood is
This is below \(\ell(0)\), so the full proposal should be rejected by a likelihood-based acceptance rule. Halving its displacement gives \(\beta=1.5\), for which
That value exceeds \(-3.124810\), so the halved step improves the objective. The diagnosis is not that every large displacement is wrong; rather, this particular full step was too aggressive for the likelihood, while a shorter one was acceptable. Notice that the binomial coefficient contributes \(\log 45\) to each log-likelihood value. Omitting it would preserve comparisons within this one model but would not give the full log-likelihood values reported here.
Using Score, Step, and Information Together
The score norm depends on parameter units: changing a coefficient’s scale changes its score component. Likewise, the raw step norm can be misleading when coefficients have different scales. Examine components in context and, where appropriate, use a parameter-scaled norm. If \(J\) is positive definite, the quantity \(U^\mathsf{T}J^{-1}U\) is a curvature-scaled measure of the score; it is not available as a reliable diagnostic when \(J\) is singular or indefinite. These checks complement, rather than replace, likelihood comparisons.
When observed information is positive definite but poorly conditioned, small changes in the score can correspond to large changes in the Newton update. When it is indefinite, first check whether the computed Newton direction is an ascent direction; if not, an unmodified Newton step is unsuitable. Fisher scoring replaces observed information with expected information. It may provide a more stable update when the expected information is positive definite and well-conditioned, but it is not a universal cure: its step still needs to be assessed, and damping may still be necessary.
Worked Example: Choosing Between Newton and Fisher Scoring
Suppose a diagnostic at a current parameter estimate reports score \(U=(2,-1)^\mathsf{T}\), observed Hessian \(H=\operatorname{diag}(1,-4)\), and expected information \(I=\operatorname{diag}(4,2)\). These are illustrative local diagnostic values. The observed Hessian is indefinite, so calculate the Newton displacement rather than assuming it ascends:
Its directional derivative is \(U^\mathsf{T}d_N=2(-2)+(-1)(-1/4)=-3.75\), so it is a descent direction locally. A line search that only shortens this direction cannot turn its negative first-order change into an ascent direction near the current point. Next calculate the Fisher-scoring displacement:
The expected-information direction is an ascent direction to first order. This evidence supports trying Fisher scoring, with likelihood-based step control, rather than repeatedly halving the non-ascent Newton direction. In an actual fit, one must also check whether the expected information is positive definite and numerically usable.
When a Small Score Does Not Mean Convergence
A finite maximum has a finite parameter vector at which the score vanishes, subject to the usual differentiability and interior-point conditions. But a sequence can approach the supremum of a likelihood while its coefficients diverge. In that case, the score may become small even though there is no finite maximum to which the optimizer can converge. A telltale pattern is continued coefficient movement, often paired with shrinking information in one or more directions.
Worked Example: An Intercept-Only Boundary Pattern
Suppose all five observations in an intercept-only Bernoulli model are successes. The log-likelihood under a logit link is
For every finite \(\beta\), the score is \(U(\beta)=5/(1+\exp(\beta))>0\), so there is no finite stationary point. As \(\beta\) increases, the log-likelihood increases toward \(0\), its supremum, but never attains \(0\) at a finite value. At \(\beta=8\), for example,
The score is small, but the conclusion “a finite maximum has been found” is false: the score remains positive at every finite coefficient. The observed information is \(J(\beta)=5p(1-p)\), which also tends to zero as \(\beta\) tends to infinity. A sequence of increasing intercepts and vanishing score and information therefore points to boundary investigation, not simply another starting value or a stricter stopping tolerance.
Common Diagnostic Patterns and Responses
| Observed pattern | Likely interpretation | Reasonable next action |
|---|---|---|
| Likelihood decreases after a proposed step; direction is ascent | The direction may be too long | Use step control such as damping or backtracking |
| Observed Hessian is indefinite; Newton direction has nonpositive score product | The Newton direction is not a local ascent direction | Do not rely on step-halving alone; consider Fisher scoring if its information is usable |
| Very small steps with a non-negligible score | Stagnation, scaling difficulty, or ill-conditioning | Inspect information eigenvalues and parameter scales; consider a revised start |
| Score and steps shrink, but coefficients continue toward an extreme and information collapses | Possible boundary fit | Assess whether a finite maximizer exists; do not equate a small score with finite convergence |
| Likelihood trace is unstable from the beginning and curvature diagnostics vary sharply | The starting point may lead into a poor region | Try a justified alternative start and compare traces and final diagnostics |
A revised start is most informative when it changes the path in a meaningful way: compare the resulting likelihood trace, final score, coefficient values, and information diagnostics. Multiple starts reaching comparable finite estimates with small scores provide useful reassurance, but do not prove global optimality in a nonconcave problem. Conversely, repeatedly restarting without examining the failure signature can reproduce the same problem at greater computational cost.
Common Mistake
Wrong reasoning: “The optimizer stopped and the score norm is below its tolerance, so the fit has converged to a finite maximum.” This treats an algorithmic stopping condition as a mathematical conclusion. In the all-successes example, the score can be arbitrarily small at a sufficiently large finite intercept, even though the likelihood has no finite maximizer.
Correction: Check whether the likelihood trace and coefficient sequence are compatible with a finite endpoint, whether the step and score are both small in a meaningful scale, and whether the information remains usable. When coefficients keep moving toward an extreme while information collapses, investigate boundary behavior. When a finite-looking solution has an unstable or indefinite curvature calculation, address that issue separately rather than interpreting the stopping message as proof.
Check Your Understanding
Use the workflow to identify what each diagnostic pattern supports, and what it does not establish.
- A Newton direction has \(U^\mathsf{T}d>0\), but the full proposed step lowers the log-likelihood. What adjustment is reasonable, and what does this observation say about the direction?
- Why can repeated step-halving fail to fix a Newton direction whose score product \(U^\mathsf{T}d\) is negative?
- A fit reports a tiny score norm and tiny parameter step, but the observed information has a very small eigenvalue. What additional checks would you make before accepting convergence?
- In the all-successes intercept-only example, why does a small score at a large intercept fail to establish a finite maximum?
- What evidence would support trying Fisher scoring rather than merely choosing a different starting value?