Why Convergence Needs a Diagnostic Check
A generalized linear model (GLM) is usually estimated by an iterative algorithm. Software begins with starting values for the coefficients, updates them, and stops when a specified convergence rule is met or when it reaches a limit. A warning that the algorithm did not converge means the fitting procedure did not meet its stopping rule; it does not, by itself, identify the cause. Conversely, a message that a model converged does not establish that its assumptions or functional form are appropriate.
The practical task is to separate two questions. First, did the numerical procedure find a stable solution for the specified model? Second, even if it did, do the fitted values and residuals suggest that the model describes the data adequately? The iteration history addresses the first question; fitted values and residual patterns help address both.
What the Iterative Algorithm Is Doing
For independent observations in an exponential-family GLM, write the linear predictor as \(\eta_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}\), and let the conditional mean be \(\mu_i=g^{-1}(\eta_i)\), where \(g\) is the link function. The model also specifies a variance function \(\operatorname{Var}(Y_i\mid \mathbf{x}_i)=\phi V(\mu_i)\), where \(\phi\) is a dispersion parameter. The iterative reweighted least squares (IRLS) method updates the coefficient estimates using a local weighted least-squares approximation.
Here \(z_i\) is a working response, \(w_i\) is an iteration-specific weight, and \(\mathbf{W}\) is the diagonal matrix of those weights. This display omits a common dispersion factor, which does not affect the coefficient update when it multiplies all weights equally. The expression presumes the weighted design matrix is invertible; if predictors are redundant or the weights become numerically problematic, the update may be unstable or undefined.
The reason for the update is that, near the current estimate, the mean can be approximated by a first-order expansion: \(\mu_i(\eta_i+\Delta\eta_i)\approx \mu_i(\eta_i)+(d\mu_i/d\eta_i)\Delta\eta_i\). The observed-minus-fitted difference \(y_i-\mu_i\), divided by the local slope \(d\mu_i/d\eta_i\), therefore gives an approximate correction to the linear predictor. This produces \(z_i\). The GLM log-likelihood’s local curvature weights that correction in proportion to \((d\mu_i/d\eta_i)^2/V(\mu_i)\). Solving the resulting weighted least-squares problem gives the displayed coefficient update. Because it is a local approximation, the process must repeat; one update is not generally the final answer.
Software may stop based on relative coefficient changes, deviance changes, a score criterion, or a combination. The precise rule and tolerance vary by implementation. Therefore, inspect the iteration history and the warning text rather than assuming that one printed statistic is the universal definition of convergence.
Worked Example: Reading Small Deviance Changes Correctly
A hypothetical binomial GLM fit reports the following final iterations. The software’s stopping rule requires both a maximum relative coefficient change below \(10^{-6}\) and a relative deviance change below \(10^{-5}\). Determine whether the reported last update meets that rule.
| Iteration | Intercept estimate | Slope estimate | Deviance |
|---|---|---|---|
| 10 | 0.5008 | -0.8010 | 102.4006 |
| 11 | 0.5000 | -0.8000 | 102.4001 |
Method. Compute coefficient changes and deviance change separately, then compare each with its stated tolerance. For this check, define the maximum relative coefficient change as the largest absolute coefficient change divided by \(\max(1,|\text{old coefficient}|)\).
The intercept changes by \(|0.5000-0.5008|=0.0008\). Its relative change is \(0.0008/\max(1,0.5008)=0.0008\). The slope changes by \(|-0.8000-(-0.8010)|=0.0010\), giving relative change \(0.0010/\max(1,0.8010)=0.0010\). The maximum is therefore \(0.0010\), which exceeds \(10^{-6}\).
The absolute deviance change is \(102.4006-102.4001=0.0005\). Its relative change, using the old deviance as denominator, is \(0.0005/102.4006\approx0.00000488\), below \(10^{-5}\). Thus the deviance condition is met but the coefficient condition is not. Conclusion: the last update does not meet the specified joint stopping rule.
Check and common mistake. A small change in deviance is not enough when the stated rule also requires stable coefficients. Calling this fit converged from the deviance alone would ignore one of the two required checks.
Extreme Estimates and Fitted Values
Large coefficients or fitted means near the boundary of their possible range warrant investigation, especially when paired with a warning. They can arise from numerical difficulties such as poor scaling or near-redundant predictors, but they can also reflect a data structure for which an ordinary finite maximum-likelihood estimate does not exist. The latter is a substantive feature of the estimation problem, not something that additional iterations necessarily fix.
For example, in a logistic GLM the conditional probability is \(p_i=1/(1+\exp(-\eta_i))\). A large positive \(\eta_i\) gives a fitted probability close to 1, and a large negative \(\eta_i\) gives one close to 0. These values may be plausible, but a pattern of increasingly extreme coefficients and probabilities can signal separation or instability. Previous study of separation and sparse factor levels is useful here: check the data pattern and category counts rather than treating an extreme estimate as an ordinary effect that can be interpreted confidently.
Worked Example: Increasing Coefficients and Boundary Predictions
Consider a hypothetical logistic regression with an intercept and a binary predictor \(x\). There are three observations with \(x=0\), all with outcome \(y=0\), and three with \(x=1\), all with \(y=1\). The iteration output gives coefficient pairs \((\hat\beta_0,\hat\beta_1)=(-6,12),(-10,20),(-14,28)\). Diagnose the pattern.
Method. For each pair, calculate the linear predictor at \(x=0\) and \(x=1\), convert each to a fitted probability, and check whether the coefficients are stabilizing.
At \(x=0\), \(\eta=\hat\beta_0\); at \(x=1\), \(\eta=\hat\beta_0+\hat\beta_1\). For the first pair, these are \(-6\) and \(6\), so the fitted probabilities are \(1/(1+e^6)\approx0.0025\) and \(1/(1+e^{-6})\approx0.9975\). For the second pair, the predictors are \(-10\) and \(10\), giving probabilities about \(0.0000\) and \(1.0000\) to four decimal places. For the third pair, the predictors are \(-14\) and \(14\), again giving \(0.0000\) and \(1.0000\) to four decimal places.
The coefficient magnitudes continue to increase rather than settle. At each predictor value, all observed outcomes agree: the \(x=0\) group has only zeros and the \(x=1\) group only ones. The likelihood for the six observations is \((1-p_0)^3p_1^3\). Along these coefficient pairs \(p_1=1-p_0\), so the likelihood is \((1-p_0)^6\), which increases toward 1 as the fitted probabilities approach 0 and 1. No finite coefficient pair attains that limiting value.
Conclusion. This is a separation pattern: increasingly extreme estimates are consistent with the likelihood continuing toward its supremum without a finite maximum-likelihood estimate. More iterations alone do not create a stable, finite estimate. Check and common mistake. Do not interpret the slope of 28 as a precise, exceptionally large effect. The diverging coefficients and boundary predictions indicate that the ordinary estimate is not finite; report the estimation problem and consider an appropriate alternative method if the analysis requires estimates.
Convergence Does Not Rule Out Lack of Fit
A model can converge while leaving a systematic pattern in its residuals. Residuals compare observed responses with fitted values; their exact form depends on the response distribution and diagnostic purpose. For a Poisson GLM with independent counts and conditional variance equal to the conditional mean, the Pearson residual for observation \(i\) is \(r_{Pi}=(y_i-\hat\mu_i)/\sqrt{\hat\mu_i}\). This standardization expresses the discrepancy relative to the model’s expected variability. It does not, by itself, prove that a model is adequate or inadequate.
Worked Example: A Converged Poisson Fit with a Residual Pattern
A hypothetical Poisson GLM with a log link reports successful convergence. For five ordered predictor groups, its observed counts and fitted means are as follows. Compute the Pearson residuals and assess whether convergence settles the diagnostic question.
| Ordered group | Observed count \(y_i\) | Fitted mean \(\hat\mu_i\) |
|---|---|---|
| 1 | 2 | 2.000 |
| 2 | 4 | 3.000 |
| 3 | 7 | 4.500 |
| 4 | 12 | 6.750 |
| 5 | 20 | 10.125 |
Method. Use \(r_{Pi}=(y_i-\hat\mu_i)/\sqrt{\hat\mu_i}\) for each group. The first residual is \((2-2)/\sqrt{2}=0\). The second is \((4-3)/\sqrt{3}=1/\sqrt{3}\approx0.577\). The third is \((7-4.5)/\sqrt{4.5}=2.5/\sqrt{4.5}\approx1.179\). The fourth is \((12-6.75)/\sqrt{6.75}=5.25/\sqrt{6.75}\approx2.021\). The fifth is \((20-10.125)/\sqrt{10.125}=9.875/\sqrt{10.125}\approx3.103\).
The residuals rise across the ordered groups, with observed counts increasingly above their fitted means. That is a pattern to investigate, not a numerical convergence failure. It suggests that the fitted model may be missing structure, such as an inadequate relationship with the predictor. The residual sequence is not proof of a particular alternative model; further model assessment is needed.
Check and common mistake. The software’s convergence message answers whether its iterative stopping rule was met. It does not erase a residual pattern. Treating the successful message as evidence of adequate fit would confuse numerical stability with substantive adequacy.
A Practical Diagnostic Sequence
When a GLM reports a warning, work through the evidence in a consistent order. Record the response distribution, link, predictors, and any weighting or offset specification before diagnosing the fitting output; otherwise, an apparent numerical problem may actually reflect a coding or model-specification error.
Check the number of iterations, whether the iteration limit was reached, and which convergence criterion failed. A generic warning and a report of fitted values at numerical boundaries suggest different follow-up checks.
Look for coefficients that stabilize, oscillate, or grow in magnitude. Compare coefficient changes and deviance changes with the actual rule used by the software; do not substitute a different criterion.
Look for empty or very small categories, predictors on markedly different scales, redundant predictors, and fitted values at or near their allowed boundaries. Verify that the response and predictor coding are as intended.
After establishing that estimates are usable, examine the residual diagnostics appropriate to the outcome and question. A stable fit can still show systematic residual patterns or other signs of inadequate specification.
Distinguish a failed numerical fit from a converged but questionable model. Depending on the cause, investigate coding, scaling, or model structure, or use a suitable estimation approach. Do not present unstable estimates as settled findings.
Common Diagnostic Traps
- Equating convergence with a good model. Convergence means a software stopping rule was met; residual patterns and model assumptions still require review.
- Equating non-convergence with substantive failure. A warning can reflect scaling, redundant predictors, a stopping tolerance, or a data structure such as separation. Diagnose the evidence before deciding what it means.
- Trusting rounded output too much. A deviance printed unchanged to a few decimals may still have changed, while coefficients may not have stabilized. Use the displayed precision cautiously and inspect more detailed output when available.
- Responding only by increasing the iteration limit. More iterations can help when an algorithm stopped before a genuine finite solution was reached. They cannot guarantee a finite estimate when the likelihood has no finite maximum, and they do not correct substantive lack of fit.
Check Your Understanding
For each hypothetical fit, identify the main diagnostic conclusion. Give priority to the stated data and stopping-rule information.
- A fit requires maximum relative coefficient change below \(10^{-4}\). The largest change on the final update is \(0.0003\), while the deviance changes by less than the software’s deviance tolerance. Does the stated coefficient condition pass? (Answer: No; \(0.0003>10^{-4}\).)
- A logistic model has fitted probabilities \(0.0067\) for all observed zeros in one group and \(0.9933\) for all observed ones in another. What should be investigated before interpreting the slope? (Answer: A separation pattern and unstable or non-finite maximum-likelihood estimates.)
- A Poisson observation has \(y=9\) and fitted mean \(\hat\mu=4\), under the Poisson variance assumption. Calculate its Pearson residual. (Answer: \((9-4)/\sqrt{4}=2.500\).)
- A model reports successful convergence, but residuals plotted against an ordered predictor rise steadily from negative to positive values. Is the convergence message enough to conclude that the model fits adequately? (Answer: No; the systematic residual pattern merits investigation as possible substantive lack of fit.)