Separating Separation From Convergence Failure

Why the distinction matters

A binary GLMM can fail to return a satisfactory fit for more than one reason. Under separation, the data support a direction in which one or more fixed-effect coefficients can keep growing while the likelihood approaches a finite limit. Under a more ordinary convergence failure, a finite optimum may exist, but the algorithm has stopped too early, taken unstable steps, or evaluated the likelihood or its derivatives inaccurately. Both situations can produce warnings, extreme coefficients, and unreliable standard errors. A warning by itself does not tell you which explanation is correct.

The distinction changes what the output means. If estimates are moving toward infinity along a separating direction, another optimizer run may only move them farther. If a finite optimum exists but the algorithm has not reached it, diagnosing the data as separated can be a serious misinterpretation. The useful evidence comes from considering coefficient trajectories, objective-function changes, gradients, and the actual outcome patterns together.

What to inspect in a GLMM fit

For observations \(i\) in clusters \(j\), write a binary random-intercept or random-slope GLMM as

Here \(x_{ij}\) and \(z_{ij}\) are the fixed- and random-effect design vectors, \(\beta\) is the fixed-effect coefficient vector, and \(D\) is the random-effect covariance matrix. The GLMM likelihood for \(\beta\) and \(D\) integrates over the random effects. This matters: a pattern that looks separating in the conditional linear predictor is important evidence, but a calculation that fixes \(b_j\) at zero is not, by itself, a calculation of the integrated GLMM likelihood.

Use output from repeated iterations or carefully controlled refits of the same likelihood and model. Record the coefficients, the log likelihood (or the objective being minimized), the gradient, and whether the algorithm has changed steps, tolerances, or integration settings. If a program minimizes negative log likelihood, the objective should decrease; if it maximizes log likelihood, it should increase. Comparisons are not meaningful if the model, data, integration approximation, or likelihood target changed between fits.

• Coefficient paths: Do estimates settle near stable finite values, or does a coefficient keep increasing in magnitude as iterations continue? Check whether the same coefficient and direction recur across sensible starting values.
• Likelihood changes: Does the log likelihood continue to improve, but by smaller and smaller amounts as a coefficient grows? Or does it stop improving while the coefficients still move substantially?
• Gradient: A small gradient is consistent with a stationary point, but does not prove that the point is a useful finite maximum. Under separation, the gradient can become small far out along a coefficient path because the fitted probabilities are approaching zero and one.
• Curvature and uncertainty: Near separation, information in the diverging direction can become very weak, making standard errors unstable or the information matrix nearly singular. This is supporting evidence, not a stand-alone test.
• Outcome patterns: Tabulate outcomes against the predictor patterns implicated by the large coefficients, retaining cluster membership where relevant. Ask whether the pattern genuinely classifies the outcomes or whether sparse cells merely make the estimate unstable.

Why a separating path can look converged

The following calculation explains why coefficient growth, a likelihood that barely changes, and a small gradient can occur together. Consider the Bernoulli log likelihood for a fixed-effects logistic model, and let \(d\) be a direction in coefficient space. For observation \(i\), define its signed margin \(m_i=(2y_i-1)x_i^{\mathsf T}d\), temporarily considering a path along which the linear predictor is \(t x_i^{\mathsf T}d\). If \(m_i\geq 0\) for every observation and \(m_i>0\) for at least one, this direction classifies at least some observations strictly and none in the wrong direction. The contribution to the log likelihood along the path is \(\log\{\operatorname{logit}^{-1}(t m_i)\}\), so

For every strictly positive margin, its likelihood contribution increases toward zero as \(t\) grows; a zero margin contributes a constant. Thus the likelihood can approach a finite supremum while \(t\) has no finite maximizing value. The positive derivative tends toward zero, and curvature in that direction can also become very small. This is why a small gradient or a tiny likelihood change cannot, on its own, distinguish a separated fit from a well-behaved finite fit.

This argument describes the Bernoulli likelihood along a fixed-effects direction. In a GLMM, the fitted objective integrates over \(b_j\), so the conditional calculation is a clue about the data and model, not a substitute for checking the integrated likelihood and the fitting procedure. The earlier material on Profile Likelihood Signs of Separation gives a complementary view of the same problem.

Worked Example: A Finite Estimate That an Optimizer Has Not Reached

Consider six hypothetical clusters, each with two observations, and an intercept-only binary model for a transparent fixed-effects check. Suppose the outcomes comprise eight 1s and four 0s. The conditional log likelihood at a common probability \(p\) is \(8\log(p)+4\log(1-p)\). Its maximum occurs at \(\hat p=8/12=2/3\), giving the finite intercept estimate \(\hat\alpha=\log\{(2/3)/(1/3)\}=\log 2\approx0.6931\).

Suppose an optimization run stops at \(\alpha=0.600\). Then \(p=\operatorname{logit}^{-1}(0.600)\approx0.6457\), and the log likelihood is

At the finite maximum, \[ \ell(\log 2)=8\log(2/3)+4\log(1/3)\approx -7.638. \] The score at \(0.600\) is \(8-12(0.6457)\approx0.2521\), not zero. The log likelihood can still improve by about \(0.0117\), and the coefficient has a nearby finite target. These values describe the fixed-effects likelihood check; a random-intercept GLMM requires its integrated likelihood and gradient to be inspected too. In that full fit, a finite, stable coefficient path and a remaining gradient or optimizer warning would favor an optimization problem over separation. The data here do not have a predictor that perfectly distinguishes the outcomes: the model has an intercept only, and both outcome values are present.

Worked Example: Cluster Patterns Are Evidence, Not a Verdict

Suppose a hypothetical random-intercept GLMM studies a binary outcome against a binary predictor \(x\). Among observations with \(x=0\), the observed outcomes include both 0 and 1. The same is true among observations with \(x=1\), and this overlap occurs in several clusters. A fit nevertheless reports a large positive coefficient for \(x\).

First, the data-level check does not show complete separation by \(x\): neither predictor level contains only one outcome class. The large estimate could still reflect sparse overlap, a cluster-specific pattern, a difficult likelihood surface, or another modeling issue; overlap in a simple table is not proof that every possible separating direction is absent. Next, inspect the coefficient trajectory. If repeated runs move the coefficient in different directions, the likelihood changes materially with starting values, and the gradient remains appreciable, numerical or optimization difficulty is plausible. If instead the same coefficient repeatedly grows in the same direction, its likelihood gains shrink, and the relevant predictor patterns approach perfect classification, separation becomes more plausible.

For a GLMM, retain the cluster labels when making these tables. A predictor may classify outcomes within some clusters but not others, and random slopes can make a single pooled table hide important structure. Conversely, a cluster with only one outcome class is not automatically proof that a fixed effect is separated: the pattern must be assessed in relation to the fixed- and random-effect design and the likelihood being fitted. The conclusion should rest on the combined evidence, not one sparse table.

A practical diagnostic sequence and common pitfalls

A common mistake is to reason, “The optimizer says convergence, so the coefficients are valid,” or the reverse, “The optimizer warns, so the data must be separated.” Neither follows. An algorithm can stop with a small gradient along a separating path, and it can fail to converge for a finite optimum because of numerical instability, poor scaling, or a difficult random-effects likelihood. The correction is to require agreement among coefficient movement, likelihood behavior, gradients, and outcome patterns.

Compared with using a convergence message alone, this combined diagnostic is more informative but still does not prove the existence or absence of separation in every complicated GLMM. A profile likelihood can add evidence about whether the likelihood levels off as a coefficient moves outward; design-based checks can identify classification patterns; and optimizer or integration checks can reveal numerical problems. Each addresses a different part of the diagnosis. Extremely large standard errors, a nearly singular information matrix, or a boundary estimate may occur alongside separation, but none uniquely identifies its cause.