Why Logistic Coefficients Diverge Under Separation

Why Separation Sends Logistic Coefficients Away

In ordinary logistic regression, maximum likelihood seeks coefficients that make the observed outcomes reasonably probable. Under separation, there is a direction in coefficient space that moves every observation toward a probability more favorable to its observed outcome, while making at least one probability strictly more favorable. Moving farther in that direction keeps improving the likelihood, but the improvement approaches a limit: the coefficients need not settle at any finite value.

The key is to inspect the likelihood observation by observation. This reveals both why a finite maximum cannot occur and why “the coefficients diverge” needs a little care: the coefficient vector becomes unbounded, but not necessarily every coordinate does.

Signed Predictors and a Separating Direction

Consider \(n\) independent Bernoulli responses \(Y_i\in\{0,1\}\) with fixed design vectors \(x_i\in\mathbb{R}^p\). The logistic model has \(P(Y_i=1)=\operatorname{expit}(x_i^\mathsf{T}\beta)\), where \(\operatorname{expit}(u)=1/(1+\exp(-u))\) and \(\beta\in\mathbb{R}^p\). The likelihood here is the likelihood of the observed response vector, conditional on the design.

The definition says that moving from \(\beta\) to \(\beta+td\), for \(t>0\), never moves an observation in the direction of a less favorable fitted probability. For a zero margin, its signed predictor stays unchanged; for a positive margin, it increases.

For either observed outcome, the Bernoulli likelihood contribution can be written as \(\operatorname{expit}(s_i^\mathsf{T}\beta)\). Thus the log-likelihood is

For \(y_i=1\), this contribution is \(P(Y_i=1)=\operatorname{expit}(x_i^\mathsf{T}\beta)\). For \(y_i=0\), it is \(P(Y_i=0)=1-\operatorname{expit}(x_i^\mathsf{T}\beta)=\operatorname{expit}(-x_i^\mathsf{T}\beta)\). The signed-vector expression covers both cases.

The Likelihood Increases Along the Direction

Proof. Write \(z_i=s_i^\mathsf{T}\beta\). Along the path, the log-likelihood is the finite sum

The derivative of \(\log\operatorname{expit}(u)\) is \(1-\operatorname{expit}(u)\). Differentiating this finite sum gives

Each summand is nonnegative. For any \(i\) with \(m_i>0\), the factor \(1-\operatorname{expit}(z_i+tm_i)\) is strictly positive at every finite \(t\). At least one such observation exists, so the derivative is strictly positive for every finite \(t\geq0\). Thus every finite move in the direction increases the log-likelihood.

If \(m_i=0\), the \(i\)th log-likelihood term is constant in \(t\). If \(m_i>0\), then \(z_i+tm_i\to\infty\), so \(\log\operatorname{expit}(z_i+tm_i)\to0\). Because there are finitely many observations, the sum has the finite limit

Finally, for any finite \(\beta\), the point \(\beta+d\) has strictly larger log-likelihood than \(\beta\). Hence \(\beta\) cannot be a global maximizer. This holds for every finite \(\beta\), so no finite maximum-likelihood estimate exists. \(\square\)

The limit formula is also informative: observations with positive margins approach a contribution of \(1\) to the likelihood, while observations with zero margins retain their original contributions. This is why the likelihood limit along one particular ray need not itself be the global supremum in a quasi-completely separated problem.

How the Global Supremum Relates to Diverging Coefficients

The strict increase proves that a finite maximum cannot exist. It does not mean every ray from every starting point reaches the global supremum. For example, in the preceding data set, start instead at \(\beta=(1,0)^\mathsf{T}\) and move in the same slope direction. The end observations again approach contribution \(1\), but the middle pair contributes \(\operatorname{expit}(1)\{1-\operatorname{expit}(1)\}\), which is less than \(1/4\). That ray has a limit below the global supremum. The intercept must also be brought to its best value for the middle pair.

Nevertheless, a separating direction can be used to construct a sequence that both approaches the global supremum and has unbounded coefficients. The next result makes this precise. Let \(M=\sup_{\beta\in\mathbb{R}^p}L(\beta)\), where \(L\) is the observed-data likelihood. Since \(0<L(\beta)\leq1\), \(M\) is finite and positive.

Proof. Suppose a maximizing sequence did not have norm tending to infinity. Then some subsequence would have bounded norm. In finite-dimensional Euclidean space, a bounded sequence has a convergent subsequence; write its limit as \(\beta^\ast\). The logistic likelihood is continuous, so along that convergent subsequence \(L(\beta)\to L(\beta^\ast)\). Since the original sequence is maximizing, this gives \(L(\beta^\ast)=M\). That would be a finite global maximum, contradicting the theorem. Thus every maximizing sequence has norm tending to infinity.

For the construction, choose \(\gamma_k\) with \(L(\gamma_k)>M-1/k\), which is possible by the definition of supremum. Since \(d\neq0\), choose \(t_k>0\) large enough that \(\|\gamma_k+t_kd\|\geq k\). The directional monotonicity already proved gives

The likelihoods therefore tend to \(M\), while the coefficient norms are at least \(k\) and hence tend to infinity. This constructs a maximizing sequence with an explicit movement in the separating direction. \(\square\)

Common Pitfall and Practical Meaning

Incorrect reasoning: “The likelihood increases as the coefficient grows, so the fitted model eventually attains its maximum at a very large but finite coefficient.” A large value is not an attained maximum. Along a separating direction, every finite step still admits a further step with a strictly higher likelihood; the limiting probability of \(1\) is approached, not reached, at a finite linear predictor.

A related overstatement is to say that every separating ray reaches the global supremum. The zero-margin observations in quasi-complete separation retain their probabilities along that ray, and those probabilities may not be optimally set at the starting point. What is guaranteed is strict improvement from any finite starting value, a finite limiting likelihood along that ray, and the absence of a finite global maximum. A maximizing sequence can be chosen to diverge, but the components that diverge depend on the design and parameterization.

This distinction matters when interpreting ordinary logistic maximum-likelihood output. An optimizer that reports a very large coefficient, a large standard error, or a convergence warning may be tracking a likelihood that continues to increase while approaching its supremum. The issue is not merely numerical instability: under the stated separation condition, the unpenalized likelihood has no finite maximizer.