Why a Likelihood May Have No Finite Maximum
Maximum-likelihood algorithms are designed to find parameter values that maximize a likelihood, but a numerical optimizer can return a finite-looking answer even when the likelihood has no maximum at any finite parameter value. Separation is a specific way this can happen: covariates distinguish the observed outcome groups so completely that making a coefficient larger continues to improve the likelihood. The likelihood approaches a limiting value, but reaching that value would require a coefficient to diverge.
The distinction matters for interpretation and inference. A very large coefficient is not necessarily evidence of an enormous, precisely estimated effect. It may be a warning that the finite maximum-likelihood estimate does not exist. In that case, standard errors and tests derived under the assumption of a finite, regular maximum need not be trustworthy.
Model and Definition of Separation
For a precise definition, consider independent binary observations \(y_i\in\{0,1\}\), with fixed covariate vectors \(x_i\) and a logistic response model. Write \(\beta\) for the coefficient vector, \(\eta_i=x_i'\beta\) for the linear index, and \(\Lambda(u)=1/(1+e^{-u})\) for the logistic function. The model assigns probability \(\Lambda(\eta_i)\) to \(y_i=1\) and probability \(1-\Lambda(\eta_i)\) to \(y_i=0\). For finite \(\eta_i\), both probabilities lie strictly between zero and one.
The likelihood above is the general Bernoulli product for this model. The logistic function also satisfies \(\Lambda(-u)=1-\Lambda(u)\). Consequently, and specifically because of that symmetry, the contribution from observation \(i\) can equivalently be written as \(\Lambda((2y_i-1)x_i'\beta)\). Define the signed covariate vector \(z_i=(2y_i-1)x_i\). A direction \(b\) separates the data when its signed indices \(z_i'b\) have the required signs.
The covariate vector generally includes an intercept, so a separating direction can involve a change in the intercept as well as changes in slopes. Separation is a property of the observed covariates and outcomes together, not a statement that the population relationship is deterministic. The definition concerns whether a direction exists that moves the sample's signed indices toward the outcome-correct side.
Why the Finite Maximum Fails
For any observation, the logistic likelihood contribution is \(\Lambda(z_i'\beta)\). Along the line \(\beta(t)=\beta_0+tb\), this contribution becomes \(\Lambda(z_i'\beta_0+t z_i'b)\). The logistic function is strictly increasing. Thus, if every \(z_i'b\) is nonnegative, no individual contribution decreases as \(t\) increases; contributions with \(z_i'b>0\) strictly increase at finite \(t\). For those observations, the contribution tends to one as \(t\) tends to infinity. Contributions with \(z_i'b=0\) stay fixed.
Under complete separation every contribution tends to one. Under quasi-complete separation, the strictly signed observations' contributions tend to one while the tied observations' contributions remain fixed. In either case, for every finite starting vector, moving a positive distance along the separating direction strictly increases the likelihood: the strictly signed contributions increase and the tied contributions stay fixed. Thus no finite coefficient vector can be a global maximizer.
The derivative follows from \(d\log\Lambda(u)/du=1-\Lambda(u)\). When all signed projections \(z_i'b\) are nonnegative, every term in the derivative is nonnegative, and any term with a strictly positive projection is positive at finite \(t\). This verifies directly that the log likelihood increases along a separating direction. This argument explains nonattainment; it does not say every numerical path or every optimizer output must display a perfectly smooth increase.
Worked Example: Complete Separation and a Likelihood Supremum
Suppose four hypothetical observations have a single covariate \(x\), with an intercept included in the model:
| Observation | \(x_i\) | \(y_i\) |
|---|---|---|
| 1 | -2 | 0 |
| 2 | -1 | 0 |
| 3 | 1 | 1 |
| 4 | 2 | 1 |
Set the intercept to zero and the slope to \(t\). The linear indices are \(-2t,-t,t,2t\). The two zero outcomes have probabilities \(1-\Lambda(-2t)=\Lambda(2t)\) and \(1-\Lambda(-t)=\Lambda(t)\); the two one outcomes have probabilities \(\Lambda(t)\) and \(\Lambda(2t)\). Therefore
For \(t=0\), \(L(0,0)=(0.5)^4=0.06250\). For \(t=2\), \(\Lambda(2)\approx0.8808\) and \(\Lambda(4)\approx0.9820\), so \(L(0,2)\approx(0.8808)^2(0.9820)^2\approx0.7481\). As \(t\to\infty\), both logistic probabilities tend to one and \(L(0,t)\to1\). Yet at every finite \(t\), the four factors are less than one. The vector \(b=(0,1)'\) gives \(z_i'b>0\) for all four observations, so this is complete separation and the likelihood supremum of one is not attained at finite coefficients.
Worked Example: Quasi-Complete Separation With Tied Covariate Values
Consider four hypothetical observations with covariate \(x\) and an intercept:
| Observation | \(x_i\) | \(y_i\) |
|---|---|---|
| 1 | -1 | 0 |
| 2 | 0 | 0 |
| 3 | 0 | 1 |
| 4 | 1 | 1 |
Again set the intercept to zero and slope to \(t\). The observations at \(x=-1\) and \(x=1\) are correctly moved toward probabilities zero and one, respectively, as \(t\) increases. The two observations at \(x=0\) each have probability \(0.5\) for their observed outcome, regardless of \(t\). The likelihood is
At \(t=0\), this is \(0.25(0.5)^2=0.06250\). At \(t=2\), it is \(0.25(0.8808)^2\approx0.1940\), and as \(t\to\infty\), it tends to \(0.25\). To verify this is the global supremum, the two tied observations have the same covariates but opposite outcomes. If their modeled probability of outcome one is \(p\), their joint contribution is \(p(1-p)\leq0.25\), because \((p-0.5)^2\geq0\). The other two contributions cannot exceed one. Thus the likelihood is at most \(0.25\), and the limit attains that supremum only as the slope diverges. The slope direction gives zero signed projection for the tied observations and positive projections for the other two: quasi-complete separation.
What Optimizer Output Can Reveal
A numerical algorithm must stop after finitely many iterations and within finite numerical bounds. It may therefore report a large coefficient as though it were an estimate, even while the likelihood is still improving along a separating direction. Convergence messages need interpretation: a small change in the objective, a small step, or a boundary imposed by the software does not establish that a finite maximum exists. Compare starting values, parameter bounds, the score, and changes in the objective as the coefficient grows.
Near separation, fitted probabilities for some observations can become extremely close to zero or one. The score in the separating direction may remain positive but become numerically small, while curvature in that direction becomes weak. As a result, estimates and standard errors can depend strongly on stopping rules, scaling, or bounds. A large standard error is not a guarantee that conventional inference is safe: the regular finite-estimate approximation may already have failed.
Worked Example: A Finite Stopping Point Is Not a Finite MLE
Return to the completely separated data in the first example. Along the intercept-zero path, the log likelihood is \(\ell(t)=2\log\Lambda(t)+2\log\Lambda(2t)\). At \(t=2\), the slope derivative is
It is still positive, so increasing the slope improves the likelihood. At \(t=4\), the derivative is \(2\{1-\Lambda(4)\}+4\{1-\Lambda(8)\}\approx0.03731\), also positive. The observed information for this slope direction is \(-\ell''(t)=2\Lambda(t)\{1-\Lambda(t)\}+8\Lambda(2t)\{1-\Lambda(2t)\}\). At \(t=4\), this is approximately \(0.03801\), giving the one-direction curvature-based standard error \(1/\sqrt{0.03801}\approx5.129\). Symmetry of this dataset makes the intercept-slope information cross term zero at intercept zero, but \(t=4\) is still not a maximum: its slope score is nonzero. A program stopping there has not found a finite MLE. The computed curvature and standard error should not be treated as ordinary inference at an MLE.
Consequences for Inference
The usual large-sample likelihood theory for coefficient estimates presumes, among other regularity conditions, a finite parameter value at which the likelihood is maximized and suitable local curvature. Under separation, the coefficient estimate may instead diverge; the score does not vanish at any finite maximum, and fitted probabilities can approach the boundary values zero and one. The usual score expansion and asymptotic normal approximation for a finite MLE therefore do not apply in their standard form.
In particular, a Wald interval formed from a large coefficient and a software-reported standard error can be misleading, and routine Wald tests should not be interpreted as if a regular finite estimate had been obtained. Likelihood-ratio procedures that rely on regular finite-parameter asymptotics also require caution. The proper conclusion is not that the coefficient has been estimated precisely at an extreme value; rather, the sample does not supply a finite unpenalized maximum-likelihood estimate under the specified model.
When separation is detected, report it and distinguish the finite numerical output from a genuine finite MLE. Depending on the analysis, a penalized likelihood, a Bayesian model with an explicit prior, or a different data/model strategy may be considered. These approaches change or supplement the estimation problem and must be described as such; they do not turn the original unpenalized likelihood into one with a finite maximum.
Common Mistakes and a Practical Check
- Mistake: treating a large returned coefficient as an estimate of a very large finite effect. A coefficient that grows when iteration limits or bounds are relaxed can indicate that the likelihood keeps improving toward a limit. Check separation and the objective path.
- Mistake: assuming a convergence message proves a finite maximum exists. An algorithm can stop because of tolerances or finite bounds. Inspect the score, whether the coefficient is at a bound, and whether further movement improves the likelihood.
- Mistake: interpreting a huge standard error as a complete remedy. A large standard error signals uncertainty, but it does not restore the regularity conditions behind standard Wald inference when there is no finite MLE.
- Mistake: using a signed-index likelihood formula for any response link. The identity \(\Lambda(-u)=1-\Lambda(u)\) justifies that shorthand for the logistic link. For a general response function \(G\), retain the Bernoulli factors \(G(x_i'\beta)\) and \(1-G(x_i'\beta)\) unless the required symmetry has been established.
A useful diagnostic is to ask whether some coefficient direction moves every observed outcome's fitted probability toward that outcome, without moving any other observation in the wrong direction. If so, examine the likelihood along that direction and check whether the maximizing sequence requires coefficients to diverge. Under the usual full-rank design condition, the absence of complete and quasi-complete separation is the standard condition supporting existence of a finite logistic MLE; rank deficiency can create a separate identification problem and should not be confused with separation.
Check Your Understanding
Use the definitions and calculations above to answer the following.
- What distinguishes complete separation from quasi-complete separation in terms of the signed projections \(z_i'b\)?
- Why can the likelihood approach its supremum under complete separation without attaining that value for any finite coefficient vector?
- In the quasi-separation example, why can the likelihood never exceed \(0.25\)?
- A numerical optimizer returns a large slope at its coefficient bound, with some fitted probabilities nearly zero or one. What additional checks would help determine whether the output is a finite MLE?
- Why does a large standard error not by itself justify ordinary Wald inference under separation?