When Logistic Maximum-Likelihood Estimates Exist

Why Existence Is a Geometric Question

In logistic regression, the fitted probability for observation \(i\) is determined by the linear predictor \(x_i^\mathsf{T}\beta\). Maximum likelihood chooses coefficients to make the observed binary outcomes plausible. Sometimes the data permit an ever-improving fit as a coefficient moves farther in one direction, without any finite coefficient vector attaining the best limiting fit. The central question here is when that does not happen.

The answer depends on the observed predictor vectors and outcomes together. A design is fully ranked if its columns are linearly independent; write \(X\) for the design matrix with row \(x_i^\mathsf{T}\), and assume \(X\) has full column rank. This condition rules out coefficient directions that leave every linear predictor unchanged. It will also ensure that a finite maximum, when it exists, is unique.

The sign convention makes the definition compact: \(z_i^\mathsf{T}b>0\) means the direction \(b\) moves the fitted linear predictor in the direction favored by observation \(i\). A zero signed margin means that observation lies on the boundary in that direction. Complete separation has no boundary observations; quasi-complete separation has at least one.

The Finite-Estimate Criterion

For a finite coefficient vector \(\beta\), put \(\eta_i=x_i^\mathsf{T}\beta\). The Bernoulli-logistic log-likelihood can be written in terms of signed margins as

This expression shows how a proposed direction affects the fit: increasing a signed margin increases that observation’s log-likelihood contribution. The precise criterion is as follows.

Proof. First suppose there is a nonzero \(b\) with \(z_i^\mathsf{T}b\geq 0\) for all \(i\). Full column rank means \(Xb\neq 0\) for every nonzero \(b\); hence at least one signed margin \(z_i^\mathsf{T}b\) is nonzero. Because all the margins are nonnegative, at least one is strictly positive. Define \(g(u)=-\log(1+\exp(-u))\). Its derivative is \(g'(u)=1/(1+\exp(u))>0\), so \(g\) is strictly increasing. For every finite \(\beta\) and every \(t>0\), each term \(g(z_i^\mathsf{T}(\beta+tb))\) is at least \(g(z_i^\mathsf{T}\beta)\), and at least one is strictly larger. Thus \(\ell(\beta+tb)>\ell(\beta)\). No finite \(\beta\) can maximize the likelihood.

Now suppose no such nonzero \(b\) exists. For every unit vector \(u\), some \(i\) must satisfy \(z_i^\mathsf{T}u<0\). Define \(m(u)=\min_i z_i^\mathsf{T}u\). This is a continuous function, since it is the minimum of finitely many continuous functions. It is strictly negative at every point of the unit sphere. The sphere is compact, so \(m\) attains its maximum there, and that maximum is still strictly negative. Write it as \(-\delta\), where \(\delta>0\). For any nonzero \(\beta\), write \(\beta=ru\), with \(r=\|\beta\|>0\) and \(\|u\|=1\). For at least one observation, \(z_i^\mathsf{T}\beta\leq-\delta r\). Since \(\log(1+\exp(a))\geq a\) when \(a\geq0\), that observation’s negative log-likelihood contribution is at least \(\delta r\). All other contributions to \(-\ell(\beta)\) are nonnegative. Therefore \(\ell(\beta)\leq-\delta\|\beta\|\), so \(\ell(\beta)\) tends to minus infinity as \(\|\beta\|\) tends to infinity.

The log-likelihood is continuous. Because it tends to minus infinity outside sufficiently large balls, its global maximum is attained on a sufficiently large closed ball, which is compact. Finally, at finite \(\beta\), let \(p_i=\operatorname{logit}^{-1}(x_i^\mathsf{T}\beta)\). The Hessian is \(-\sum_i p_i(1-p_i)x_ix_i^\mathsf{T}\). For every nonzero \(v\),

Indeed, each \(p_i(1-p_i)>0\), and full column rank ensures \(Xv\neq0\), so at least one squared term is positive. Thus the log-likelihood is strictly concave and its finite maximizer is unique. \(\square\)

Worked Designs

Reading the Criterion Correctly

The theorem is about whether an ordinary, unpenalized logistic likelihood attains its maximum at finite coefficients. It does not say that every finite estimate must be large when the design is nearly separated, nor does it quantify uncertainty or small-sample bias. Its conclusion is exact: for a full-column-rank design, any nonzero direction that makes every signed margin nonnegative rules out a finite maximum. A direction with at least one positive margin is enough; the remaining margins may be zero.

A common mistake is to check only whether every observation can be placed strictly on the correct side of a separating boundary. That checks complete separation but misses quasi-complete separation. In the third example, the shared predictor value forces two margins to equal zero, yet the third margin is positive, and the log-likelihood still increases along the direction. The correction is to check for a nonzero direction with all signed margins nonnegative and at least one strictly positive. Under full column rank, the all-zero case cannot occur for a nonzero direction.

Full column rank also matters for the uniqueness statement. If a nonzero coefficient direction leaves all fitted linear predictors unchanged, a finite maximum might still be attained, but it cannot be unique as a coefficient vector. The theorem’s rank assumption excludes that nonidentifiability and allows the separation criterion and uniqueness conclusion to be stated together.