Why Separation Does Not Send Firth Estimates to Infinity
Under separation, an ordinary logistic likelihood can keep increasing as a coefficient grows without bound: the fitted probabilities move closer to the observed zeroes and ones. Firth’s method changes the objective by adding a penalty based on the determinant of the information matrix. Near a separated boundary, some fitted probabilities become extreme, the information degenerates, and this penalty tends to minus infinity. The likelihood gain cannot prevent that decline.
The finiteness result is about the objective, not about an algorithm’s numerical behavior. Under a finite-sample, full-column-rank condition, the penalized objective has a finite maximizer. This does not by itself assert that the maximizer is unique.
Logistic Likelihood and the Information Penalty
Let \(Y_1,\ldots,Y_n\) be independent Bernoulli responses, with observed values \(y_i\in\{0,1\}\). Let \(x_i\in\mathbb{R}^p\) be fixed, finite design vectors, and let \(X\) be the \(n\times p\) matrix with rows \(x_i^\mathsf{T}\). In logistic regression, \(P(Y_i=1)=\pi_i\), where \(\operatorname{logit}(\pi_i)=x_i^\mathsf{T}\beta\). Write \(w_i=\pi_i(1-\pi_i)\). For every finite \(\beta\), \(0<\pi_i<1\) and \(w_i>0\).
Full column rank means that the columns of \(X\) are linearly independent. It ensures that \(I(\beta)\) is positive definite for every finite \(\beta\): for nonzero \(a\in\mathbb{R}^p\), \(a^\mathsf{T}I(\beta)a=\sum_i w_i(x_i^\mathsf{T}a)^2>0\), since \(Xa\ne0\) and all \(w_i>0\). Thus the log determinant in the definition is finite at every finite coefficient vector.
A separating direction is a nonzero vector \(d\) such that \((2y_i-1)x_i^\mathsf{T}d\geq0\) for every observation, with strict inequality for at least one. Along a suitable move in that direction, the ordinary log-likelihood can approach its supremum as coefficients diverge. The key fact here is broader: for a full-rank design, the information determinant tends to zero along any sequence of coefficients whose norm tends to infinity, whether or not that sequence follows a separating direction.
Worked Example: All Successes in an Intercept-Only Model
Take \(n\) independent successes and an intercept-only model, so every \(\pi_i=\pi\) and \(\operatorname{logit}(\pi)=\beta\). The data are completely separated: increasing \(\beta\) makes the ordinary log-likelihood \(n\log\pi\) approach zero, its supremum.
Here \(I(\beta)=n\pi(1-\pi)\), so the penalized objective, expressed in terms of \(\pi\in(0,1)\), is \(\ell^*(\pi)=n\log\pi+\frac12\log n+\frac12\log\pi+\frac12\log(1-\pi)\). Its derivative is \(\frac{n+1/2}{\pi}-\frac{1/2}{1-\pi}\). Setting it to zero gives \((n+1/2)(1-\pi)=\pi/2\), hence \(\hat\pi=(n+1/2)/(n+1)\). This is strictly between zero and one, and the corresponding coefficient is \(\hat\beta=\log\{(n+1/2)/(1/2)\}=\log(2n+1)\), which is finite.
The objective tends to minus infinity at both endpoints: as \(\pi\to0\), the log terms in \(\ell^*\) diverge downwards, and as \(\pi\to1\), the term \(\frac12\log(1-\pi)\) does. The stationary point is therefore the global maximum. For \(n=1\), it gives \(\hat\pi=3/4\) and \(\hat\beta=\log3\).
The Information Determinant Collapses at Unbounded Coefficients
The reason for finiteness is not that the log-likelihood itself becomes unfavorable: it is always at most zero and can approach zero under separation. Instead, along an unbounded coefficient sequence there is a direction in coefficient space in which the information becomes arbitrarily small. The next result makes this precise.
Proof. Set \(v_m=\beta_m/\|\beta_m\|\). From any subsequence of these unit vectors, one can extract a further subsequence converging to a unit vector \(v\), because the unit sphere in \(\mathbb{R}^p\) is compact. Work along this further subsequence. We show that \(v_m^\mathsf{T}I(\beta_m)v_m\to0\). Expand the quadratic form:
For an index with \(x_i^\mathsf{T}v\ne0\), we have \(x_i^\mathsf{T}\beta_m=\|\beta_m\|x_i^\mathsf{T}v_m\), whose absolute value tends to infinity. The logistic probability therefore tends to zero or one, so \(w_i(\beta_m)=\pi_i(1-\pi_i)\to0\). Also \((x_i^\mathsf{T}v_m)^2\) is bounded. For an index with \(x_i^\mathsf{T}v=0\), the factor \((x_i^\mathsf{T}v_m)^2\) tends to zero, while \(w_i(\beta_m)\leq1/4\). Thus every summand tends to zero; since there are finitely many observations, the whole sum tends to zero.
The smallest eigenvalue \(\lambda_{\min}\{I(\beta_m)\}\) is no larger than this quadratic form, because \(v_m\) is a unit vector. All eigenvalues are bounded above by \(C=\|X\|_{\mathrm{op}}^2/4\): indeed \(W(\beta_m)\preceq I_n/4\), so \(I(\beta_m)\preceq X^\mathsf{T}X/4\). Therefore \[ 0\leq |I(\beta_m)| \leq \lambda_{\min}\{I(\beta_m)\}C^{p-1}\longrightarrow0. \] The subsequence argument proves convergence for the original sequence as well: if the determinant failed to tend to zero, a subsequence bounded below by a positive constant would have a further subsequence for which the argument just given forces it to tend to zero, a contradiction. Finally, each Bernoulli log-likelihood contribution is at most zero, so \(\ell(\beta_m)\leq0\). Hence \(\ell^*(\beta_m)=\ell(\beta_m)+\frac12\log|I(\beta_m)|\to-\infty\). \(\square\)
Worked Example: A Separated Two-Observation Design
Let \(x_1=(1,0)^\mathsf{T}\), \(x_2=(1,1)^\mathsf{T}\), with \(y_1=0\) and \(y_2=1\). The design matrix has determinant \(1\), so it has full rank. The direction \(d=(-1,2)^\mathsf{T}\) gives \(x_1^\mathsf{T}d=-1\) and \(x_2^\mathsf{T}d=1\), strictly matching both outcomes; the data are completely separated.
Along \(\beta_t=(-t,2t)^\mathsf{T}\), the linear predictors are \(-t\) and \(t\). If \(s(t)=1/(1+e^{-t})\), the likelihood is \(\ell(\beta_t)=\log\{1-s(-t)\}+\log s(t)=2\log s(t)\to0\). Both information weights equal \(w(t)=s(t)\{1-s(t)\}\). Since the design matrix is square with determinant \(1\), \(|I(\beta_t)|=w(t)^2\). Thus \(\ell^*(\beta_t)=2\log s(t)+\log w(t)\). As \(t\to\infty\), \(s(t)\to1\), while \(w(t)\sim e^{-t}\), so the first term tends to zero and the second tends to minus infinity. The penalized objective therefore tends to minus infinity even along this direction of perfect separation.
Existence of a Finite Maximizer
The determinant result rules out escape to infinity as a way of maximizing the penalized objective. Continuity then supplies the existence argument.
Proof. For finite \(\beta\), the probabilities and weights are continuous functions of \(\beta\), and \(I(\beta)\) is positive definite. Consequently, \(\ell^*(\beta)\) is real-valued and continuous on \(\mathbb{R}^p\). Fix any finite \(\beta_0\). By the theorem, \(\ell^*(\beta)\to-\infty\) whenever \(\|\beta\|\to\infty\); hence there is a radius \(R\) such that \(\ell^*(\beta)<\ell^*(\beta_0)\) whenever \(\|\beta\|>R\). The continuous function \(\ell^*\) attains a maximum on the closed ball of radius \(R\), which is compact. That maximum is at least \(\ell^*(\beta_0)\), and values outside the ball are smaller than \(\ell^*(\beta_0)\). It is therefore a global maximum over all of \(\mathbb{R}^p\), attained at a finite coefficient vector. \(\square\)
Worked Example: Information Collapses Even When One Row Stays Informative
Consider the three full-rank design rows \(x_1=(1,0)^\mathsf{T}\), \(x_2=(1,1)^\mathsf{T}\), and \(x_3=(1,-1)^\mathsf{T}\), with outcomes \(0,1,0\). Along \(\beta_t=(0,t)^\mathsf{T}\), the linear predictors are \(0,t,-t\). The first observation remains at probability \(1/2\); the other two probabilities tend to opposite extremes.
Write \(w= s(t)\{1-s(t)\}\); the weights for the second and third rows are both \(w\), and the first weight is \(1/4\). Summing \(w_i x_i x_i^\mathsf{T}\) gives \(I(\beta_t)=\begin{pmatrix}1/4+2w&0\\0&2w\end{pmatrix}\), and therefore \(|I(\beta_t)|=2w(1/4+2w)\to0\).
The log-likelihood is \(-\log2+2\log s(t)\), which tends to \(-\log2\), not to minus infinity. But \(w\sim e^{-t}\), so the penalty \(\frac12\log\{2w(1/4+2w)\}\) tends to minus infinity. This calculation shows why a single informative row does not prevent the full information determinant from collapsing: information in the slope direction disappears.
Interpretation and a Common Mistake
The penalty is a function of the information volume across all coefficient directions. A near-boundary fit can retain information in some directions while losing it in another; the determinant records that loss. Under full column rank, an unbounded coefficient sequence always has at least one direction in which information vanishes, which is enough to force the determinant down.
Common mistake: “The Jeffreys penalty is positive, so it must push the separated coefficients farther toward infinity.” This confuses the determinant with its logarithm. Although \(|I(\beta)|\) is positive at every finite \(\beta\), it tends to zero near an unbounded boundary, and \(\log|I(\beta)|\to-\infty\). The penalty therefore lowers, rather than raises, the objective along such a sequence.
Full column rank matters. If the design is rank-deficient, \(I(\beta)\) is singular even at finite \(\beta\), so this log-determinant objective is not defined as stated. Also, the result establishes existence, not uniqueness or convergence of a particular fitting algorithm. Those are separate questions.
Check Your Understanding
Use the assumptions and argument in this tutorial to answer the following.
- Why is \(I(\beta)\) positive definite for every finite \(\beta\) when \(X\) has full column rank?
- Along an unbounded coefficient sequence, what happens to the information weight for an observation whose linear predictor diverges in absolute value?
- Why does \(\ell(\beta)\leq0\) suffice, together with \(|I(\beta)|\to0\), to show that the penalized objective tends to minus infinity?
- In the three-row example, which coefficient direction loses information as \(t\) grows, and how is that visible in the information matrix?
- Does the existence result guarantee a unique Firth estimate or convergence of every numerical fitting procedure? Explain.