Firth Logistic Estimation Under Separation

Why use Firth estimation?

In a logistic propensity model, the likelihood can keep improving as one or more coefficients grow without bound. This happens under complete or quasi-complete separation: covariate values, or a linear combination of them, distinguish treated from untreated units. Ordinary maximum likelihood then does not provide finite coefficient estimates, and the corresponding fitted propensity scores can approach zero or one. A penalty is one way to regularize estimation; Firth estimation uses a particular penalty motivated by reducing the first-order bias of maximum likelihood estimates.

Write \(Z_i=1\) for treatment and \(Z_i=0\) otherwise, let \(x_i\) be the covariate vector including an intercept, and let \(\beta\) be the coefficient vector. The logistic model specifies \(p_i=P(Z_i=1\mid x_i)=\operatorname{logit}^{-1}(x_i^\top\beta)\), where \(\operatorname{logit}^{-1}(u)=1/(1+\exp(-u))\). The fitted propensity score is \(\hat e_i=\operatorname{logit}^{-1}(x_i^\top\hat\beta)\). The usual Bernoulli log-likelihood is \(\ell(\beta)=\sum_i\{Z_i\log(p_i)+(1-Z_i)\log(1-p_i)\}\).

The added term is one-half the log determinant of the information matrix, not a fixed ridge penalty on the coefficients. It is also the log of the square root of the Jeffreys-information factor, which gives the penalty its invariance motivation under smooth reparameterizations. The important practical feature here is that it changes the score equations most strongly where the data provide little information about a coefficient.

From the penalty to the adjusted score

The ordinary logistic score is \(U(\beta)=X^\top(Z-p)\), where \(p\) is the vector of fitted probabilities. Differentiating the Firth penalty gives an additional term that depends on each observation’s leverage in the weighted design. Define the weighted hat matrix as \(H=W^{1/2}X(X^\top W X)^{-1}X^\top W^{1/2}\), and let \(h_i=H_{ii}\).

To see the last equality, note that \(\partial w_i/\partial\beta_j=w_i(1-2p_i)x_{ij}\). Since \(I=X^\top W X\), its derivative is \(X^\top\operatorname{diag}\{w_i(1-2p_i)x_{ij}\}X\). Substitution into the trace expression gives one contribution per observation, with \(w_i x_i^\top I^{-1}x_i=h_i\). Thus the Firth estimating equation is the ordinary score plus this adjustment:

where \(\odot\) denotes elementwise multiplication and \(h\) is the vector of diagonal leverages. This derivation explains why the method is not equivalent to simply adding a constant to every outcome: the adjustment depends on the fitted probabilities and on the design. In regular, identifiable models, this adjustment removes the leading, order-\(1/n\) bias term of the maximum likelihood coefficient estimator. It does not guarantee that all finite-sample bias disappears.

Worked examples

When the method helps—and what it does not fix

Under separation, the ordinary likelihood has no finite maximizer, while the Jeffreys-information penalty can yield finite estimates for a full-rank logistic design with the usual finite covariate values. The separation example shows the mechanism: as a group probability tends to zero or one, its information contribution tends to zero, and the log-determinant penalty discourages that boundary. In the common logistic setting, Firth estimation is a standard bias-reducing remedy for separation.

The information matrix must still be nonsingular for the penalty and adjusted score to be defined. Redundant columns, an unidentified model, or an empty outcome class can cause problems that the penalty does not repair. Firth estimation also does not select a propensity-model specification, correct omitted covariates, or ensure that estimated scores will be appropriate for a later analysis. Finite estimates are not, by themselves, evidence of adequate covariate overlap or successful adjustment.

Ordinary maximum likelihood is suitable when it exists and is not unduly unstable; it is also the unpenalized reference point. Ridge and lasso instead penalize coefficient magnitude, with the amount of shrinkage controlled by a tuning parameter; lasso can also set coefficients to zero. Firth’s penalty is determined by the information matrix and targets first-order bias reduction, rather than imposing a chosen coefficient norm. Each approach has different goals, and none substitutes for checking the model and the observed data.

Common mistake

A common mistake is to interpret finite Firth estimates as proof that the separation problem, or the broader modeling problem, has been solved. Firth estimation addresses the existence and first-order bias of logistic coefficient estimates; it does not create information where covariate patterns are absent. A second mistake is to describe the method as simply adding 0.5 to every observation. The half-count formula applies to the saturated group examples above; in a general design the adjustment uses the observation-specific leverage \(h_i\) and fitted probability \(p_i\).