A Worked Example of Firth Correction Under Separation

A Small Dataset With Complete Separation

A concrete dataset makes the contrast between ordinary maximum likelihood and Firth correction visible in every calculation. Consider four independent binary observations, with predictor \(x\) and response \(Y\). Two observations have \(x=0\) and both have \(Y=0\); two have \(x=1\) and both have \(Y=1\). The predictor perfectly separates the outcomes: every observation in the first group is a failure, and every observation in the second group is a success.

Use the logistic model \(P(Y_i=1\mid x_i)=p_i\), with \(\operatorname{logit}(p_i)=\beta_0+\beta_1x_i\). Write \(\eta_0=\beta_0\) for the linear predictor at \(x=0\), and \(\eta_1=\beta_0+\beta_1\) for the linear predictor at \(x=1\). These two group logits are also convenient coordinates for the calculations: \(\beta_0=\eta_0\) and \(\beta_1=\eta_1-\eta_0\).

Ordinary Likelihood: The Supremum Is Not Attained

For this dataset the likelihood, expressed in terms of the group probabilities, is \(L=(1-p_0)^2p_1^2\), where \(p_j=\operatorname{logit}^{-1}(\eta_j)\). Each factor is at most one, so \(L\leq 1\). At any finite pair of logits, both probabilities lie strictly between zero and one, and therefore \(L<1\).

The path is a witness to non-attainment: it shows that likelihood values can get arbitrarily close to 1. The upper bound \(L\leq1\) shows that they cannot exceed 1. Together, these facts establish the supremum. The group formulation also makes clear why merely stopping an ordinary fitting algorithm at a large coefficient does not produce a finite maximum-likelihood estimate.

Firth’s Adjustment and the Exact Fit

For a logistic model, Firth’s method maximizes the Jeffreys-penalized log-likelihood \(\ell^*(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|\), where \(I(\beta)=X^\mathsf{T}WX\) is the expected information, \(X\) is the design matrix, and \(W\) has diagonal entries \(p_i(1-p_i)\). Here \(|I|\) denotes the determinant. In the group-logit coordinates, the information is diagonal, with entries \(2p_0(1-p_0)\) and \(2p_1(1-p_1)\). The coordinate transformation from \((\beta_0,\beta_1)\) to \((\eta_0,\eta_1)\) has determinant 1, so it leaves this determinant unchanged.

Here \(y_0=0\) and \(y_1=2\). The likelihood and the penalty separate by group, apart from the constant \(\tfrac12\log4\). Differentiating the contribution from group \(j\) with respect to \(\eta_j\), using \(dp_j/d\eta_j=w_j\), gives the adjusted score \(y_j+\tfrac12-3p_j\).

Proof. The group log-likelihood, omitting a constant that does not depend on \(\eta\), is \(y\eta-n\log(1+e^\eta)\). Its information is \(nw\), where \(w=p(1-p)\), so the group’s Jeffreys penalty is \(\tfrac12\log(nw)\). The derivative of the log-likelihood is \(y-np\). Also, \(d\log(w)/d\eta=1-2p\), since \(dp/d\eta=w\) and \(d\log(p(1-p))/dp=1/p-1/(1-p)\). The derivative of the penalty is consequently \(\tfrac12(1-2p)=\tfrac12-p\). Adding the derivatives gives \(y+\tfrac12-(n+1)p\). Setting this equal to zero gives the stated probability. Because \(0\leq y\leq n\), \(0<y+\tfrac12<n+1\), so the probability is interior. Finally, the derivative of the adjusted score with respect to \(\eta\) is \(-(n+1)p(1-p)<0\) at every finite \(\eta\). Thus the score is strictly decreasing and its zero is unique; the penalized group objective increases before that zero and decreases after it. \(\square\)

Tracing the Adjusted-Score Updates

The adjusted-score equations can also be followed through Fisher-scoring updates. In individual-observation notation, the logistic adjusted score is \(U^*(\beta)=X^\mathsf{T}\{y-p+h\mathbin{\odot}(\tfrac12-p)\}\), where \(h_i\) is the \(i\)th diagonal element of \(H=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}\), and \(\odot\) means elementwise multiplication. For this two-group design, each observation in a group has leverage \(1/2\): the two observations with the same predictor value share the one fitted group direction. Summing their adjusted residuals gives the group score \(y_j+\tfrac12-3p_j\), as derived above.

The finite solution follows from the score equations rather than from a claim that an algorithm must converge from every possible starting value. The calculations above show specific updates from the zero start and verify the exact endpoint by substitution. In this saturated two-group example, the adjusted objective separates into two strictly concave group contributions, each with its unique finite maximum at the probability given in the theorem. Together they establish the unique finite maximum for this example.

Why the Example Matters

This dataset isolates the mechanism of the correction. Ordinary likelihood rewards making the observed failures’ probabilities approach zero and the observed successes’ probabilities approach one. Under separation, it can keep increasing in that direction without attaining a finite maximum. The Jeffreys penalty changes the score by a leverage-weighted adjustment; here its group totals add one-half success to each group and increase the probability multiplier from \(n\) to \(n+1\). That adjustment prevents either fitted group probability from reaching a boundary.

The fitted values are not the observed proportions: those proportions are 0 and 1, while the Firth values are \(1/6\) and \(5/6\). This difference is the point of the example. It does not imply that every separated dataset has these particular estimates; the calculation depends on this design and these group sizes.

• Common mistake: “The optimizer returned a very large slope, so that must be the maximum-likelihood estimate.” A large finite output can instead be an algorithmic stopping point while the likelihood is still increasing.
• Correction: Check whether a finite maximizer exists. Here every finite fit has \(L<1\), while the explicit diverging path has \(L(t)\to1\), so no finite ordinary maximum-likelihood estimate exists.
• Common mistake: “Firth correction just replaces each observed group proportion by a value halfway toward one-half.” That shortcut does not explain the fitted coefficients or the role of the information penalty.
• Correction: Derive the adjusted score, solve its zero, and then transform the group logits back to the model coefficients. In this example that calculation gives \(1/6\), \(5/6\), \(-\log5\), and \(2\log5\).