Why Compare the Two Fits?
In sparse logistic data, a small change in the observed counts can noticeably change a fitted coefficient or odds ratio. When a group has no events or no non-events, the ordinary maximum-likelihood estimate (MLE) may not even be finite. Firth’s bias-reduced fit addresses this boundary behavior, but it is not simply the MLE with a smaller coefficient. Comparing both fits shows how the adjustment affects the estimated probabilities, their log-odds, and the model-based standard errors.
We use two groups, indexed by \(j=0,1\), with \(m_j\) independent binary responses and \(y_j\) successes in group \(j\). Write \(p_j\) for the success probability. The saturated logistic model assigns a separate logit to each group:
Thus \(\beta=\eta_1-\eta_0\) is the difference in group log-odds, and \(\exp(\beta)\) is the odds ratio comparing group 1 with group 0. The two parameterizations, \((\eta_0,\eta_1)\) and \((\alpha,\beta)\), describe the same model.
The Half-Count Adjustment in a Two-Group Model
For independent grouped-binomial observations, the likelihood, apart from factors that do not depend on the parameters, is
The ordinary MLE in a group with both successes and failures is \(\hat p_j=y_j/m_j\). Firth’s fit maximizes the Jeffreys-penalized log-likelihood \(\ell^*=\ell+\tfrac12\log|I|\), where \(I\) is the expected Fisher information matrix. In group-logit coordinates, that matrix is diagonal, with entries \(m_jp_j(1-p_j)\).
Proof. In the group-logit coordinates, the determinant of the expected information is \(m_0m_1\prod_{j=0}^1 p_j(1-p_j)\). The terms involving \(m_j\) are constant in the parameters. Therefore the penalized log-likelihood separates into a sum of two terms, one for each group:
For a fixed group, differentiating its term with respect to \(p_j\), for \(0<p_j<1\), gives
Setting this derivative to zero and multiplying by \(p_j(1-p_j)\) gives \(y_j+\tfrac12=(m_j+1)p_j\). The resulting probability is strictly between zero and one. The second derivative of the group term is
so this stationary point is the unique maximum. This argument applies at the count boundaries as well, since both adjusted counts are positive. \(\square\)
A direct consequence is that the Firth odds ratio is finite for every pair of group counts, including boundary counts:
Proof. By the theorem, the fitted odds in group \(j\) are \(\hat p_{F,j}/(1-\hat p_{F,j})=(y_j+\tfrac12)/(m_j-y_j+\tfrac12)\). The group-logit contrast is the logarithm of the ratio of these odds. Exponentiating gives the displayed expression. Since \(0\leq y_j\leq m_j\), both adjusted counts are at least \(\tfrac12\); hence neither group’s fitted odds is zero or infinite. \(\square\)
Worked Comparisons
Worked Example: Sparse Data Without Separation
Suppose group 0 has 1 success in 10 trials and group 1 has 8 successes in 10 trials. Both groups contain successes and failures, so the ordinary MLEs are finite. Compare the fitted probabilities, log-odds ratio, odds ratio, and standard errors.
Maximum likelihood. The fitted probabilities are \(\hat p_0=1/10=0.1\) and \(\hat p_1=8/10=0.8\). Their logits are \(\hat\eta_0=\log(1/9)=-2.1972\) and \(\hat\eta_1=\log(4)=1.3863\). Therefore \(\hat\beta_{\text{ML}}=1.3863-(-2.1972)=3.5835\), and the odds ratio is \(\exp(\hat\beta_{\text{ML}})=36\).
Firth fit. The adjusted probabilities are \(\hat p_{F,0}=1.5/11=3/22=0.1364\) and \(\hat p_{F,1}=8.5/11=17/22=0.7727\). Their fitted odds are \(3/19\) and \(17/5\), respectively. Thus \(\exp(\hat\beta_F)=(17/5)/(3/19)=323/15=21.5333\), and \(\hat\beta_F=\log(323/15)=3.0696\). In this example the Firth contrast is smaller, though the method does not guarantee that every coefficient moves toward zero.
Model-based standard errors. For this comparison, use the usual plug-in standard error from the inverse expected information evaluated at each fit. In group-logit coordinates, the information weight is \(w_j=m_jp_j(1-p_j)\). Since \(\beta=\eta_1-\eta_0\), the variance estimate for this contrast is \(1/w_0+1/w_1\). At the MLE, \(w_0=0.9\) and \(w_1=1.6\), so \(\operatorname{SE}(\hat\beta_{\text{ML}})=\sqrt{1/0.9+1/1.6}=1.318\). At the Firth fit, \(w_0=570/484\) and \(w_1=850/484\), giving \(\operatorname{SE}(\hat\beta_F)=\sqrt{484/570+484/850}=1.191\).
These are model-based plug-in standard errors, not a claim that the two procedures have identical inferential interpretations. They make the local information comparison explicit: both estimates are finite, while the fitted probabilities and estimated contrast differ.
Worked Example: Complete Separation
Now suppose group 0 has 0 successes in 8 trials and group 1 has 8 successes in 8 trials. The predictor perfectly separates the outcomes: every observation in group 0 is a failure and every observation in group 1 is a success.
Maximum likelihood. The group probabilities are driven toward 0 and 1, respectively. Their logits therefore tend to \(-\infty\) and \(+\infty\), so the coefficient contrast tends to \(+\infty\). There is no finite ordinary MLE for \(\beta\), and a finite MLE-based standard error cannot be reported.
Firth fit. The adjusted probabilities are \(\hat p_{F,0}=0.5/9=1/18\) and \(\hat p_{F,1}=8.5/9=17/18\). The corresponding odds are \(1/17\) and \(17\), so \(\hat\beta_F=\log(17/(1/17))=2\log(17)=5.6664\), with odds ratio \(289\). The fitted probabilities remain inside the unit interval, despite the boundary counts.
For the plug-in standard error, both information weights equal \(8(17/324)=34/81\). Hence \(\operatorname{SE}(\hat\beta_F)=\sqrt{81/34+81/34}=\sqrt{81/17}=2.183\). This finite curvature-based quantity accompanies a finite estimate; it does not turn the nonexistent ordinary MLE into a finite one.
Worked Example: Sensitivity to One Outcome in a Sparse Separated Pattern
Take 5 trials in each group, with 0 successes in group 0. First let group 1 have 3 successes; then change one group 1 failure to a success, giving 4 successes. In both datasets, group 0 has only failures while group 1 contains both outcomes, so the ordinary estimate has a separated boundary pattern and no finite coefficient contrast.
With 3 successes in group 1, the Firth probabilities are \(\hat p_{F,0}=0.5/6=1/12\) and \(\hat p_{F,1}=3.5/6=7/12\). The odds are \(1/11\) and \(7/5\). Therefore the odds ratio is \(77/5=15.4\), and the coefficient is \(\log(77/5)=2.734\).
After changing one outcome, the group 0 estimate remains \(1/12\), but the group 1 estimate becomes \(4.5/6=3/4\). Its odds are 3, so the odds ratio is \(3/(1/11)=33\), and the coefficient is \(\log(33)=3.497\). One additional success has more than doubled the estimated odds ratio, from 15.4 to 33. The finite Firth estimates make the change measurable, not negligible.
The plug-in standard errors also change. In the first dataset the weights are \(55/144\) and \(175/144\), giving \(\operatorname{SE}(\hat\beta_F)=\sqrt{144/55+144/175}=1.855\). In the second dataset the weights are \(55/144\) and \(15/16\), giving \(\operatorname{SE}(\hat\beta_F)=\sqrt{144/55+16/15}=1.920\). The adjustment prevents divergence, but sparse data can still produce substantial sensitivity to individual outcomes.
Reading the Comparison Carefully
In the two-group setting, the comparison can be organized around the quantities each fit produces:
| Quantity | Ordinary maximum likelihood | Firth bias reduction |
|---|---|---|
| Group probability with an interior count | \(y_j/m_j\) | \((y_j+\tfrac12)/(m_j+1)\) |
| Probability at a boundary count | 0 or 1; may imply no finite logit | Strictly between 0 and 1 |
| Odds ratio | May be infinite or undefined as a finite estimate | Finite and positive in the two-group model |
| Standard error shown here | Inverse-information plug-in at the MLE, when finite | Inverse-information plug-in at the Firth fit |
The half-count formula is exact for this saturated two-group model; it should not be applied as an independent correction to every coefficient in an arbitrary logistic regression. With additional covariates or a different design, the Firth fit is determined by the model’s penalized likelihood and information structure. Likewise, standard-error conventions can differ: the calculations above deliberately use the inverse expected information evaluated at the fitted probabilities so the comparison is defined consistently.
A common mistake is to say that Firth “shrinks every coefficient toward zero,” then interpret any smaller coefficient as a universal property. The worked examples show why that reasoning is too broad. The half-count adjustment changes group probabilities; the resulting logit contrast depends on both groups and their counts. It can reduce a contrast, but its direction and magnitude must be calculated for the particular data. Another mistake is to treat a finite estimate as evidence that the data are no longer sparse or sensitive. The one-outcome comparison shows that finiteness and stability are different properties.
The practical value of the comparison is diagnostic as well as numerical. A finite Firth estimate lets analysts describe effects when the ordinary likelihood has a boundary problem, while side-by-side probabilities and odds ratios show how much that adjustment matters. When patterns are sparse, report the data structure and interpret the estimated effect with attention to sensitivity; do not present the finite number alone as proof of precise information.
Check Your Understanding
Use the two-group formulas and distinguish finite estimation from stability.
- For group counts \(y_0=2,m_0=12\) and \(y_1=9,m_1=12\), calculate both Firth fitted probabilities.
- In the complete-separation example, why is there no finite ordinary coefficient, even though each group has a well-defined observed proportion?
- For the counts in the first worked example, verify the Firth odds ratio using the adjusted group odds.
- Explain why the half-count formula should not automatically be applied separately to coefficients in a logistic model with several covariates.
- What does the one-outcome sensitivity example show about the difference between finite estimates and stable estimates?