Rare Events Do Not Necessarily Mean Separation
When an outcome is uncommon, a modest sample can contain very few events. Ordinary logistic regression may then produce estimates with noticeable small-sample bias, even when its maximum-likelihood estimates are finite. It is important to distinguish this situation from separation: rarity describes how often the outcome occurs, while separation concerns whether the predictors can distinguish outcomes perfectly.
We will compare ordinary and Firth estimates in a logistic model with a binary exposure. Both exposure groups will have many non-events and a small number of events. Since each group contains both outcomes, neither group perfectly predicts the response. The example lets us calculate the estimates and risks directly, then see what the Firth correction changes.
The penalty adjusts the likelihood to reduce first-order bias. In this tutorial, its exact effect can be derived for a saturated model with two exposure groups. It is not a general recipe for adding one-half to every cell in any logistic dataset.
A Rare-Outcome Dataset With Finite Maximum-Likelihood Estimates
Suppose a study records a binary outcome for 2,000 people, half unexposed and half exposed. There are 20 events in total, so the event rate is one percent. Every exposure group has both events and non-events.
| Exposure | Events | Non-events | Total |
|---|---|---|---|
| Unexposed | 5 | 995 | 1,000 |
| Exposed | 15 | 985 | 1,000 |
Use the logistic model \(\operatorname{logit}(\pi_i)=\beta_0+\beta_1 x_i\), where \(x_i=0\) for an unexposed person and \(x_i=1\) for an exposed person. The model has one fitted probability per exposure group, so it is saturated for these two groups.
Worked Example: Ordinary Estimates From the Event Counts
For a group of \(n\) independent Bernoulli observations with \(y\) events, the log-likelihood, apart from a constant, is \(y\log(\pi)+(n-y)\log(1-\pi)\). Its derivative with respect to \(\pi\) is \(y/\pi-(n-y)/(1-\pi)\). Setting this equal to zero gives \(\hat{\pi}=y/n\), provided \(0<y<n\).
Both groups meet that condition. Thus \(\hat{\pi}_0=5/1000=0.005\) and \(\hat{\pi}_1=15/1000=0.015\). The intercept is the unexposed group’s log odds, and the slope is the difference between the exposed and unexposed log odds:
The fitted exposed risk is \(\operatorname{logit}^{-1}(\hat{\beta}_0+\hat{\beta}_1)=0.015\), as required by the observed group proportion. The estimates are finite: the log odds for each group are finite because each group contains at least one event and one non-event. This is a rare-outcome example without separation, not a case in which an ordinary estimate diverges.
Deriving the Firth Estimates for Two Groups
For a saturated two-group model, it is convenient to use the two group logits \(\theta_0\) and \(\theta_1\) as parameters. They are related to the regression coefficients by \(\theta_0=\beta_0\) and \(\theta_1=\beta_0+\beta_1\). This change of coordinates has Jacobian determinant one, so it does not change the Jeffreys penalty. Each group has its own fitted probability \(\pi_j=\operatorname{logit}^{-1}(\theta_j)\).
To establish the result, let \(q_j=1-\pi_j\). In group-logit coordinates the ordinary log-likelihood is \(\sum_j\{y_j\theta_j-n_j\log(1+e^{\theta_j})\}\). The information is diagonal, with entries \(n_j\pi_jq_j\). Consequently, each group’s contribution to the penalized log-likelihood is
The constant \(\tfrac12\log n_j\) does not affect the maximizer. Since \(d\pi_j/d\theta_j=\pi_jq_j\), differentiation gives
Equating this derivative to zero yields \(y_j+\tfrac12=(n_j+1)\pi_j\), hence the stated Firth estimate. The second derivative is \(-(n_j+1)\pi_jq_j<0\) at every finite \(\theta_j\), so the stationary point is a strict maximum. It lies strictly between zero and one, and therefore corresponds to a finite group logit. For ordinary likelihood, the score is \(y_j-n_j\pi_j\), giving \(\hat{\pi}_j=y_j/n_j\); when \(0<y_j<n_j\), this too is an interior, finite maximum. \(\square\)
Worked Example: Firth Coefficients and Risks in the Study
Apply the derived formula separately to the unexposed and exposed groups. For the unexposed group, \(y_0=5\) and \(n_0=1000\); for the exposed group, \(y_1=15\) and \(n_1=1000\).
The unexposed logit is \(\log(5.5/995.5)=-5.1985\). The exposure coefficient is the difference between the group logits:
The Firth slope is smaller than the ordinary slope, \(1.0472\) rather than \(1.1087\), and the fitted risks are \(0.54945\%\) and \(1.54845\%\), rather than \(0.5\%\) and \(1.5\%\). In this example, the correction moves both group risks slightly toward one-half. That direction is not a universal description of every Firth fit; here it follows from the specific two-group formula.
Comparing Risks and Relative Measures
Coefficients are on the log-odds scale, so comparing them alone does not show the absolute size of the predicted risks. The following calculations separate the absolute risk difference from relative measures. They describe fitted associations in this dataset; without a causal design and assumptions, they should not be read as causal effects.
Worked Example: Absolute and Relative Comparisons
Under ordinary maximum likelihood, the fitted risk difference is \(0.015-0.005=0.010\), or 1.0 percentage point. The fitted risk ratio is \(0.015/0.005=3\). The odds ratio is the exponential of the logistic slope:
Under Firth, the risks have the same denominator, \(1001\). Their difference is \((15.5-5.5)/1001=10/1001=0.009990\), or about 0.999 percentage point. Their risk ratio is \(15.5/5.5=2.81818\). Their odds ratio is \(\exp(1.0472)=2.84967\).
The two methods therefore tell a similar broad story: the exposed group has a higher fitted risk, and the absolute difference is about one percentage point. But their numerical relative comparisons differ: the Firth odds ratio is closer to one than the ordinary odds ratio. With rare outcomes, odds and risks are close in scale, but they are not identical; the odds ratio here should not be substituted for the risk ratio.
A Second Rare-Outcome Calculation
The size of the correction depends on the data, including the event counts and group sizes. A second two-group table illustrates the same calculation with fewer events. It also emphasizes that finite estimates and a useful bias correction can coexist: a finite ordinary estimate does not imply that small-sample bias is irrelevant.
Worked Example: Fewer Events, Still No Separation
Suppose each group contains 500 people, with 2 events among the unexposed and 8 among the exposed. Every cell remains positive, so the ordinary group risks and logits are finite. The ordinary risks are \(2/500=0.004\) and \(8/500=0.016\), and the ordinary odds ratio is
The Firth risks are \((2+0.5)/501=2.5/501=0.0049900\) and \((8+0.5)/501=8.5/501=0.0169661\). Its odds ratio is
The correction again reduces the estimated odds ratio in this particular table, from about 4.05 to 3.44, while both fitted risks remain small. All four observed cell counts are positive, so neither method is being used to repair an infinite ordinary estimate in this example.
Interpretation and a Common Pitfall
Firth correction is relevant here because logistic estimates can have appreciable small-sample bias when events are scarce. It produces a different fitted model by maximizing a penalized likelihood; it does not merely report a transformed version of the ordinary estimate. In the two-group saturated model, the penalty has a simple exact expression, making the comparison transparent. In a model with additional covariates, the information penalty depends on the whole design and fitted probabilities.
- Wrong reasoning: “There are few events, so ordinary logistic regression must have separation and its coefficient must be infinite.” Few events do not establish separation. In the first table, each group has both outcomes, and the ordinary coefficients are finite.
- Correction: Check whether the predictor values distinguish the outcomes in a way that causes separation. Here they do not: the ordinary group risks are \(5/1000\) and \(15/1000\), both strictly between zero and one.
- Wrong reasoning: “Firth regression always means adding one-half to every cell.” The half-count formula was derived for this saturated two-group setting. It is not a general manual adjustment for arbitrary logistic regression.
- Correction: In a general model, the Firth estimate uses \(\ell(\beta)+\tfrac12\log|I(\beta)|\). Its adjustment depends on the information matrix for the fitted model, not on independently adding half an observation to every possible category.
The practical interpretation should focus on the fitted quantities relevant to the scientific question. In the first example, the ordinary and Firth estimates both imply low risks in both groups and an absolute difference of about one percentage point. Their odds ratios differ, which matters if conclusions are phrased on the relative odds scale. Neither coefficient alone describes uncertainty; estimation and inference are distinct questions.
Check Your Understanding
Use the formulas and distinctions developed above to answer these questions.
- Why are the ordinary estimates finite in the table with 5 events among the unexposed and 15 among the exposed?
- For a group with 12 events among 800 observations, calculate its ordinary fitted risk and its Firth fitted risk under the saturated two-group result.
- In the first example, which is larger: the ordinary odds ratio or the Firth odds ratio? Give both values approximately.
- Explain why the half-count result for the two-group model should not automatically be applied to a logistic model with several covariates.
- In the first example, compare the ordinary and Firth absolute risk differences. What does the comparison say, and what does it not establish about causation?