Why Rare Features Need Special Attention
A binary diagnosis or procedure code may occur in only a small number of people, yet strongly predict which treatment they receive. In a propensity model, that combination can cause a problem more serious than an imprecise coefficient: the likelihood may have no finite maximum, or the estimated treatment probabilities may be so close to zero or one that they are unstable. These problems matter because propensity methods need credible estimates of treatment assignment across the covariate patterns relevant to the analysis.
The issue is not that rare features must always be removed. A rare feature may be an important confounder. The task is to diagnose what the data support, understand what a proposed remedy changes, and avoid treating a computationally finite estimate as evidence that treatment overlap is adequate.
For a feature vector \(X_i\), the propensity score is \(e(X_i)=P(A_i=1\mid X_i)\). A common model is
The model is a working specification: its coefficients and fitted probabilities depend on the included features and functional form. Ordinary logistic maximum likelihood assumes independent observations and a model for which a finite maximizer exists. Causal interpretation of an adjusted treatment comparison additionally requires the relevant causal assumptions, including conditional exchangeability given measured covariates and positivity for the target population. A well-behaved optimizer does not establish those assumptions.
How Sparse Cells Produce Separation
For a binary feature alone, the treatment-by-feature table makes the problem visible. Let \(n_{11}\) be the treated count with \(X=1\), \(n_{01}\) the untreated count with \(X=1\), \(n_{10}\) the treated count with \(X=0\), and \(n_{00}\) the untreated count with \(X=0\). If any cell is small, the fitted log odds ratio can be imprecise. If a cell is zero in a separating pattern, an ordinary logistic estimate may diverge.
Worked Example: Quantifying Sparse-Cell Uncertainty
Suppose a hypothetical small dataset has 4 treated and 8 untreated people with a binary feature, and 1 treated and 6 untreated people without it. All four cells are positive, so this table does not exhibit complete separation. The estimated treatment odds ratio comparing \(X=1\) with \(X=0\) is
The estimated log odds ratio is \(\log(3)=1.098612\). For this simple table, the usual large-sample standard error is the square root of the sum of the reciprocals of the four cell counts:
A rough 95% Wald interval for the log odds ratio is
or approximately \([-1.335,\ 3.532]\). Exponentiating the endpoints gives an odds-ratio interval of approximately \([0.263,\ 34.2]\). This is a wide interval: the estimated treatment odds are three times as high for people with the feature, but these sparse data are compatible with a much smaller or much larger association. This is an association in the treatment model, not a causal effect of the feature.
The zero-cell case has a different problem. In a model with an intercept and \(X\), suppose every person with \(X=1\) is treated. Raising the coefficient of \(X\) makes the fitted treatment probability for that group approach one. The likelihood keeps improving toward its supremum as the coefficient increases; it does not attain that supremum at any finite coefficient. For quasi-separation, some observations are perfectly classified while boundary observations prevent the same strict improvement for everyone, but a finite ordinary maximum-likelihood estimate can still fail to exist.
Worked Example: Recognizing Complete Separation
In a hypothetical cohort of 1,000 people, 8 treated and 0 untreated people have a rare feature. Among the 992 people without it, 392 are treated and 600 are untreated. With an intercept and this binary feature, the fitted treatment probability for \(X=0\) is \(392/992=0.3952\). For \(X=1\), the observed treated proportion is \(8/8=1\).
There is no untreated observation with \(X=1\). The feature therefore perfectly identifies treatment among its observed cases. If the coefficient for \(X\) increases, its fitted probability approaches one; the log-likelihood contribution from those 8 treated cases approaches \(8\log(1)=0\). No finite coefficient gives probability exactly one, so the ordinary maximum-likelihood coefficient diverges rather than settling at a finite value. A software warning, very large coefficient, or fitted probability rounded to 1.000 is consistent with this diagnosis.
This table also signals an empirical overlap problem for the feature-positive group: there are no observed untreated people with that feature. It does not prove that untreated people with the feature are impossible in the target population, but the data alone cannot support a direct treated-versus-untreated comparison within that group.
Diagnostics Before Choosing a Remedy
Start with the data rather than the optimizer’s summary. For every concerning binary feature, tabulate treated and untreated counts at each feature level. In a high-dimensional model, inspect the rare-feature columns and their treatment-specific counts, then check whether apparent separation involves one feature or a combination of predictors. Look for zero or near-zero cells, extreme coefficients, large standard errors, convergence warnings, and fitted probabilities concentrated near zero or one. These are clues to investigate, not proof that a particular feature is the sole cause.
Also distinguish two related but different concerns. Separation is a likelihood-estimation problem: a finite ordinary logistic maximum-likelihood estimate may not exist. Poor overlap is a design problem: some covariate patterns may have little or no representation in one treatment group. Penalization can produce finite coefficients under separation, but it cannot create untreated observations in a feature-positive group. Conversely, a rare feature may cause an unstable coefficient without fitted probabilities near the extremes throughout the sample.
Count treated and untreated observations separately for each level. A zero cell or a very small cell is an immediate warning.
Inspect convergence messages, coefficient magnitudes and standard errors, and the distribution of fitted probabilities. A finite output alone is not evidence of a stable estimate.
Ask whether the feature is a plausible confounder and whether nearby categories genuinely represent a shared construct before considering pooling or exclusion.
After any remedy, determine whether the relevant treatment groups still have support in the covariate patterns needed for the causal question.
Remedies and Their Trade-Offs
Aggregation combines levels or related codes into a broader feature. It may reduce sparse cells when the categories are substantively similar and the combined feature has observations in both treatment groups. The decision should be clinically or scientifically defensible, not merely a way to silence a warning. Pooling unlike categories can conceal meaningful treatment differences, and aggregation cannot guarantee good overlap.
Penalization changes the estimation criterion to discourage extreme coefficients. Firth’s bias-reduced logistic method uses a Jeffreys-information penalty and can give finite estimates under separation. Ridge and L1 penalties also shrink coefficients; L1 penalization can set some to zero. These procedures stabilize estimation, but they do not make a separated sample informative about an absent treatment group. The penalty and tuning choices can change fitted probabilities, so assess their implications for the causal analysis rather than choosing a penalty solely to improve prediction.
Feature exclusion removes the feature from the propensity model. Exclusion may be reasonable if the feature is not needed to control confounding or is redundant under a defensible causal argument. Dropping a feature only because it predicts treatment strongly or causes numerical difficulty can leave confounding uncontrolled. A treatment-predictive feature is not automatically a confounder, but sparsity alone is not evidence that it is safe to omit. The earlier material on treatment prediction versus confounder control is relevant to that distinction.
Worked Example: What Penalization Can and Cannot Fix
Use the separated hypothetical counts above: for \(X=1\), treated \(=8\), untreated \(=0\); for \(X=0\), treated \(=392\), untreated \(=600\). To illustrate Firth’s correction in this simple two-by-two setting, add \(0.5\) to each cell. The resulting finite odds ratio comparing treatment odds across \(X\) levels is
The corresponding corrected treatment proportions are \(8.5/(8.5+0.5)=0.9444\) for \(X=1\), and \(392.5/(392.5+600.5)=0.3953\) for \(X=0\). The estimates are finite, unlike the ordinary maximum-likelihood coefficient. Yet the rare-feature group still has an estimated treatment probability near one, and the original data still contain no untreated person with \(X=1\). Penalization has stabilized estimation; it has not supplied empirical treatment overlap or established that a causal comparison for this group is supported.
Worked Example: Evaluating a Possible Aggregation
Suppose two clinically related codes are being considered for a combined binary feature. In a hypothetical dataset, code A occurs in 4 treated and 0 untreated people, while code B occurs in 3 treated and 2 untreated people. Each code considered separately has a zero untreated cell for at least one level, but the combined feature has 7 treated and 2 untreated people with either code.
Pooling has removed the zero cell for the combined feature, but the untreated count is still only 2. Before using this aggregation, investigators must justify that A and B represent a sufficiently coherent construct for the causal adjustment being attempted. If they do not, the apparent numerical improvement comes at the cost of changing the feature’s meaning. Even if the aggregation is defensible, two untreated observations are limited evidence about treatment assignment for the combined group; the remaining model and causal assumptions still require scrutiny.
Comparison, Limitations, and Common Pitfall
| Approach | Potential benefit | Main caution |
|---|---|---|
| Unpenalized logistic model | Uses the ordinary likelihood when a finite estimate is supported. | May diverge under separation and be unstable with sparse cells. |
| Aggregation | Can reduce sparsity for genuinely related categories. | Can erase meaningful distinctions and does not guarantee overlap. |
| Penalization | Can yield finite, less extreme estimates. | Changes estimates; cannot create missing treatment-group observations. |
| Feature exclusion | May simplify a model when exclusion is justified by the causal question. | May leave confounding uncontrolled if the feature matters causally. |
A common mistake is to reason, “The unpenalized coefficient is infinite, so I will use a penalty; the resulting finite propensity score solves the problem.” The correction is that finite estimation and adequate overlap are separate requirements. Inspect the treatment-by-feature counts and fitted probabilities after penalization, and determine whether the intended causal comparison is supported. These diagnostics do not by themselves resolve all uncertainty in a high-dimensional model, and this tutorial does not replace broader balance assessment or a causal argument for the feature set.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In a treatment-by-feature table, all 6 people with \(X=1\) are treated and none are untreated. What problem does this suggest, and what does it imply for ordinary logistic maximum likelihood?
- Why does a finite Firth estimate not establish overlap for a feature-positive group with no observed untreated people?
- When is aggregation a defensible response to a rare feature, and what is one risk?
- Why is excluding a sparse feature solely because it predicts treatment strongly not automatically safe?
Solutions. (1) The zero cell suggests complete separation in the binary-feature logistic model; the ordinary maximum-likelihood coefficient may diverge rather than have a finite maximizer. (2) Penalization changes the estimation criterion, not the observed treatment-group counts; it cannot provide an untreated comparison that is absent from the data. (3) Aggregation is defensible when the categories share a meaningful scientific construct; it risks hiding real differences if unlike categories are pooled. (4) A strongly treatment-predictive feature could also help control confounding, so removing it without a causal justification may leave confounding uncontrolled.