Why Rare Treatment Changes the Estimation Problem
Let \(D_i=1\) indicate that unit \(i\) received treatment and \(D_i=0\) indicate that it did not. A propensity score is the conditional treatment probability \(e(X_i)=P(D_i=1\mid X_i)\), where \(X_i\) is the vector of measured covariates. In a logistic score model, the probability is represented as
When treatment is rare, the data contain relatively few treated units. This can limit the information available for estimating the model, especially the effects of covariates that distinguish treated from untreated units. A low overall treatment prevalence does not, by itself, mean the model is invalid or that causal positivity fails: the relevant question for positivity is whether treatment probabilities are sufficiently above zero across covariate patterns of interest. But rare treatment makes it especially important to check what the observed data can support.
Likelihood and Information When Events Are Few
Under independent observations and a correctly specified logistic model, the likelihood for the treatment indicators is
The maximum-likelihood estimate (MLE) solves the score equation \(\sum_i X_i\{D_i-e(X_i)\}=0\). The observed information matrix for this model is \(\sum_i e(X_i)\{1-e(X_i)\}X_iX_i^\mathsf{T}\), evaluated at the fitted coefficients. When the model is identifiable and the usual large-sample approximation is reasonable, the estimated covariance matrix is the inverse of this information matrix.
Each unit contributes a weight \(e(X_i)\{1-e(X_i)\}\) to the information. At a covariate pattern where the treatment probability is near zero, that weight is small. Thus, a large number of untreated observations does not necessarily compensate for a shortage of treated observations when estimating how covariates relate to treatment. If some covariate patterns have no treated units, their treatment effects may be weakly identified or, in cases of separation, have no finite ordinary logistic MLE.
Worked Example: Intercept Uncertainty With Five Treated Units
Suppose a hypothetical sample has \(n=200\) units, of whom 5 receive treatment. Fit an intercept-only logistic model. This model assumes a common treatment probability for every unit; it is a useful baseline, not a claim that covariates are irrelevant in a substantive analysis.
The sample prevalence is \(\hat p_D=5/200=0.025\). The fitted intercept is
The fitted probability is \(\operatorname{expit}(\hat\beta_0)=0.025\), where \(\operatorname{expit}(z)=\exp(z)/\{1+\exp(z)\}\). The estimated information for the intercept is \(200(0.025)(0.975)=4.875\), giving
A conventional Wald 95% interval for the log-odds intercept is \(-3.6636\pm1.96(0.4529)\), or approximately \((-4.5513,-2.7759)\). Transforming both endpoints with the expit function gives an approximate interval of \((0.0104,0.0587)\) for the common treatment probability. This is a model-based interval, not a statement that there is a 95% probability the fixed probability lies within these particular endpoints. With only five treated observations, a Wald approximation can be imprecise; a likelihood-based interval or a suitable small-sample approach may be preferable.
Covariate Effects Need Covariate-Specific Information
An overall event count is a warning signal, not a complete measure of whether a model can estimate each coefficient. Information for a covariate effect depends on how treatment outcomes vary across that covariate and on the fitted probabilities. Consider a binary covariate \(X\) and a logistic model with an intercept and a coefficient for \(X\). If the fitted probability is constant within each of the two groups, the information for the slope after accounting for the intercept is
To see why, the intercept-and-slope information matrix has entries \(W_0+W_1\), \(W_1\), and \(W_1\) in its first row and second diagonal position. Removing the information shared with the intercept gives the slope information \(W_1-W_1^2/(W_0+W_1)=W_0W_1/(W_0+W_1)\). The slope variance is approximately its reciprocal. If either group contributes little information, the slope is imprecise, even if the other group is large.
Worked Example: A Rare Treatment and a Binary Covariate
In a hypothetical sample, 900 units have \(X=0\), with 9 treated, and 100 units have \(X=1\), with 10 treated. The overall treatment prevalence is \(19/1000=0.019\). Fit a logistic model with an intercept and this binary covariate. There is at least one treated and one untreated unit in each group, so the group-specific MLEs are finite.
The fitted probabilities are \(\hat p_0=9/900=0.01\) and \(\hat p_1=10/100=0.10\). The fitted intercept and slope are
The information contributions are \(W_0=900(0.01)(0.99)=8.91\) and \(W_1=100(0.10)(0.90)=9.00\). Thus the information for the slope after accounting for the intercept is
The estimated odds ratio for \(X=1\) versus \(X=0\) is \(\exp(2.3979)\approx11.00\). This is an association in the fitted treatment-assignment model, not a treatment effect on an outcome. Its relatively large estimated value does not make it precise: the standard error reflects that each group provides only about nine units of logistic information. A model with many additional terms would need further evidence that those terms can be estimated reliably.
Calibration and Fitting Choices
Calibration concerns whether predicted probabilities agree with observed treatment frequencies. In an intercept-only logistic fit with an intercept estimated by ordinary maximum likelihood, the fitted probabilities average exactly to the observed prevalence: the intercept score equation is \(\sum_i(D_i-\hat e_i)=0\). With covariates, the score equations impose corresponding model-based constraints, but do not guarantee calibration in every subgroup or exact balance of covariates. Calibration is not evidence that a score model captures all relevant treatment assignment patterns.
Worked Example: Intercept Recalibration at Low Prevalence
Suppose a hypothetical model produces the same initial prediction, \(0.04\), for each of 100 units, but the observed sample contains 3 treated units. To adjust only the intercept, add a constant \(\delta\) to every fitted logit and choose \(\delta\) so that the average recalibrated probability equals \(3/100=0.03\). Since all initial predictions are equal, the solution is
The recalibrated prediction for each unit is \(\operatorname{expit}\{\log(0.04/0.96)+\delta\}=0.03\), so the predicted total is \(100(0.03)=3\), matching the observed treated count. The adjustment corrects the overall prevalence in this example. It cannot make predictions discriminate between units, verify subgroup calibration, or establish covariate balance. For nonconstant predictions, the intercept adjustment is found by solving \(\sum_i\operatorname{expit}\{\hat\eta_i+\delta\}=\sum_iD_i\), typically with a one-dimensional numerical method.
In practice, specify a parsimonious, substantively justified set of predictors and inspect the amount of treatment information available for each important term. Avoid adding many weakly supported interactions or nonlinear terms merely because the total sample size is large. The number of treated observations is often a useful practical warning about model complexity, but no universal treated-units-per-parameter threshold guarantees reliable estimation.
Ordinary maximum likelihood is appropriate when the logistic model is estimable and the available data support its complexity. If sparse cells or a predictor that nearly perfectly separates treated from untreated units cause coefficient estimates to grow without bound, the ordinary MLE may not exist or may be extremely unstable. Firth logistic estimation, introduced in the earlier material on Firth logistic estimation under separation, can provide finite estimates in separation settings. Penalization can also stabilize estimation, but changes the fitting objective and can affect calibration; it does not create information that the data lack. In particular, a finite estimate should not be mistaken for strong evidence or for adequate overlap.
Diagnostics, Alternatives, and Limitations
Review convergence warnings, coefficient magnitudes and standard errors, fitted probabilities, and treatment counts across important covariate patterns. Large standard errors, very large coefficient estimates, fitted probabilities near zero or one, and empty or nearly empty treatment-by-covariate cells are signs that the fit may be unstable. Check calibration overall and in meaningful subgroups, while remembering that subgroup checks are themselves noisy when treatment is rare. A model can converge and have an apparently reasonable average prediction while still giving unreliable estimates in particular regions of the covariate space.
| Fitting choice | When it can help | What it does not solve |
|---|---|---|
| Parsimonious ordinary logistic MLE | The model is estimable and a small set of prespecified terms is supported | It cannot repair separation, misspecification, or a lack of information in a subgroup |
| Intercept recalibration | The overall predicted treatment prevalence needs adjustment while retaining score ordering | It does not improve discrimination or guarantee subgroup calibration and covariate balance |
| Firth or other penalized fitting | Sparse data, separation, or unstable estimates motivate a regularized fit | It does not add treated observations or guarantee good propensity-score overlap |
These choices address different problems. Recalibration adjusts the average level of predictions; a penalty stabilizes coefficient estimation; simplifying the model reduces demands on limited information. None substitutes for examining whether treatment and control units are represented across the covariate patterns needed for the intended analysis. Rare treatment can also yield very small estimated scores for many units, so downstream analyses may be sensitive to extreme weights or poor overlap. Those are substantive limitations of the available treatment assignment, not merely numerical inconveniences.
Common Mistakes
- Equating a large sample with abundant information. A large sample that contains few treated units may still provide little information about treatment-model coefficients.
- Treating low prevalence as proof of positivity failure. Low marginal prevalence and a near-zero conditional probability for a subgroup are distinct issues; examine covariate patterns.
- Assuming calibration guarantees balance. Matching the predicted and observed treatment totals does not guarantee balance in covariates or subgroup calibration.
- Interpreting a finite penalized estimate as certainty. Regularization can prevent divergent estimates, but uncertainty and sparse support still matter.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A hypothetical intercept-only model has 8 treated units among 400. What are the fitted treatment probability, intercept, and approximate standard error of the intercept?
- Why can a binary covariate’s slope be imprecise even when the untreated group is very large?
- Does an overall treatment prevalence of 0.01 by itself prove a violation of positivity? Explain.
- A fit’s predicted probabilities sum to the observed number treated. What does this establish, and what does it not establish?
- When might Firth logistic estimation be considered, and what limitation remains even after fitting it?
Solutions. (1) The fitted probability is \(8/400=0.02\), and the intercept is \(\log(0.02/0.98)\approx-3.8918\). Its approximate standard error is \(1/\sqrt{400(0.02)(0.98)}=1/\sqrt{7.84}\approx0.3571\). (2) Slope information depends on treatment variation and fitted probabilities in both covariate groups. A very large untreated group can still contribute little information if the treated group is sparse or has near-zero fitted probabilities. (3) No. Marginal prevalence does not reveal whether treatment probabilities are near zero in particular covariate patterns; those patterns must be examined. (4) It establishes agreement in the overall predicted and observed treatment totals. It does not establish subgroup calibration, covariate balance, or adequate overlap. (5) It may be useful when sparse data or separation makes ordinary logistic estimates divergent or unstable. It cannot create information or remedy inadequate treatment-control support in relevant covariate regions.