Why Overfitting Matters for Propensity Scores
A propensity model estimates the probability of receiving treatment given measured covariates. If the model is too flexible for the available sample, it can fit accidental patterns in the observed treatment indicators rather than stable features of treatment assignment. In-sample predictions may then look highly accurate while estimated probabilities are pushed close to zero for some controls and close to one for some treated units.
Those extreme estimates matter because propensity-score methods depend on comparing units with relevant treatment probabilities. Very small or large fitted scores can indicate little empirical overlap, but they can also be artifacts of an unstable model. The central task is therefore not to make every score moderate. It is to distinguish plausible information about assignment from noise fitted by an overly complex specification.
For a logistic propensity model, the fitted score is obtained from a linear predictor \(\hat\eta_i\):
As \(\hat\eta_i\) becomes strongly negative, \(\hat e_i\) approaches zero; as it becomes strongly positive, \(\hat e_i\) approaches one. A flexible model can create large positive or negative predictors by combining many terms, interactions, or sparse category indicators. This can happen even when the underlying population probabilities are not so extreme.
How Complexity Can Fit Treatment Noise
Adding terms to a model can improve its fit to the observed treatment indicators. In a nested set of logistic models, the maximized likelihood cannot decrease when new parameters are added, because the larger model can reproduce the smaller model by setting the new coefficients to zero. That fact alone does not mean the larger model estimates the assignment mechanism better. Each additional parameter can also respond to chance differences in a limited sample.
The concern is especially acute when there are few observations relative to the number of estimated coefficients, when categories contain very few units, or when treatment is uncommon in parts of the covariate space. A flexible model may then nearly classify the observed treatment labels. Its predictions can be sharp in the observed data but unstable: a small change in the sample, or in a few treatment indicators, may substantially change coefficients and fitted scores.
Overfitting and separation are related but distinct. Separation occurs when covariates or their modeled combinations perfectly distinguish treated from untreated units, so the ordinary logistic maximum-likelihood estimate may not be finite. Overfitting is broader: the model may be excessively adapted to the sample and produce extreme, unstable predictions even without perfect separation. The earlier material on separation and penalized logistic regression describes those estimation issues in more detail.
Worked Example: A Flexible Model Nearly Classifies a Small Sample
Consider a hypothetical sample of eight units, with four treated and four controls. A model uses a highly flexible set of covariate terms. Suppose its fitted linear predictors are as follows; they are hypothetical model output, not population probabilities.
| Unit | Treatment \(T_i\) | Fitted predictor \(\hat\eta_i\) |
|---|---|---|
| A | 0 | -3.9 |
| B | 0 | -2.8 |
| C | 0 | -1.1 |
| D | 0 | -0.4 |
| E | 1 | 0.4 |
| F | 1 | 1.1 |
| G | 1 | 2.8 |
| H | 1 | 3.9 |
For unit A, the fitted score is
For unit H, it is
At the conventional classification threshold of 0.5, these fitted scores classify all eight observed treatment indicators correctly. That apparent success is not proof that the assignment model is good: the model used a very flexible specification for only eight observations. The remaining fitted scores range from approximately 0.0573 for unit B to 0.9427 for unit G, so most units also appear close to one assignment group. Without evidence that these sharp differences reflect the actual assignment process, they are a warning of possible overfitting.
A useful interpretation is conditional: the model has found a pattern that separates the treatment labels in this sample. Whether that pattern is stable, scientifically justified, and useful for covariate balance requires further assessment. Predicting the observed labels is not the objective of propensity estimation.
Why a Unit-Level Model Can Produce Scores at the Boundaries
The limiting case makes the mechanism clear. Suppose a logistic model includes a separate indicator for every unit, in addition to an intercept. Each unit then has its own adjustable predictor. For a treated unit, increasing its predictor raises its fitted probability toward one; for a control unit, decreasing its predictor lowers its fitted probability toward zero.
For one observation with treatment indicator \(T_i\) and fitted probability \(p_i\), the likelihood contribution is \(p_i\) when \(T_i=1\), and \(1-p_i\) when \(T_i=0\). In the unit-indicator model, the likelihood can approach one for every observation by sending each predictor in the direction that matches its observed label. The log-likelihood contributions approach zero, their largest possible value. But reaching probabilities exactly zero or one requires predictors to diverge, so a finite ordinary maximum-likelihood estimate need not exist.
This is an extreme illustration, not a recommended model. Real overfitting more often occurs through many interactions, flexible functions, or sparse categories rather than a separate parameter for every person. The example shows why flexibility can drive fitted scores to the boundaries: the model has enough freedom to accommodate the observed labels individually.
Extreme Fitted Scores Do Not Have One Explanation
The positivity or overlap condition concerns the treatment-assignment probabilities in the population. For a target population in which both treatment states are to be compared, a common formal version requires \(0<e(X)<1\) for covariate values occurring in that population; stronger uniform-overlap conditions bound the probabilities away from both boundaries. The appropriate condition depends on the estimand and target population.
An extreme \(\hat e_i\) does not by itself prove that this population condition fails. It may reflect genuine assignment patterns, sparse covariate support, separation, an overfit model, or some combination. Conversely, a model that returns only moderate scores does not prove adequate overlap: an overly restrictive model can conceal real differences in treatment assignment.
This distinction matters because the responses differ. If the data provide little support for comparing treatment groups in a region, changing the fitted model cannot create information that is absent. If the extremes are mainly due to unstable fitting, revising the model may produce more reliable estimates. Diagnostics should therefore examine the specification and the covariate distributions, not treat a score cutoff as a universal test.
Diagnostics for Possible Overfitting
No single diagnostic establishes overfitting. Look for a pattern of evidence, including:
- Many extreme fitted scores. Scores concentrated very close to zero or one can indicate sparse support, separation, or an overly flexible model. Report the distribution and range rather than relying on an unexamined cutoff.
- Model complexity relative to sample information. A large number of coefficients, especially with few treated units, sparse categories, or many interactions, makes unstable fitting more plausible. A simple parameter count is a warning indicator, not a universal rule for an acceptable model size.
- Signs of unstable estimation. Very large coefficients, large standard errors, convergence warnings, or marked sensitivity to small specification changes warrant investigation. A converged optimizer alone does not guarantee a trustworthy model.
- In-sample prediction that looks too successful. Near-perfect treatment classification is not the goal. It may signal that the model is learning sample-specific labels rather than yielding useful comparisons.
- Covariate balance after fitting. Assess whether the fitted scores support balance on the covariates relevant to the analysis. Good treatment prediction does not ensure balance, and poor balance may reveal that the fitted model is not serving its intended purpose.
Cross-validation, covered in the earlier material on cross-validation for propensity model tuning, can help compare specifications and guard against choosing a model solely for its in-sample fit. It should not be used as a substitute for checking balance and support. A model that predicts treatment labels well under a validation criterion is not automatically the best model for a causal comparison.
Model-Fitting Remedies and Their Trade-Offs
The most direct remedy is to fit a specification whose flexibility is justified by the sample and the scientific problem. Retain covariates needed to address confounding, but avoid adding terms merely to increase treatment prediction. Use defensible functional forms and interactions rather than an unrestricted collection of candidate terms. This does not mean omitting important confounders for the sake of moderate scores; it means balancing substantive requirements with what the data can estimate reliably.
Penalized logistic regression can constrain unstable coefficients. Ridge estimation shrinks coefficients toward zero, while lasso estimation can also set some coefficients to zero. These penalties may reduce extreme fitted scores, but they do not guarantee covariate balance, solve a genuine lack of overlap, or automatically identify the correct confounders. The amount of penalization should be selected and assessed with the model's purpose in mind.
Firth logistic estimation can produce finite estimates in settings with separation. It addresses the nonexistence or bias problems associated with ordinary maximum likelihood under separation; it is not a universal cure for excessive complexity. If many unnecessary terms remain, a finite estimate can still be unstable or poorly targeted.
The following comparison summarizes the choices. Each approach has a different role; none should be selected solely because it moves all fitted scores away from zero and one.
| Approach | Potential benefit | Important limitation |
|---|---|---|
| Simplify the prespecified model | Reduces the number of unstable terms and can improve interpretability. | Oversimplification can miss important confounding structure or yield poor balance. |
| Use ridge or lasso penalization | Constrains coefficients and can reduce sensitivity to noisy terms. | Penalty choice affects estimates; shrinkage does not guarantee adequate overlap or balance. |
| Use Firth estimation when separation is present | Can give finite estimates when ordinary logistic maximum likelihood fails under separation. | Does not by itself make an unnecessarily complex model appropriate. |
| Compare specifications using cross-validation and balance checks | Provides evidence about predictive stability and the intended balancing purpose. | Predictive performance alone does not determine whether a model is suitable for causal estimation. |
Worked Example: How Shrinkage Changes Fitted Scores
Consider a hypothetical logistic model with one covariate \(x\), where the fitted intercept is zero. Suppose the unpenalized fit has coefficient \(\hat\beta_1=4\), while a ridge-penalized fit to the same hypothetical data gives \(\hat\beta_1=2\). These are stipulated fitted coefficients for illustrating the score calculation, not a claim that ridge estimation always halves a coefficient.
For a unit with \(x=1\), the unpenalized predictor is 4, giving
For the penalized coefficient, the predictor is 2:
At \(x=-1\), the corresponding predictors are -4 and -2. The scores are approximately 0.0180 and 0.1192, respectively. In this illustration, shrinkage moves both fitted scores toward 0.5. Whether that movement improves propensity-score estimation still depends on whether the resulting model adequately addresses confounding and balance, and whether the observed covariates have sufficient overlap.
Common Mistake: Treating a Score Cutoff as a Cure
A common error is to see very small or large fitted scores, delete every unit beyond an arbitrary cutoff, and conclude that the overfitting problem has been fixed. Trimming may change the target population and discard units; it does not explain why the original model produced the extremes. If those scores result from an overfit specification, the model may still be unstable among retained units. If they reflect genuine lack of overlap, removing units changes which population is represented and must be justified accordingly.
The correction is to diagnose the source of the extremes: inspect the specification, sparse covariate patterns, separation, estimation stability, and covariate support. Then assess model-fitting alternatives and state what population the analysis represents. There is no universal numerical boundary that separates acceptable from unacceptable fitted scores in every setting.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions below.
- A logistic model gives \(\hat\eta=-2\) for a unit. Calculate its fitted propensity score and interpret it.
- Why can a model with more terms have a higher in-sample likelihood but still be a worse choice for propensity estimation?
- Does observing \(\hat e_i=0.01\) prove that the population positivity condition fails? Explain.
- Name two diagnostics and one model-fitting response that could help investigate potentially overfit scores.
Solutions. (1) \(\hat e=\exp(-2)/(1+\exp(-2))=1/(1+\exp(2))\approx 0.1192\). The fitted model assigns this unit an estimated treatment probability of about 11.9%, conditional on its measured covariates; this is not a statement about the probability that the model is correct. (2) Additional terms can fit accidental sample patterns and increase estimation variance, so a higher in-sample likelihood need not mean better representation of the assignment process or better covariate balance. (3) No. The extreme fitted score may reflect genuine limited support, sparse data, separation, or overfitting; a fitted estimate alone does not establish the population condition. (4) Suitable diagnostics include inspecting the fitted-score distribution, coefficient sizes and convergence, sensitivity to specification changes, and balance on measured covariates. Possible fitting responses include removing unjustified terms or considering ridge penalization, with balance and support reassessed afterward.