Estimating Propensity Scores for Treatment Adjustment

Why Estimate a Propensity Score?

In an observational study, people who receive a treatment may differ systematically from those who do not. The propensity score summarizes measured pretreatment differences in a single probability: the probability of receiving treatment given those covariates. Estimating it is a way to operationalize an adjustment strategy, not a way to make treatment assignment random or to guarantee that unmeasured confounding has disappeared.

Let $D_i$ equal 1 if unit $i$ receives treatment and 0 otherwise, and let $X_i$ denote the vector of observed pretreatment covariates. The population propensity score is $e(X_i)=P(D_i=1\mid X_i)$. For causal adjustment, the relevant assumptions include conditional independence of treatment and potential outcomes given $X_i$, and positivity: units in the target population have a nonzero probability of either treatment level given their covariates. Estimating a score does not establish either assumption.

Definition: The propensity score is the conditional treatment probability $e(X)=P(D=1\mid X)$, where $D$ is a binary treatment indicator and $X$ contains measured pretreatment covariates. An estimated propensity score, $\hat e(X_i)$, is a fitted estimate of this probability for unit $i$.

The estimation task is to model $P(D_i=1\mid X_i)$ while representing the covariate structure well enough for the intended adjustment. The score is not an outcome model: the response being modelled is treatment receipt, not the study outcome.

Logistic and Probit Models

A common specification uses logistic regression. Write $\eta_i=X_i^\mathsf{T}\beta$, where $X_i$ includes an intercept and the chosen covariate terms. The model sets the log-odds of treatment equal to this linear predictor, then transforms those log-odds to a probability.

\log\left(\frac{e(X_i)}{1-e(X_i)}\right)=X_i^\mathsf{T}\beta, \qquad \hat e(X_i)=\frac{\exp(X_i^\mathsf{T}\hat\beta)} {1+\exp(X_i^\mathsf{T}\hat\beta)}.

The coefficients are typically estimated by maximum likelihood. Under the logistic model, the likelihood for independent observations is $L(\beta)=\prod_{i=1}^n e(X_i)^{D_i}[1-e(X_i)]^{1-D_i}$, with $e(X_i)$ given by the logistic expression. Maximizing this likelihood produces $\hat\beta$. A coefficient describes a change in conditional log-odds per unit change in its covariate, holding included terms fixed; it is not a change in probability of the same size.

Probit regression is another standard choice. It uses the standard normal cumulative distribution function $\Phi$ to map the linear predictor to a probability:

e(X_i)=\Phi(X_i^\mathsf{T}\beta), \qquad \hat e(X_i)=\Phi(X_i^\mathsf{T}\hat\beta).

Both links produce probabilities between 0 and 1. Their fitted probabilities can be similar over much of the covariate range, but the link alone does not determine whether the model is appropriate. The covariates and their functional forms matter: a linear term for age, for example, does not automatically capture curvature, and a model without interactions assumes that covariate relationships do not vary across the included factors.

Worked Example: Calculating Scores From Fitted Models

Suppose a hypothetical study has fitted a logistic model with estimated linear predictor $\hat\eta=-1.2+0.03(\text{age})+0.7(\text{rural})$, where age is in years and rural is 1 for a rural residence and 0 otherwise. Calculate the estimated score for a 50-year-old rural participant and a 30-year-old urban participant.

For the 50-year-old rural participant, $\hat\eta=-1.2+0.03(50)+0.7(1)=1.0$. Therefore, $\hat e=1/(1+\exp(-1.0))=0.7311$. The fitted model assigns this participant an estimated treatment probability of about 0.731.

For the 30-year-old urban participant, $\hat\eta=-1.2+0.03(30)+0.7(0)=-0.3$. Thus, $\hat e=1/(1+\exp(0.3))=0.4256$, or about 0.426.

These are fitted probabilities conditional on the model and included covariates, not known treatment probabilities. For comparison, suppose a fitted probit model gives $\hat\eta=-1.0+0.02(\text{age})+0.4(\text{rural})$. For the first participant, $\hat\eta=0.4$, so $\hat e=\Phi(0.4)\approx0.6554$. The different number reflects a different fitted specification and link; it does not by itself show that either model is preferable.

Why the Score Can Support Adjustment

The key result is a property of the true propensity score. If treatment is conditionally independent of potential outcomes given $X$, then reducing $X$ to the true score preserves the treatment comparability needed for adjustment. The balancing property says that, among units with the same true score, measured covariates have the same distribution in the treated and untreated groups.

Theorem: For the true propensity score $e(X)=P(D=1\mid X)$, treatment assignment is conditionally independent of $X$ given $e(X)$: $D\perp X\mid e(X)$. This is a statement about the true score. An estimated score from a misspecified model need not balance covariates.

A short derivation makes the result clear. Since $e(X)$ is a function of $X$, conditioning on both $X$ and $e(X)$ gives $P(D=1\mid X,e(X))=P(D=1\mid X)=e(X)$. Now condition only on the score and use iterated expectations:

P(D=1\mid e(X)) =E[D\mid e(X)] =E\{E[D\mid X,e(X)]\mid e(X)\} =E\{e(X)\mid e(X)\} =e(X).

Thus $P(D=1\mid X,e(X))=P(D=1\mid e(X))$, which is the conditional-independence statement. The result explains why a score can be useful for adjustment, but it does not say that any fitted score will work. In practice, $\hat e(X)$ is estimated, the model may omit important features, and balance must be examined after applying the intended adjustment.

Choose Functional Forms by Balance

The score model should include pretreatment covariates that are important for treatment assignment and confounding, with functional forms flexible enough to represent their relationships. Depending on the application, this may mean including nonlinear terms, categories, or interactions. Such terms should be substantively motivated and assessed empirically; adding every possible term can create unstable estimates, especially when treatment groups are small or covariate combinations are sparse.

A central diagnostic is whether the proposed adjustment produces similar covariate distributions between treatment groups. For a quantitative covariate $X$, a common summary is the standardized mean difference:

\operatorname{SMD}(X)= \frac{\bar X_1-\bar X_0} {\sqrt{(s_1^2+s_0^2)/2}},

where $\bar X_1,\bar X_0$ and $s_1,s_0$ are the means and standard deviations for treated and untreated units in the population being compared. Balance assessments should also examine categorical covariates, nonlinear terms, interactions, and distributional features where relevant. A small difference in means alone does not establish that the full distributions are similar.

Worked Example: Comparing Balance Under Two Specifications

Consider hypothetical balance summaries for age and annual income, measured after applying the same prespecified score-based adjustment under two candidate propensity models. Under the linear specification, the adjusted treated and untreated age means are 51 and 50 years, with both standard deviations equal to 10 years. For income, the corresponding means are $58,000 and $57,000, with both standard deviations equal to $15,000. Under a more flexible specification that adds a nonlinear age term and a treatment-predictor interaction, the adjusted age means are 50.5 and 50 years, with standard deviations of 10 years, and income means are $57,500 and $57,000, with standard deviations of $15,000.

For the linear specification, the age difference is $51-50=1$ year, so $\operatorname{SMD}_{\text{age}}=1/\sqrt{(10^2+10^2)/2}=1/10=0.100$. The income difference is $1,000, giving $\operatorname{SMD}_{\text{income}}=1{,}000/\sqrt{(15{,}000^2+15{,}000^2)/2}=1{,}000/15{,}000=0.0667$.

For the flexible specification, the age difference is 0.5 years, giving $\operatorname{SMD}_{\text{age}}=0.5/10=0.0500$. The income difference is $500, giving $\operatorname{SMD}_{\text{income}}=500/15{,}000=0.0333$. In these summaries, the flexible specification yields smaller mean differences for both covariates. This is evidence in its favor for these balance features, but a complete decision would also inspect other covariates and distributional features, and check that estimates remain stable.

The adjustment procedure must be specified when balance is reported: balance before adjustment is not the same as balance after adjustment. Do not choose a score model solely because its estimated probabilities look plausible or because its coefficients are statistically significant. The practical question is whether the fitted scores, used in the intended adjustment, reduce relevant measured differences without creating severe instability or discarding the population of interest.

Prediction Is Not the Objective

A model can predict treatment assignment accurately while providing poor covariate balance for causal adjustment. For instance, if only 10% of a sample is treated, a rule that predicts “untreated” for everyone achieves 90% classification accuracy. Yet it distinguishes no treated units, and the resulting constant score cannot account for differences in covariates between groups. Classification accuracy rewards correct labels; propensity-score adjustment needs a probability model whose use addresses measured confounding.

Worked Example: High Accuracy Can Hide Poor Adjustment

In a hypothetical sample of 100 people, 10 receive treatment and 90 do not. A classifier that labels everyone untreated is correct for 90 people, so its accuracy is $90/100=0.900$, or 90%. It assigns the same treatment prediction to all 100 people and therefore cannot distinguish the covariate profiles of the treated group from those of the untreated group.

Suppose instead that a candidate fitted score model has a classification accuracy of 84% at a chosen threshold, but the prespecified adjustment using its scores yields an age SMD of 0.06 and an income SMD of 0.04, compared with respective absolute SMDs of 0.45 and 0.32 before adjustment. Those diagnostics are more directly relevant to measured covariate balance, despite the lower classification accuracy. They do not prove that confounding is eliminated: unmeasured covariates, poor overlap, and model misspecification remain possible concerns.

The comparison depends on the adjustment procedure and the covariates examined. Neither accuracy nor a single balance statistic is a complete validation criterion. Report how scores were estimated, which covariates and functional forms were included, and what balance diagnostics were assessed.

Practical Checks and Common Mistakes

Before using estimated scores, confirm that the treatment indicator is coded correctly and that predictors are measured before treatment. A variable affected by treatment should not be inserted as a pretreatment confounder: conditioning on it can change the causal question and introduce bias. Check for convergence problems, sparse covariate patterns, and fitted probabilities extremely close to 0 or 1; these can signal difficulty estimating a stable model. Detailed overlap assessment is a separate diagnostic, but score estimation should not conceal its importance.

A common mistake is to choose the model with the highest classification accuracy and conclude that it is the best propensity model. That reasoning optimizes prediction of the treatment label, often favoring the majority group, rather than evaluating whether the intended adjustment achieves covariate comparability. The correction is to compare candidate specifications using balance diagnostics for the measured covariates relevant to the causal question, alongside checks on model stability and the target population.

Key takeaway: Logistic and probit regression provide standard ways to estimate treatment probabilities from pretreatment covariates. The balancing result motivates using a true propensity score, but an estimated score must be assessed in context: specify the functional form, apply the intended adjustment, and evaluate covariate balance rather than treating predictive accuracy as the goal.

Check Your Understanding

Try these questions without looking back.

In a logistic propensity model, what does the fitted linear predictor represent, and how is it converted to an estimated treatment probability?
State the balancing property of the true propensity score. Why does that property not guarantee balance for every estimated score?
A candidate model has excellent classification accuracy in a sample where treatment is rare. Explain why this alone is not evidence that it is suitable for causal adjustment.
For a covariate with adjusted treated and untreated means of 12 and 10 and standard deviations of 4 in each group, calculate its standardized mean difference.
Give two examples of functional-form terms that might be considered when a linear covariate specification does not achieve adequate balance.