Propensity Score Matching in Observational Studies

Introduction

Randomization is what makes a clinical trial's naive treatment-vs-control comparison valid — it guarantees, on average, that treated and untreated groups are balanced on every characteristic, measured or not. Observational data has no such guarantee. When treatment assignment depends on patient characteristics that also affect the outcome — "confounding by indication," where sicker patients are more likely to receive a drug precisely because they're sicker — a naive comparison can be biased in either direction, including reversing the true effect entirely. This tutorial builds that failure and its correction from real, simulated data: propensity score matching, balance diagnostics, and inverse probability weighting, all computed and compared against a known true effect.

A Real Demonstration: Confounding Reverses the Apparent Effect

Simulating an 800-patient observational study of a drug's effect on hospitalization risk, where treatment assignment genuinely depends on age and comorbidity burden (older, sicker patients are more likely to receive the drug — realistic confounding by indication), and the drug's true causal effect is protective (true log-odds effect = −0.500, true OR = 0.607):

The naive comparison shows the treated group with a higher hospitalization rate — the confounding is strong enough to completely reverse the apparent direction of a truly beneficial treatment. Fitting a naive logistic regression confirms this numerically:

Wrong sign, and not even statistically significant — a naive analyst would conclude the drug shows no benefit or possibly harm, when it is genuinely protective.

The Propensity Score

The propensity score is the probability of receiving treatment given observed covariates:

estimated here via logistic regression on the two known confounders:

Rosenbaum & Rubin's (1983) key theoretical result is what makes this useful: conditioning on the propensity score alone is sufficient to balance the full set of observed covariates that went into estimating it — matching or weighting on a single number achieves what would otherwise require balancing every confounder separately.

Checking Common Support

Propensity score distributions by treatment group. Substantial overlap exists across most of the range, which is a prerequisite for valid matching — if treated and control patients occupied entirely non-overlapping propensity score ranges, no amount of matching could recover a valid comparison, since there would be no comparable control patients for the treated population's characteristics.

Matching, and the Balance It Achieves

1:1 nearest-neighbor matching on the logit of the propensity score, with a caliper of 0.2 standard deviations (a standard default), matched 191 of the 193 treated patients to a comparable control. Checking covariate balance before and after, using the standardized mean difference (SMD) — a common rule of thumb flags SMD > 0.1 as meaningful imbalance:

A "Love plot" — standardized mean differences before and after matching, with the conventional ±0.1 balance threshold marked. Both confounders start well outside acceptable balance and land almost exactly at zero after matching. This diagnostic, not the outcome analysis itself, is the first thing to check in any real propensity score analysis: if balance isn't achieved, the outcome comparison downstream isn't trustworthy regardless of what it shows.

The Outcome Analysis, Three Ways

With balance confirmed, comparing outcomes on the matched sample — and alongside it, two alternative approaches to the same confounding problem:

All three adjustment methods correctly recover the true direction and land close to the true magnitude — matching's OR of 0.602 is remarkably close to the true 0.607 here. The naive comparison remains the clear outlier: wrong sign, non-significant, and would lead to an entirely incorrect clinical conclusion if taken at face value.

Inverse Probability of Treatment Weighting (IPTW)

Rather than discarding unmatched subjects, IPTW reweights the entire sample so that, on average, the weighted population is balanced on the propensity score — each subject is weighted by \(1/e(X_i)\) if treated or \(1/(1-e(X_i))\) if control (commonly stabilized to avoid extreme weights from patients with propensity scores near 0 or 1). IPTW uses all 800 subjects rather than only the 382 in matched pairs, at the cost of being more sensitive to propensity score model misspecification and extreme weights when scores approach the boundaries.

Shortcomings and Key Details

• Only measured confounders can be balanced. Propensity score methods, like covariate adjustment, cannot correct for confounding by an unmeasured variable — if age and comorbidity hadn't both been recorded and included in the propensity model here, no amount of matching would have fixed the bias. This is the same fundamental limitation observational methods share with any regression adjustment; it does not solve unmeasured confounding, only measured confounding.
• Common support is a real requirement, not a formality. Checking the propensity score distributions (above) for adequate overlap should happen before interpreting any matched or weighted result — poor overlap means the "comparable" patients being compared aren't actually comparable.
• Matching discards data (2 of 193 treated patients here found no adequate match), while IPTW retains everyone but can be destabilized by extreme weights for patients with propensity scores near 0 or 1 — the trade-off between these two approaches is a real design decision, not a default.
• The propensity score model itself needs the same diagnostic care as any regression — correct functional form, relevant interactions, adequate covariates — a poorly specified propensity model produces poor balance regardless of the matching or weighting algorithm applied afterward.

Summary

The demonstration here is about as stark as confounding gets: a truly protective treatment (true OR=0.607) appeared harmful and non-significant (OR=1.129, p=0.47) under a naive comparison, purely because sicker patients were more likely to receive it. Propensity score matching, verified through real balance diagnostics rather than assumed, recovered an estimate (OR=0.602) within a hair of the true effect — and covariate-adjusted regression and IPTW, while less precise here, both correctly recovered the true direction as well. The common thread across all three corrections: they all depend entirely on having measured the confounders in the first place, which is the one thing no adjustment method, however sophisticated, can substitute for.

References

Rosenbaum, P.R. & Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika.
Austin, P.C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research.
Rubin, D.B. (2001). Using propensity scores to help design observational studies. Health Services and Outcomes Research Methodology.
Robins, J.M., Hernan, M.A. & Brumback, B. (2000). Marginal structural models and causal inference in epidemiology. Epidemiology.