Why Propensity-Score Matching May Not Balance Covariates

Why Score Matches Can Still Be Poor Covariate Matches

Propensity-score matching compresses several pretreatment characteristics into one estimated probability of treatment. That reduction can be useful, but it can also obscure an important diagnostic question: after matching, do the treated and comparison groups actually look alike in the covariates that matter? A small difference in estimated scores is not itself evidence that every covariate is balanced.

Balance is a property of the matched sample, not a guarantee supplied by the matching algorithm. This matters because residual differences in measured covariates can undermine the credibility of a treatment-effect comparison. Balance checks can reveal such differences; they cannot establish that unmeasured confounding is absent or that the causal assumptions hold.

The Balancing Property and Its Limits

Let \(T\) indicate treatment, with \(T=1\) for treated units and \(T=0\) for comparison units, and let \(X\) denote the vector of pretreatment covariates. The propensity score is the conditional probability of treatment given those covariates. For a causal interpretation, a common set of assumptions includes consistency, conditional exchangeability, and positivity. In particular, conditional exchangeability requires that, given \(X\), treatment assignment is independent of the potential outcomes \(Y(1)\) and \(Y(0)\); positivity requires a nonzero probability of either treatment status at covariate values relevant to the target population.

The balancing result can be seen directly. Conditional on a value \(e\) of the true score, the treatment probability is the same for every covariate value in that score group. For any covariate set \(A\), the probability of a treated unit having covariates in \(A\), conditional on \(e\), therefore factors as follows:

The same argument applies to comparison units when \(0<e<1\), so the covariate distributions agree between treatment groups conditional on the true score. But matching usually uses an estimated score, \(\hat e(X)\), and selects a finite set of pairs or neighbors. The estimate need not preserve the true balancing property exactly. Even if the score model is well specified, estimation error and the discrete selection of matches can leave residual differences. And even when scores are nearly equal, a single scalar distance does not require each component of \(X\) to be close.

How to Assess Covariate Balance

A balance table compares each pretreatment covariate between the treated and comparison groups, both before and after matching. For a continuous covariate \(X_j\), a common summary is the standardized mean difference (SMD): the difference in group means divided by a standard deviation used as a reference. A useful convention is to use the pooled standard deviation from the original, unmatched sample for both the before- and after-matching values. Keeping the denominator fixed makes the two SMDs easier to compare.

Here the subscripts 1 and 0 indicate treated and comparison groups, and the reference standard deviation is calculated before matching. For a binary covariate, the analogous standardization uses the pooled proportion \(p_{\mathrm{pool}}\):

After matching with replacement, comparison units that are selected multiple times should contribute according to the matching weights when calculating post-match summaries. The same principle applies to any matching scheme that assigns unequal weights. The chosen convention for standardization should be stated, since different denominators can produce different numerical SMDs.

SMDs describe the size of a difference relative to a scale; they are not hypothesis tests. A small p-value can reflect a large sample even when a covariate difference is practically modest, while a nonsignificant test does not establish balance. Absolute SMDs below 0.1 are sometimes used as a rough screening convention, but this is not a universal pass/fail rule. Consider the covariates’ substantive importance, their distributions, and the matching design.

A table should be supplemented with plots when distributional features may be hidden by a mean. Useful displays include side-by-side histograms or empirical cumulative distribution functions for continuous variables, and dot plots of absolute SMDs across covariates (often called Love plots). Check not only the covariates entered as simple terms, but also substantively important nonlinear transformations and interactions. If matching was intended to retain treated units, also report how many treated units were retained and make clear that any resulting effect estimate pertains to the retained group rather than automatically to all treated units.

Why the Estimated Score Can Hide Imbalance

There are several distinct reasons that score matching can fail to balance observed covariates. First, the score model may omit nonlinear terms or interactions that help explain treatment assignment. Second, even a well-specified model produces estimates from finite data. Third, matching on one-dimensional score distance may not select the covariate combinations that best balance the full vector. Finally, ties, caliper choices, limited overlap, or the order in which neighbors are selected can affect which units are retained. These mechanisms are different: a table can reveal imbalance, but it does not by itself identify which mechanism caused it.

The following example shows the difference between a fitted-score match and balance on a covariate transformation. It is a hypothetical illustration of how a model using only a linear term can fail to represent a nonlinear treatment-assignment pattern.

Small Score Distances Do Not Guarantee Covariate Balance

Even without a nonlinear-model example, a score match can have tiny distances and still leave a large difference in an observed covariate. The next example separates two quantities that are easy to confuse: closeness of estimated scores and balance of covariates.

A Practical Diagnostic Sequence

A disciplined diagnostic sequence makes it less likely that a single reassuring score summary will be mistaken for evidence of balance. It also keeps the design decision separate from the subsequent outcome comparison.

Common Mistake: Treating a Good Score Match as a Balance Test

A tempting but incorrect conclusion is: “The matched scores are nearly equal, so the covariates must be balanced.” This confuses a theoretical property of the true propensity score in the population with a finite-sample result obtained by matching on an estimated score. Score closeness is useful design information, but it is not a substitute for checking each covariate and its important distributional features.

The correction is to report balance diagnostics directly and interpret them as diagnostics, not as proof of causal identification. Good measured-covariate balance supports the design’s comparability on those measured characteristics. It does not verify conditional exchangeability, guarantee overlap, or rule out bias from unmeasured factors. Conversely, a detected imbalance is a reason to investigate the design before interpreting an outcome contrast as a treatment effect.