Distances for Propensity Score Matching

Why Measure Distance Between Propensity Scores?

After estimating a propensity score for each unit, a matching procedure needs a way to compare treated units with potential control units. A score distance summarizes how far apart two units are on the estimated probability scale. It is a calculation on the fitted scores, not a guarantee that the units are similar on every covariate or that a causal effect has been identified.

Let \(T_i\) denote the treatment indicator for unit \(i\), where \(T_i=1\) indicates treatment and \(T_i=0\) indicates control. The propensity score is \(e(X_i)=P(T_i=1\mid X_i)\), where \(X_i\) is the vector of measured pretreatment covariates. Write \(\hat e_i\) for the estimated propensity score of unit \(i\). For a treated unit \(i\) and a control unit \(j\), compare \(\hat e_i\) with \(\hat e_j\).

Because estimated propensity scores are probabilities between 0 and 1, absolute distances are between 0 and 1, and squared distances are also between 0 and 1. An absolute distance of 0.04 means the estimated probabilities differ by 0.04, or 4 percentage points. A squared distance of \(0.04^2=0.0016\) is the square of that probability difference; it should not be interpreted as a 0.16-percentage-point difference.

Calculating Pairwise Distances

To calculate a distance, subtract one estimated score from the other. Take the absolute value for the absolute distance. For the squared distance, square the difference; because squaring removes its sign, it gives the same result whether the subtraction is written treated minus control or control minus treated.

The score is a fitted quantity. For the distance calculation, use scores estimated on a comparable basis for all units being considered—for example, scores from the same model and scoring procedure. The arithmetic itself does not require a causal assumption. However, using matching to support a causal comparison requires appropriate design and identification conditions, including measured pretreatment confounders, conditional exchangeability given those covariates, positivity in the target population, and a suitable score-estimation and matching strategy. A small score distance alone establishes none of these conditions.

Why Absolute and Squared Distances Give the Same Pair Ranking

For any single pair, the squared distance is the square of the absolute distance. Since squaring is increasing for nonnegative numbers, the pair with the smaller absolute distance also has the smaller squared distance. This result means the two measures cannot disagree about which individual pair is closer by score. It does not mean their numerical values are interchangeable.

If \(d_1=d_2\), then \(d_1^2=d_2^2\) as well, so ties are preserved. This argument applies to a comparison of pairwise distances: the order of candidate pairs is unchanged when each nonnegative distance is squared. It does not imply that a procedure aggregating several distances—such as minimizing a sum—will generally have the same optimum under absolute and squared distances. Summing after a nonlinear transformation is a different calculation from transforming a sum.

Precision, Rounding, and Ties

Distances are computed from the estimated scores available to the analyst. When scores are rounded for display, distinct underlying scores can appear equal, and genuinely close distances can be difficult to order from the displayed values. Keep adequate precision for calculations and round only when reporting. A tie created by rounded values should not automatically be treated as an exact tie in the underlying estimates.

Even with unrounded estimates, two candidate controls may have exactly the same score distance from a treated unit. For example, scores 0.47 and 0.53 are each 0.03 away from a treated score of 0.50. Both have squared distance \(0.0009\). The distance calculation says they are equally close on the score scale; it does not provide a basis for declaring one a better match on other grounds.

What Score Distance Does—and Does Not—Measure

A score distance measures separation in one scalar summary of the covariates: the estimated probability of treatment conditional on those covariates. It is not a direct measure of the difference on every covariate. Two units can have similar estimated scores while differing on particular covariates, especially if those differences offset one another in the fitted score model or the model represents the treatment assignment process poorly. Balance diagnostics after matching remain necessary; score closeness is not a substitute for them.

The choice of distance should be understood in context. Absolute distance reports a gap directly in probability points and is easy to interpret. Squared distance retains the same pairwise ordering but gives relatively greater numerical weight to larger gaps, which can matter when distances are combined in an objective. Other approaches compare covariates directly or use a transformed score scale. Those approaches measure a different notion of similarity; their numerical distances should not be confused with probability-scale score differences.

Estimated scores also carry estimation uncertainty. The pairwise arithmetic treats the fitted values as given and does not represent that uncertainty. A very small difference between fitted scores is not evidence that the true propensity scores are known precisely or that the two units are exchangeable. The uncertainty and the adequacy of score estimation should be considered separately from the distance calculation.

Common Mistakes and Diagnostics

• Forgetting the absolute value. A treated score of 0.62 and a control score of 0.70 differ by \(-0.08\) if subtracted in that order, but their absolute distance is \(0.08\), not \(-0.08\).
• Calling a squared distance a percentage-point gap. The distance \(0.0064\) is the square of an \(0.08\) probability difference; it is not an 0.64-percentage-point gap.
• Assuming score closeness proves covariate balance. It does not. Inspect covariate balance after matching and assess whether estimated scores and overlap are adequate for the intended comparison.
• Comparing scores estimated on inconsistent bases. Scores produced using different models or scoring procedures may not be comparable as though they came from one common scale. Verify how each score was obtained before calculating pairwise distances.
• Using rounded values for close comparisons. Display rounding can create apparent ties or zero distances. Calculate from the available unrounded estimates when possible.
• Treating a distance rule as a causal assumption. A distance is a numerical criterion for comparing fitted scores. It does not establish conditional exchangeability, positivity, or correct measurement of confounders.

As a basic diagnostic, check that every score is between 0 and 1, recompute a few distances by hand, and confirm that squared distances equal the squares of the corresponding absolute distances. If candidate rankings differ between the two measures for pairwise comparisons, check for arithmetic errors, inconsistent rounding, or a different definition of distance.