Calipers and Distance Metrics in Matching

Why Matching Needs a Distance Rule

Nearest-neighbor matching requires a way to compare units. When covariates include age, income, and disease severity, the question “Which control is closest to this treated unit?” has no unique answer until a distance metric is specified. A second choice is whether to accept every nearest neighbor, however distant, or to impose a maximum permitted distance. That maximum is a caliper.

These choices affect more than the mechanics of selecting a control. A metric determines which covariate differences count as large, and a caliper can leave treated units without matches. The resulting matched sample may be more comparable on measured covariates, but it may represent a narrower group than the original treated population. Neither a particular metric nor a tight threshold guarantees a causal interpretation; that still depends on the design and identification assumptions.

Definition: Let \(T_i\) indicate treatment for unit \(i\), and let \(X_i\) denote its measured pretreatment covariates. A distance metric \(d(i,j)\) assigns a nonnegative measure of dissimilarity to two units. A caliper \(c\) permits a candidate pair only if \(d(i,j) \leq c\). Among eligible pairs, a matching rule, such as nearest-neighbor matching, selects matches using the specified distance.

Distance Metrics: What Counts as Close?

For covariate vectors \(X_i=(X_{i1},\ldots,X_{ip})\), raw Euclidean distance is

d_E(i,j)=\sqrt{\sum_{k=1}^{p}(X_{ik}-X_{jk})^2}.

This treats a one-unit difference on every covariate as equally large. That may be a poor choice when covariates use different units or have very different ranges. For example, a difference of one year in age and a difference of one dollar in income both contribute \(1\) to the squared distance, even though their substantive scales are very different.

Standardized Euclidean distance divides each covariate difference by a scale \(s_k\), often its standard deviation in a stated reference sample:

d_S(i,j)=\sqrt{\sum_{k=1}^{p}\left(\frac{X_{ik}-X_{jk}}{s_k}\right)^2}.

It compares differences relative to the observed variation of each covariate. The choice of reference sample and scale should be reported, because changing them can change the ranking of candidate matches. Standardization does not automatically give every covariate equal scientific importance; it is a scale adjustment, not a substitute for design judgment.

A common alternative is Mahalanobis distance. Let \(S\) be a positive-definite covariance matrix for the covariates in a chosen reference sample. Then

d_M(i,j)=\sqrt{(X_i-X_j)^\mathsf{T}S^{-1}(X_i-X_j)}.

Unlike standardized Euclidean distance, which uses only individual covariate scales, Mahalanobis distance also accounts for covariation among the covariates. A difference in a direction that commonly occurs together across covariates may count differently from a difference in an unusual direction. The covariance estimate and the sample used to calculate it therefore matter.

Matching may instead use the propensity score \(e(X_i)=P(T_i=1\mid X_i)\), or its logit, as a one-dimensional distance. For example, a logit propensity-score distance is \(\left|\operatorname{logit}e(X_i)-\operatorname{logit}e(X_j)\right|\). This gives a compact ranking rule, but a small score distance is not the same thing as a guarantee that every covariate is individually close. A hybrid rule can use a propensity-score caliper to restrict candidates, then use Mahalanobis distance on selected covariates to rank the eligible controls.

Worked Example: Raw Versus Standardized Euclidean Distance

Suppose a treated patient has age 50 years and severity score 6. Two hypothetical controls have covariates \(C_1=(51,8)\) and \(C_2=(54,6)\), with age in years and severity on a numerical scale. First compare the controls using raw Euclidean distance.

For \(C_1\), the squared distance is \((51-50)^2+(8-6)^2=1^2+2^2=5\), so \(d_E=\sqrt{5}\approx2.236\). For \(C_2\), it is \((54-50)^2+(6-6)^2=4^2+0^2=16\), so \(d_E=4\). Raw Euclidean distance selects \(C_1\).

Now suppose the standard deviations in the chosen reference sample are 5 years for age and 1 point for severity. The standardized distances are \(d_S(T,C_1)=\sqrt{(1/5)^2+(2/1)^2}=\sqrt{4.04}\approx2.010\) and \(d_S(T,C_2)=\sqrt{(4/5)^2+0^2}=0.800\). Standardized distance selects \(C_2\).

The answer changes because raw distance treats numerical differences in their recorded units: \(C_1\)'s squared distance is \(1^2+2^2=5\), versus \(4^2=16\) for \(C_2\). Standardization instead expresses each difference relative to its observed variation. Here, a two-point severity difference is large relative to a one-point standard deviation, while a four-year age difference is modest relative to a five-year standard deviation.

Calipers Restrict Eligibility

A caliper is defined on a particular distance scale. A threshold of 0.05 on a propensity-score scale is not interchangeable with 0.05 on a logit propensity-score scale, or with 0.05 on a Mahalanobis scale. Thus, a report of “caliper 0.2” is incomplete unless it identifies the distance, units or standardization, and reference scale.

One widely used design convention is a caliper equal to 0.2 standard deviations of the logit propensity score. This is a heuristic, not a universal optimum. Its suitability depends on the application, the sample, and the target population. A very wide caliper allows more matches but permits more distant pairs. A very narrow caliper limits permitted discrepancies but may leave more units unmatched.

For treated-unit matching, define \(R_i=1\) when treated unit \(i\) is retained with an eligible control, and \(R_i=0\) otherwise. If a caliper excludes every possible control for a treated unit, that unit cannot contribute to a matched treated-group effect. The retained treated group is therefore part of the interpretation, not merely an implementation detail.

Worked Example: A Propensity-Score Caliper and Retention

Consider three hypothetical treated units with estimated propensity scores 0.20, 0.50, and 0.80, and three controls with scores 0.18, 0.46, and 0.62. Match each treated unit to its nearest control by absolute score difference, without replacement. Impose a caliper of 0.05 on the propensity-score scale.

The eligible differences are calculated directly. For treated unit 1, the nearest control is the one at 0.18, with difference \(|0.20-0.18|=0.02\), so it is retained. For treated unit 2, the nearest is 0.46, with difference \(|0.50-0.46|=0.04\), so it is retained. Treated unit 3's nearest available control is 0.62, with difference \(|0.80-0.62|=0.18\), which exceeds 0.05, so that treated unit is not matched.

Suppose the hypothetical observed outcomes for the two retained pairs are as follows: for the first pair, treated outcome 12 and control outcome 9; for the second pair, treated outcome 15 and control outcome 11. The matched treated-minus-control differences are \(12-9=3\) and \(15-11=4\). Their average is \((3+4)/2=3.5\) outcome units.

This estimate describes the average matched difference for the two retained treated units, not automatically the average treatment effect for all three treated units. With a wider caliper of 0.20, the third treated unit could be paired to the control at 0.62, but that changes both the set of pairs and the degree of permitted score discrepancy. A wider caliper does not itself make the additional pair a valid comparison.

What Population Does the Matched Estimate Target?

Let \(Y_i(1)\) and \(Y_i(0)\) be unit \(i\)'s potential outcomes under treatment and control, and let \(\tau_i=Y_i(1)-Y_i(0)\) be its individual treatment effect. The average treatment effect for the treated (ATT) in the original treated group is \(E[\tau_i\mid T_i=1]\). If a caliper retains only some treated units, the natural retained-group target is \(E[\tau_i\mid T_i=1,R_i=1]\). The two targets need not be equal.

The reason follows by partitioning the original treated group into retained and nonretained units. If both groups have positive probability, then

E[\tau_i\mid T_i=1] = P(R_i=1\mid T_i=1)E[\tau_i\mid T_i=1,R_i=1] + P(R_i=0\mid T_i=1)E[\tau_i\mid T_i=1,R_i=0].

This is a weighted average of the treatment effects among retained and nonretained treated units. It equals the retained-group average only under additional conditions, such as equal average effects in the two groups or no nonretained treated units. Thus, a caliper can change the estimand when it changes which treated units remain.

For a causal interpretation among retained treated units, the usual identification conditions must still be credible. In particular, conditional independence requires \((Y_i(1),Y_i(0))\) to be independent of \(T_i\) given the measured pretreatment covariates \(X_i\), and the retained treated units must have suitable control comparisons in the relevant covariate region. The matching rule cannot adjust for unmeasured confounding, and a caliper cannot create comparable controls where none are available.

Mahalanobis Distance With a Caliper

The following example illustrates how a covariance-based metric can be calculated and used for eligibility. It also emphasizes that a caliper is applied after specifying the metric and its scale.

Worked Example: Covariate Correlation and Mahalanobis Distance

Suppose two covariates have covariance matrix \(S=\begin{pmatrix}4&2\\2&4\end{pmatrix}\) in the stated reference sample. A treated unit has covariate vector \((0,0)\). Two hypothetical controls have vectors \(C_1=(1,1)\) and \(C_2=(2,0)\). The inverse matrix is \(S^{-1}=\frac{1}{12}\begin{pmatrix}4&-2\\-2&4\end{pmatrix}\), as can be verified by multiplying it by \(S\).

For \(C_1\), the difference vector is \(v_1=(1,1)^\mathsf{T}\). Its squared Mahalanobis distance is \(v_1^\mathsf{T}S^{-1}v_1=(4-2-2+4)/12=4/12=1/3\), so \(d_M(T,C_1)=\sqrt{1/3}\approx0.5774\).

For \(C_2\), \(v_2=(2,0)^\mathsf{T}\). Its squared distance is \(v_2^\mathsf{T}S^{-1}v_2=16/12=4/3\), so \(d_M(T,C_2)=\sqrt{4/3}\approx1.155\). Thus, \(C_1\) is closer by this metric.

If the design sets a Mahalanobis caliper of 0.8, \(C_1\) is eligible because \(0.5774\leq0.8\), whereas \(C_2\) is not because \(1.155>0.8\). The caliper here is in Mahalanobis-distance units; it should not be read as a threshold in the original covariate units. The result also depends on the covariance matrix used, so that matrix and its reference sample should be specified.

Choosing and Reporting a Rule

A defensible distance choice starts with the covariates that matter for comparability and the scales on which they are measured. A useful design can combine an explicit restriction with a ranking rule: for example, require a sufficiently close propensity score, then choose among eligible controls using Mahalanobis distance on selected covariates. Exact restrictions can be used for important categorical variables, rather than treating arbitrary numeric codes as meaningful distances.

Decisions should be made and reported transparently. At minimum, state the distance metric, any transformations or scaling, the caliper threshold and scale, the matching rule, and how unmatched treated units are handled. Also report retention and describe the resulting target population. The balance and outcome analyses are then interpreted in light of this design, rather than treating a chosen threshold as proof that the samples are comparable.

Common Mistakes

Comparing caliper numbers without their scales. A caliper of 0.1 is meaningful only after naming the distance and its scale. State whether it is on the propensity score, logit score, standardized covariates, or another metric.
Assuming raw Euclidean distance is unit-neutral. It uses numerical differences directly. A large-scale covariate can dominate the squared distance, so consider the units and justify the metric.
Assuming a tighter caliper always improves the overall analysis. It restricts eligible pairs, but can also reduce retention and narrow the population represented by the estimate.
Calling a retained-sample estimate the original ATT without qualification. If some treated units are excluded, describe the retained treated population and explain why its target may differ.
Treating a small propensity-score distance as proof of covariate similarity. The score is a scalar summary; the distance rule alone does not guarantee close values on every covariate.

Key takeaway: A metric defines what “near” means; a caliper defines how near is close enough. Together they affect eligible matches, retained units, and potentially the causal estimand. Specify both choices and interpret the matched result for the population that actually remains.

Check Your Understanding

Answer each question by stating the relevant distance or target in context.

Why can raw Euclidean distance rank controls differently from standardized Euclidean distance when covariates use different scales?
A treated unit's nearest control has propensity-score distance 0.07. Is it eligible under a propensity-score caliper of 0.05? What additional information should be reported about this rule?
What does a Mahalanobis distance use that standardized Euclidean distance does not?
If a caliper excludes some treated units, what population does a matched treated-group estimate most directly describe?
Under what broad condition could the retained-group average treatment effect equal the original ATT even when some treated units are unmatched?