Caliper Matching and the Local ATT

Why a Caliper Changes the Question

Matching treated units to controls that are similar on observed covariates can make a comparison more credible. Yet similarity is rarely perfect. A caliper sets a maximum permitted distance between a treated unit and a potential control on a chosen matching scale, often the estimated propensity score or its logit. Treated units with no eligible control under that rule may be left out.

That exclusion is not only a computational detail. If some treated units are omitted, an average effect among the remaining treated units targets a different population from the average treatment effect on all treated units. The caliper-matched estimate is therefore often best understood as a local ATT: an effect for treated units that satisfy the specified matchability rule. “Local” refers to this selected population, not necessarily to a geographic location.

Definition: Let \(A_i=1\) indicate that unit \(i\) is treated, \(X_i\) denote its observed pretreatment covariates, \(Y_i(1)\) and \(Y_i(0)\) its potential outcomes, and \(e(X_i)=P(A_i=1\mid X_i)\) its propensity score. Choose a score scale \(g\), such as \(g(e)=e\) or \(g(e)=\operatorname{logit}(e)\), and a caliper \(c>0\). A control \(j\) is eligible for treated unit \(i\) when \(|g(e(X_i))-g(e(X_j))|\le c\). Define \(R_c=1\) for treated units retained by a prespecified matching rule that requires at least one eligible control. The local ATT is \(\tau_c=E[Y(1)-Y(0)\mid A=1,R_c=1]\).

A caliper must be stated on its scale: a distance of 0.05 on the raw propensity-score scale is not equivalent to a distance of 0.05 on the logit scale. In a sample, eligibility also depends on which controls happen to be observed and on the specified matching algorithm. Thus the practical retained group is determined by the score, the threshold, the available controls, and the rule for selecting matches.

Assumptions Behind the Interpretation

A small score distance does not by itself establish that a matched comparison is causal. The usual potential-outcomes assumptions are still needed. Consistency requires that the observed outcome equals the potential outcome under the treatment actually received. Conditional exchangeability requires \((Y(1),Y(0))\perp A\mid X\): after conditioning on the measured pretreatment covariates, treatment assignment carries no further information about the potential outcomes. This is an assumption about measured and unmeasured causes of treatment and outcome; it cannot be verified from a close score match.

There must also be adequate positivity for the target being studied: treated units retained by the rule need comparable controls in the relevant covariate region. A caliper can enforce a defined degree of score closeness, but it cannot create controls where none exist, and close propensity scores do not guarantee close values on every covariate in a finite sample. The score model and the matching implementation must be appropriate, and covariates should be measured before treatment.

The propensity score is a balancing score under the standard assumptions: conditional on the true propensity score, treatment assignment is independent of the measured covariates. In practice, the score is estimated. Therefore, balance must be assessed after matching rather than inferred merely from the fact that a propensity-score caliper was used.

From the Matched Comparison to the Local ATT

For a simple one-to-one matched analysis with \(m\) retained treated units, let \(j(i)\) denote the control matched to treated unit \(i\). The usual estimator averages the treated-minus-control outcome differences:

\widehat{\tau}_c=\frac{1}{m}\sum_{i:A_i=1,\ R_{c,i}=1}\{Y_i-Y_{j(i)}\}.

Here \(Y_i=Y_i(1)\) for treated units and \(Y_{j(i)}=Y_{j(i)}(0)\) for controls. This formula describes the simple matched comparison; other matching and analysis rules can give different estimates and should be specified. The target interpretation follows from which treated units appear in the sum: the estimate does not automatically represent treated units excluded by the caliper.

The justification can be seen by considering the population comparison. For retained treated units, the expected control outcome at covariates \(X=x\) equals the conditional mean of \(Y(0)\), if conditional exchangeability holds and suitable controls are available. Consistency then lets the observed treated outcome represent \(Y(1)\). Averaging these conditional comparisons over the covariate distribution of retained treated units gives:

\begin{aligned} &E[Y\mid A=1,R_c=1]-E_X\{E[Y\mid A=0,X]\mid A=1,R_c=1\}\\ &=E[Y(1)\mid A=1,R_c=1]-E[Y(0)\mid A=1,R_c=1]\\ &=E[Y(1)-Y(0)\mid A=1,R_c=1]=\tau_c. \end{aligned}

The equality requires the assumptions just described and a comparison that adequately represents the relevant control outcomes for retained treated units. In a finite matched sample, a close match is an approximation to that population comparison, not proof that the equality holds exactly.

Worked Example: Calculating an ATT for Caliper-Retained Treated Units

Suppose a hypothetical dataset has four treated units and three controls. The table gives estimated propensity scores and observed outcomes. Use a raw-score caliper of \(c=0.10\), matching each eligible treated unit to its closest available control. A treated unit without an eligible control is excluded.

Unit	Treatment status	Estimated propensity score	Observed outcome
T1	Treated	0.20	12
T2	Treated	0.45	18
T3	Treated	0.70	25
T4	Treated	0.95	31
C1	Control	0.22	8
C2	Control	0.48	13
C3	Control	0.68	15

T1 matches C1 because \(|0.20-0.22|=0.02\le0.10\), giving an outcome difference of \(12-8=4\). T2 matches C2 because \(|0.45-0.48|=0.03\le0.10\), giving \(18-13=5\). T3 matches C3 because \(|0.70-0.68|=0.02\le0.10\), giving \(25-15=10\). T4 has no eligible control: its closest score is 0.68, at distance \(|0.95-0.68|=0.27>0.10\).

Three treated units are retained, so the matched estimate is:

\widehat{\tau}_c=\frac{(12-8)+(18-13)+(25-15)}{3} =\frac{4+5+10}{3}=\frac{19}{3}=6.333.

The estimate is 6.333 outcome units for the treated units retained by this caliper and matching rule, subject to the causal assumptions and the adequacy of the matches. It is not automatically an estimate for all four treated units: T4 is outside the matched target. Since this is hypothetical data, no real study or real treatment effect is being described.

How the Caliper Shifts the Target

The caliper controls a trade-off. A tighter threshold generally demands closer score matches, but may exclude more treated units. A looser threshold may retain more treated units while allowing less similar controls. Because treatment effects may differ across people, changing which treated units are retained can change the target effect even when the matching procedure is otherwise unchanged. The estimate can also change because the selected controls change.

Worked Example: A Tighter Caliper Produces a Different Local ATT

In a second hypothetical dataset, three treated units have estimated scores 0.30, 0.55, and 0.80, with observed outcomes 14, 22, and 35. Their respective closest controls have scores 0.31, 0.62, and 0.96, with outcomes 10, 15, and 21. Assume the stated closest-control matches are used.

With \(c=0.08\) on the raw score scale, the first two treated units qualify: their score distances are 0.01 and 0.07. The third does not qualify because its distance is 0.16. The estimate for these two retained treated units is:

\widehat{\tau}_{0.08}=\frac{(14-10)+(22-15)}{2} =\frac{4+7}{2}=5.5.

With \(c=0.20\), the third treated unit also qualifies. The estimate becomes:

\widehat{\tau}_{0.20}=\frac{(14-10)+(22-15)+(35-21)}{3} =\frac{4+7+14}{3}=\frac{25}{3}=8.333.

These two estimates target different retained treated populations: the first two units under the tighter caliper versus all three under the looser one. The numerical difference is not, by itself, evidence that either estimate is biased; it may reflect a real change in the population effect, differences in the matched comparisons, or both. Interpretation requires both the target definition and the assumptions to be considered.

This distinction connects caliper matching to other ways of restricting the target. A common-support restriction defines a population using a support rule; a caliper adds a matching-distance rule that may exclude treated units even when their scores lie within a broad region of overlap. Neither restriction should be described as estimating the full-population ATT unless the retained treated population is, in fact, the full treated population.

Diagnostics, Alternatives, and Limitations

Diagnostics should check whether the matching operation achieved its intended comparison and how much the target changed. Report the caliper scale and value, the number and proportion of treated units retained, and the distribution of treated units’ nearest-control distances. Examine covariate balance before and after matching, including important covariates and nonlinear terms where relevant. Standardized differences are useful for comparing covariate balance across variables on different scales. Also inspect whether any covariate remains poorly balanced despite close propensity scores.

Warning signs include many treated units excluded, very few controls near retained treated units, substantial residual covariate imbalance, or estimates that change markedly under reasonable caliper choices. These findings do not identify a single correct answer, but they call for caution: the match may represent a narrow subset, leave important confounding uncontrolled, or depend strongly on an arbitrary threshold. Unmeasured confounding remains a limitation even when observed balance is good.

The main alternatives answer somewhat different questions. Matching without a caliper may retain more treated units but can permit distant matches; the resulting comparisons may be less credible in regions with poor overlap. Restricting to common support can remove unsupported regions, but it need not impose the same pairwise distance requirement. Regression adjustment or weighting can use a different comparison and target, depending on how they are specified. These approaches are not automatically interchangeable: state the target population and assumptions before comparing their estimates.

A caliper choice should be prespecified when possible, justified by the score scale and design goals, and accompanied by sensitivity checks. There is no universal threshold that guarantees valid causal inference. A caliper is a design constraint, not a test of exchangeability and not a cure for poor covariate measurement, model misspecification, or lack of overlap.

A Common Mistake

Common pitfall: “The caliper estimate is an ATT, so it represents the effect for every treated person.” This conclusion is wrong when the caliper excludes treated units. The correction is to identify the retained treated population and describe the estimate as its local ATT. If the target is the ATT for all treated units, the excluded units’ treatment effects are not supplied by the matched estimate; additional assumptions and a method that addresses those units would be needed.

Worked Example: Why a Distant Match Is Not a Neutral Substitute

In a third hypothetical setting, treated unit A has score 0.40 and outcome 15, while treated unit B has score 0.90 and outcome 30. A close control for A has score 0.42 and outcome 10. The nearest available control for B has score 0.45 and outcome 12. With a raw-score caliper of 0.10, A qualifies at distance 0.02, while B does not qualify because its distance is 0.45.

For A, the matched difference is \(15-10=5\) outcome units. The caliper estimate for the retained treated group—here, A alone—is therefore 5 outcome units. If the caliper were ignored and B were compared with that distant control, B’s difference would be \(30-12=18\) outcome units. Averaging those two comparisons would give:

\frac{(15-10)+(30-12)}{2}=\frac{5+18}{2}=11.5.

That 11.5 calculation is not a caliper-matched estimate, and it does not estimate the same target as A’s local estimate. The distant comparison may reflect covariate differences as well as treatment effects. Nor can the change from 5 to 11.5 establish which number is closer to a true causal effect. The example shows why widening or ignoring a caliper changes both the comparison and, when treated units become eligible, the target population.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A treated unit has score 0.63. A control has score 0.71. Does the control meet a raw-score caliper of 0.10? Show the distance.
Three retained treated units have matched outcome differences of 2, 6, and 7 units. Calculate the simple matched estimate and state its target in words.
Why can a caliper-matched estimate differ from the ATT for all treated units even if every retained match is close?
Name two diagnostics that can reveal problems after caliper matching, and explain what each checks.
True or false: If treated and control units are close on estimated propensity scores, conditional exchangeability is proven. Explain.

Solutions. (1) Yes. The distance is \(|0.63-0.71|=0.08\), which is no greater than 0.10. (2) The estimate is \((2+6+7)/3=5\) outcome units. Its target is the average effect for the treated units retained by the stated matching rule, assuming the required causal conditions hold. (3) The caliper may exclude treated units; their effects can differ from the retained units’ effects, so the retained-group average need not equal the all-treated average. (4) Report the proportion of treated units retained to assess how much the target was restricted, and inspect post-match covariate balance to check whether observed differences remain. Match-distance summaries are another useful diagnostic. (5) False. Score closeness is an observed-design feature; exchangeability concerns potential outcomes and cannot be proven by matching or balance diagnostics.