Why a Caliper Can Change the Target
A caliper limits how far apart a treated unit and a control unit may be on a chosen matching score, such as the propensity score. The restriction can prevent poor matches, but it can also leave some treated units unmatched. If those treated units differ in their treatment effects from the retained treated units, the effect targeted after matching need not be the original average treatment effect on the treated (ATT).
The key is to distinguish two questions. First, which treated units remain eligible for the matched analysis? Second, what is the average treatment effect for those units? The first is determined by the matching rule and available controls. The second is a causal estimand defined by the retained treated population. Neither question is answered just by reporting the number of matched pairs.
In a finite data set, the corresponding averages use the treated units in each group. The retention indicator depends on the specified matching algorithm as well as the caliper. For example, under matching without replacement, a control assigned to one treated unit may no longer be available to another. In the examples below, matching is with replacement: a control may be reused, so a treated unit is retained exactly when it has a control within the stated caliper.
Assumptions and Interpretation
The numerical examples use potential outcomes \(Y_i(1)\) and \(Y_i(0)\), the outcomes unit \(i\) would have under treatment and control. The individual treatment effect is \(Y_i(1)-Y_i(0)\). In real data, only one of these two outcomes is observed for each unit, so the average treatment effect among retained treated units must be estimated rather than calculated directly from both potential outcomes.
A causal interpretation of a matched estimate requires assumptions about treatment assignment and outcomes, not merely a successful match. Consistency means the observed outcome equals the potential outcome under the treatment actually received. No interference means one unit’s outcome is not changed by other units’ treatment assignments. Conditional exchangeability requires \((Y(1),Y(0))\) to be independent of treatment given the measured pretreatment covariates \(X\). For a treated-population effect to be identified from controls, there must also be controls with relevant covariate support for the treated units being targeted. A caliper can restrict the target to retained treated units, but it does not establish exchangeability, eliminate unmeasured confounding, or make the propensity score a sufficient adjustment set by itself.
To describe a retained target, report which treated units were excluded and how the retained units differ in relevant pretreatment characteristics. The retained population is not automatically “the overlap population” or the full treated population: it is the treated population selected by the matching rule in the data at hand. Its definition should be tied to the actual score, caliper, and matching algorithm.
A Decomposition of the ATT Shift
The relationship between the original ATT and the retained-treated effect follows from splitting treated units into retained and excluded groups. Let \(q=P(R=1\mid A=1)\), let \(\tau_R=E[Y(1)-Y(0)\mid A=1,R=1]\), and let \(\tau_E=E[Y(1)-Y(0)\mid A=1,R=0]\). Provided both groups are nonempty, the original ATT is their weighted average:
Subtracting this expression from \(\tau_R\) gives the target shift:
Thus, excluding treated units changes the target effect when the retained and excluded groups have different average effects. If their effects are equal, the target shift is zero even though some units were removed. The direction is not determined by the caliper alone: it depends on how treatment effects differ between the retained and excluded treated units.
Worked Example: Finding the Retained Treated ATT
Suppose five treated units have the following propensity scores and potential outcomes. A control can be reused, and the caliper is \(c=0.10\) on the absolute propensity-score difference. The control scores are \(0.18, 0.31, 0.48,\) and \(0.63\).
| Treated unit | Propensity score | \(Y(1)\) | \(Y(0)\) | Individual effect |
|---|---|---|---|---|
| T1 | 0.20 | 12 | 8 | 4 |
| T2 | 0.30 | 15 | 10 | 5 |
| T3 | 0.45 | 20 | 12 | 8 |
| T4 | 0.60 | 24 | 14 | 10 |
| T5 | 0.80 | 27 | 15 | 12 |
For T1, the nearest listed control score is \(0.18\), giving distance \(|0.20-0.18|=0.02\), so T1 is retained. T2 has distance \(|0.30-0.31|=0.01\); T3 has distance \(|0.45-0.48|=0.03\); and T4 has distance \(|0.60-0.63|=0.03\). All are within the \(0.10\) caliper. T5 has nearest-control distance \(|0.80-0.63|=0.17\), which exceeds \(0.10\), so T5 is excluded.
The original ATT, calculated from all five treated units’ effects, is
The retained-treated effect is the average among T1 through T4:
The caliper-induced target shift is \(6.75-7.8=-1.05\) outcome units. This is also obtained from the decomposition: \(q=4/5=0.8\), and the excluded group’s effect is \(\tau_E=12\), so
In this hypothetical example, the retained treated population has a lower average effect than the full treated population because the excluded unit has a larger effect. The calculation describes a change in the target, not a claim that caliper matching necessarily lowers an estimate in other data.
Three Things to Keep Separate
A caliper-based analysis involves the retention rule, the causal target, and the estimator. They are related but not interchangeable. The retention rule tells which treated units can contribute under the matching procedure. The causal target is the average effect for those retained treated units. The estimator uses observed outcomes from matched treated and control units to estimate that target. Even if the retained target is well defined, the estimator can differ from it because matched controls are not the treated units’ unobserved counterfactual outcomes.
In a simple one-to-one matched analysis, an estimator may average treated-minus-control outcome differences across retained pairs. That sample average is not, by definition, the true retained-treated effect. Its causal interpretation depends on the assumptions above and on adequate matching of covariates relevant to the outcome. Matching only on a propensity score distance does not ensure that each pair has identical covariates or identical untreated potential outcomes.
Worked Example: Target Shift Versus Matching Error
Consider four treated units with effects \(4,6,8,\) and \(12\). A caliper retains the first three and excludes the fourth. The original ATT is
The target shift is \(6-7.5=-1.5\) outcome units. Now suppose a one-to-one match for the three retained treated units produces observed outcome differences of \(5,6,\) and \(8\). The matched estimate is
The difference between this estimate and the retained target is approximately \(6.333-6=0.333\) units. In an actual study, the true retained target is unknown; this hypothetical example makes the potential outcomes available to separate the two quantities. The shift from \(7.5\) to \(6\) comes from changing the treated population. The difference between \(6.333\) and \(6\) represents estimation or matching discrepancy in this constructed data. Reporting only the matched estimate would conceal that conceptual distinction.
Diagnosing the Retained Population
A useful diagnostic begins with an explicit account of the treated units removed. At minimum, report their number and proportion, the matching-score range for retained and excluded treated units, and the covariate characteristics that matter to the scientific question. Compare the retained treated group with the original treated group, not only the matched treated and control groups. Good balance between matched groups does not show that the retained treated units represent all treated units.
The matching process can also fail operationally. A unit may have a control within the nominal caliper but still be unmatched if a no-replacement algorithm has already used every eligible control. Conversely, a unit with no control within the caliper cannot be matched under the stated rule. The report should therefore define retention by the matching procedure that was actually used, rather than describing it only as “within caliper” when other assignment constraints apply.
A caliper estimate has a limited scope. It concerns retained treated units, under the causal assumptions and matching design used. It does not by itself recover the original ATT, the ATE, or an effect for treated units excluded by the caliper. Extending the estimate to a broader population requires additional assumptions and a method designed for that target.
Worked Example: Checking the Direction of the Shift
Suppose four treated units have individual effects \(2,4,10,\) and \(12\). A stated caliper retains the first two and excludes the last two. The retained effect is
The excluded-group effect is \(\tau_E=(10+12)/2=11\), while the original ATT is \((2+4+10+12)/4=7\). Here \(q=2/4=0.5\), so the decomposition gives
Direct calculation agrees: \(3-7=-4\). Unlike the first example, the target shift is larger in magnitude because half the treated units were excluded. The direction is downward because their average effect is greater than that of the retained units. This conclusion follows from the stated effects, not from a general property of calipers.
Alternatives and Common Pitfalls
Compared with matching without a caliper, caliper matching can rule out especially distant controls, at the cost of excluding treated units or changing which controls can be assigned. A different caliper or matching ratio may produce a different retained population; those choices should be treated as design decisions whose consequences are reported, not as ways to guarantee a desired effect. Weighting approaches can target a population through explicit weights rather than through this same retention rule, while outcome-model extrapolation may include units without close matches but relies more heavily on its modeling assumptions. These approaches do not automatically share the caliper-matched estimand.
- Mistake: “The matched effect estimates the original ATT because the treated units were the same kind of people.” If treated units were excluded, the matched treated group is a subset. The estimate targets that subset unless additional assumptions and procedures justify broader generalization.
- Mistake: “The caliper changed the effect.” The caliper changes which treated units define the target. It does not alter any individual unit’s potential outcomes; a different average can result because the composition of the group changed.
- Mistake: “A small matched estimate proves the treatment has a small effect for everyone treated.” The estimate applies to the retained treated population under the stated assumptions. It need not describe excluded treated units.
- Mistake: “All matched pairs are close, so selection into the retained group is irrelevant.” Pairwise closeness addresses a different issue from representativeness of retained treated units relative to all treated units.
Check Your Understanding
Answer each question, then compare your work with the solutions below.
- A caliper retains 6 of 10 treated units. What is the retained fraction \(q\), and what percentage of treated units were excluded?
- The retained group has average effect \(5\), and the excluded group has average effect \(9\). If \(q=0.75\), calculate the original ATT and the retained-minus-original target shift.
- Why does good balance between matched treated and control units not establish that the matched estimate targets the original ATT?
- Under a with-replacement rule, a treated unit’s nearest control is \(0.08\) propensity-score units away. Is it eligible under a caliper of \(0.05\)? What if the distance is exactly \(0.05\), assuming the rule allows distances no greater than the caliper?
Solutions. (1) \(q=6/10=0.6\); \(4/10=40\%\) of treated units were excluded. (2) The original ATT is \(0.75(5)+0.25(9)=3.75+2.25=6\). The target shift is \(5-6=-1\), also \((1-0.75)(5-9)=0.25(-4)=-1\). (3) Balance compares the matched treated and control groups; it does not show that retained treated units have the same effect distribution as all treated units. Excluded treated units may have different effects, so the matched target can differ from the original ATT. (4) A distance of \(0.08\) exceeds \(0.05\), so the unit is not eligible. A distance exactly \(0.05\) is eligible under the stated inclusive rule.