Why Use a Caliper?
Nearest-neighbor matching pairs a treated unit with a comparison unit whose estimated propensity score is close. Without a distance limit, however, a treated unit can be paired with a comparison unit that is still quite far away. A caliper sets a maximum permitted distance: if no eligible comparison unit is close enough, the treated unit remains unmatched. This can reduce reliance on poor matches, at the cost of excluding treated units and changing which population the estimate describes.
The focus here is 1:1 matching without replacement: each retained treated unit is paired with one comparison unit, and a comparison unit cannot be used in another pair. The procedure estimates an ATT for the treated units that are successfully matched. It does not automatically estimate the ATT for every treated unit in the original sample.
Assumptions and the Estimand
Matching is a design step, not a guarantee that treatment assignment is unconfounded. A causal interpretation requires, at minimum, consistency (the observed outcome corresponds to the treatment received), no interference between units, and conditional exchangeability: given the measured pre-treatment covariates, treatment assignment is independent of the potential outcomes. For an ATT, there must also be comparison units with a positive probability of being untreated for the covariate profiles of the treated units whose effects are targeted. This is the relevant overlap or positivity condition.
The propensity-score model should use appropriate pre-treatment covariates and estimate treatment probabilities adequately. Under the standard propensity-score result, conditioning on the true score balances the measured covariates used to construct it. In finite samples, a close score distance does not ensure that every covariate is balanced, so balance must be checked directly. These conditions cannot rule out confounding by unmeasured variables.
If some treated units have no comparison unit within the caliper, the matched estimate pertains to the retained treated units. Extending it to all treated units requires additional assumptions about the excluded units’ effects. The target population and exclusions should therefore be reported explicitly.
Worked Example: Translating a Caliper Rule Into a Distance
Suppose an analysis prespecifies a caliper equal to 0.2 times the standard deviation of the estimated logit propensity scores in the analysis sample. In a hypothetical sample, that standard deviation is 1.50 logit units. What is the maximum permitted distance?
The rule is \(c=0.2s_{\operatorname{logit}(\hat e)}\). Substituting the stated value gives \(c=0.2(1.50)=0.300\) logit units. A treated-comparison pair is eligible when its absolute logit-score difference is at most 0.300.
This is a common starting convention, not a universal law or a guarantee of good balance. The score scale matters: a caliper of 0.300 on the logit scale is not the same as a raw propensity-score difference of 0.300. The caliper definition, scale, and chosen value should be stated before outcomes are compared.
Applying the Caliper and Checking Balance
A transparent matching procedure specifies the propensity-score model, the distance scale, the caliper, whether comparison units may be reused, and how ties are handled. In greedy nearest-neighbor matching, the order in which treated units are processed can affect the final pairing when comparison units cannot be reused. The example below processes treated units in the listed order and uses the nearest available comparison unit. A different order or assignment rule can produce different pairs.
Worked Example: Matching Treated Units and Identifying an Exclusion
Consider hypothetical estimated logit propensity scores for four treated units and five comparison units. Use 1:1 nearest-neighbor matching without replacement, in the order T1 through T4, with caliper \(c=0.30\).
| Unit | Group | Logit propensity score | Covariate X |
|---|---|---|---|
| T1 | Treated | -1.00 | 10 |
| T2 | Treated | -0.40 | 12 |
| T3 | Treated | 0.20 | 15 |
| T4 | Treated | 0.80 | — |
| C1 | Comparison | -1.05 | 9 |
| C2 | Comparison | -0.55 | 13 |
| C3 | Comparison | 0.05 | 14 |
| C4 | Comparison | 0.45 | — |
| C5 | Comparison | 1.60 | — |
For T1, the closest comparison is C1: \(|-1.00-(-1.05)|=0.05\), so pair T1–C1. For T2, C2 is the nearest available comparison: \(|-0.40-(-0.55)|=0.15\), so pair T2–C2. For T3, the closest available comparison is C3: \(|0.20-0.05|=0.15\), so pair T3–C3. T4’s closest remaining comparison is C4, but \(|0.80-0.45|=0.35>0.30\). Thus T4 is unmatched. C5 is farther still. There are three matched pairs and one excluded treated unit.
For the measured covariate X, the matched treated mean is \((10+12+15)/3=12.33\), and the matched comparison mean is \((9+13+14)/3=12.00\). The absolute mean difference is \(0.333\) units of X. If the prespecified reference pooled standard deviation is 2.58, the standardized mean difference is \(0.333/2.58=0.129\). This indicates a modest standardized difference for this covariate, not proof of overall balance. Repeat diagnostics for each important covariate and inspect distributions; a small difference in means can conceal differences in spread or tails.
A useful balance assessment compares covariates before and after matching. Standardized mean differences express mean differences in standard-deviation units, making them less dependent on sample size than significance tests. They are diagnostics rather than pass-or-fail tests with a universal cutoff. Also inspect propensity-score overlap, covariate distributions, and the number and characteristics of treated units excluded. A match can have close propensity scores and still show residual imbalance in an important covariate.
Estimating the Matched ATT and Its Uncertainty
For \(M\) retained treated units, let \(Y_{Ti}\) be the outcome for treated unit \(i\), and \(Y_{C(i)}\) the outcome for its matched comparison. The matched ATT estimate is the mean of the \(M\) pair differences. If outcomes are continuous, units of the estimate are the same as the outcome units.
One simple uncertainty interval uses a paired t procedure. Define \(D_i=Y_{Ti}-Y_{C(i)}\), let \(\bar D\) be the mean of the pair differences, and let \(s_D\) be their sample standard deviation. If the pair differences are independent across pairs and approximately normally distributed (or the number of pairs is large enough for a suitable approximation), the standard error is \(s_D/\sqrt{M}\). A two-sided \(100(1-\alpha)\%\) interval is \(\bar D\pm t_{1-\alpha/2,M-1}s_D/\sqrt M\). With very few pairs, the normality assumption is especially consequential.
Worked Example: A Matched ATT and Paired t Interval
For the three matched pairs in a hypothetical study, suppose the outcome is hospital stay in days. The treated outcomes are 8, 10, and 7 days for T1, T2, and T3. The matched comparison outcomes are 5, 7, and 6 days for C1, C2, and C3. The pair differences are \(D=(3,3,1)\) days.
The point estimate is \(\bar D=(3+3+1)/3=7/3=2.333\) days. The sample standard deviation of the differences is \(s_D=\sqrt{\{(3-7/3)^2+(3-7/3)^2+(1-7/3)^2\}/(3-1)}=\sqrt{4/3}=1.155\) days. Therefore, \(SE=s_D/\sqrt{3}=1.155/1.732=0.667\) days.
For a 95% paired t interval, \(df=M-1=2\) and \(t_{0.975,2}=4.303\). The interval is \(2.333\pm4.303(0.667)=2.333\pm2.869\), or approximately \((-0.536,\ 5.202)\) days. Under the paired t model, this interval is compatible with effects from about 0.536 fewer days to 5.202 more days of hospital stay for the retained treated population. It is wide, and it includes zero; the estimate alone does not establish a nonzero effect.
This interval treats the formed pairs as the analysis units and relies on the stated pair-difference assumptions. In this simple calculation it does not account for uncertainty from estimating the propensity scores, choosing the matching design, or trying alternative specifications. The interval also does not repair unmeasured confounding.
Diagnostics, Alternatives, and Limitations
Caliper matching makes a deliberate trade-off: a tighter caliper can avoid distant matches but may leave more treated units unmatched; a wider caliper can retain more treated units while allowing less comparable pairs. The caliper is not chosen by looking for the outcome result that appears most favorable. Prespecify a defensible rule, examine balance and retention, and report sensitivity to reasonable design choices without selectively presenting only favorable results.
Compare caliper matching with the main alternatives in terms of their design choices. Nearest-neighbor matching without a caliper can retain treated units for whom only distant comparisons exist. Regression adjustment models outcomes conditional on covariates rather than enforcing a maximum score distance. Weighting uses propensity-score-based weights rather than forming one-to-one pairs. Each approach has its own assumptions and failure modes; none makes unmeasured confounding disappear. The earlier material on propensity-score matching establishes the role of the score as a balancing device, while the relevant design should still be judged by observed covariate balance and overlap.
A common mistake is to say, “Every treated unit was matched to its nearest control, so the full treated-group ATT has been estimated.” In the example, T4 has no eligible comparison unit and is excluded. The computed mean difference therefore describes the three retained treated units, not automatically all four. Another mistake is to infer adequate balance from similar propensity scores alone. The correction is to report the retained population and assess balance on the original measured covariates after matching.
- Warning signs: many treated units fail the caliper, few comparison units are available in parts of the treated score range, important covariates remain imbalanced, or the result changes substantially under modest design changes.
- Limitations: matching can reduce precision and alter the target population; close scores do not ensure exact covariate balance; the analysis remains vulnerable to unmeasured confounding.
- Reporting essentials: state the caliper scale and value, matching ratio and replacement rule, matching order or assignment rule, number of retained and excluded treated units, post-match balance, outcome contrast, and uncertainty method.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A logit-score standard deviation is 2.00 and the selected caliper is 0.2 standard deviations. What is the caliper in logit units?
- A treated unit’s closest available comparison is 0.42 logit units away. Under a caliper of 0.35, is that unit matched?
- Three matched pair differences are 2, 0, and 4 outcome units. Find the matched ATT estimate.
- Why does excluding treated units without eligible matches change the interpretation of the ATT?
Solutions: (1) \(c=0.2(2.00)=0.400\) logit units. (2) No: \(0.42>0.35\), so that treated unit is unmatched under the stated rule. (3) The estimate is \((2+0+4)/3=2.00\) outcome units. (4) The average is calculated over retained treated units; excluded treated units contribute no pair difference, so the result does not automatically represent the ATT for all treated units.