Why Caliper Choice Is a Design Decision
A caliper sets the largest allowed distance between a treated unit and a comparison unit when matching on an estimated propensity score. A narrow caliper can prevent poor matches, but may leave treated units unmatched. A wider caliper can retain more treated units while accepting more distant matches. The choice therefore changes both the quality of the matched comparisons and the population represented by the analysis.
The caliper should be chosen using covariates, propensity-score overlap, and a prespecified retention rule—not by whichever choice produces the most favorable outcome estimate. The average treatment effect on the treated (ATT) is the average treatment effect among treated units. When matching discards treated units, however, the direct matched estimate pertains to the retained treated units; it need not equal the ATT for all originally treated units.
For the worked study, matching is one-to-one and without replacement: each retained treated unit is paired with a distinct comparison unit. The propensity-score model and candidate calipers are fixed before examining outcomes. Assume consistency, conditional exchangeability given the measured baseline covariates, and positivity for the population targeted by the analysis. These assumptions are needed for a causal interpretation; matching does not establish them. In particular, a caliper cannot control confounding by unmeasured covariates.
The examples use two prespecified calipers, 0.10 and 0.20 standard deviations of the logit propensity score. These values illustrate a comparison, not universal recommendations. The matching algorithm, caliper scale, tie-breaking rule, covariates, and criteria for acceptable balance and retention should all be documented.
Worked Study: Balance and Retention
Suppose a hypothetical cohort contains 8 treated patients and 12 eligible comparison patients. Baseline age and a risk score are measured before treatment. The risk score is a continuous baseline covariate, with larger values indicating greater predicted health risk. The same fitted propensity-score model and nearest-neighbor algorithm are used for both calipers. The matched pairs under the narrower caliper are also retained under the wider one.
Worked Example: Comparing Matched-Sample Balance
The 0.10 caliper retains 6 of the 8 treated patients. The 0.20 caliper retains all 8. The following are the matched-group means and sample variances; variances are reported in squared measurement units.
| Caliper | Covariate | Treated mean | Comparison mean | Treated variance | Comparison variance |
|---|---|---|---|---|---|
| 0.10 | Age (years) | 55.00 | 55.67 | 14.00 | 27.07 |
| 0.10 | Risk score | 0.42 | 0.45 | 0.0144 | 0.0169 |
| 0.20 | Age (years) | 56.00 | 54.50 | 16.00 | 20.25 |
| 0.20 | Risk score | 0.44 | 0.49 | 0.0196 | 0.0225 |
For a covariate \(X\), calculate the standardized mean difference (SMD) as the treated-minus-comparison mean difference divided by the square root of the average of the two sample variances:
For age under the 0.10 caliper, substitution gives
The negative sign indicates that the retained treated group is slightly younger on average. The absolute SMD is 0.1479, and its units cancel: it is measured in standard-deviation units, not years. For the risk score under that caliper,
Repeating the calculation for the 0.20 caliper gives
For both measured covariates, the narrower caliper has a smaller absolute SMD, although neither caliper brings both absolute SMDs below 0.10. The value 0.10 is a commonly used balance reference, not a significance test or a guarantee of negligible confounding. Balance should be considered across all important baseline covariates, not just these two.
Retention and Propensity-Score Overlap
Retention is calculated relative to the original treated group. The 0.10 caliper retains \(6/8=0.75\), or 75%, of treated patients; the 0.20 caliper retains \(8/8=1.00\), or 100%. These proportions describe how many treated patients remain eligible for the direct matched comparison. They do not by themselves establish that the retained patients represent all treated patients.
Worked Example: Reading Match Distances and Retention
Suppose the absolute standardized logit-propensity-score distances for the six closest pairs are 0.02, 0.04, 0.05, 0.06, 0.07, and 0.08. The two additional pairs available under the wider caliper have distances 0.14 and 0.18.
With a 0.10 caliper, all six listed close pairs qualify because each distance is at most 0.10. The other two do not qualify because \(0.14>0.10\) and \(0.18>0.10\). Thus the caliper retains 6 treated patients, or 75%. With a 0.20 caliper, all eight listed pairs qualify, so retention is 100%. The maximum observed pair distance is 0.08 for the narrower design and 0.18 for the wider design.
This calculation checks that the stated matches obey the caliper. A useful overlap diagnostic is to inspect the distribution of eligible distances and the propensity-score ranges of retained and discarded treated units. A large gap in distances between accepted and rejected matches may make the retention change easy to explain. Conversely, many matches piled close to the cutoff can make results sensitive to small changes in the propensity-score model or caliper.
A high retention percentage is not automatically preferable. The extra two treated patients in the wider design have more distant available comparisons, and the balance calculations show larger absolute SMDs for both measured covariates. If the omitted treated patients differ systematically from those retained, the narrower-caliper estimate also targets a more limited treated population. Report both the retention and the covariate profile of the treated patients who were excluded.
ATT Estimates After the Design Is Fixed
For one-to-one matching without replacement, a simple matched outcome estimator is the average of the treated-minus-comparison outcome differences across retained pairs. It estimates an ATT for the retained treated units under the causal assumptions and adequate control of confounding. If some treated units are discarded, extending that interpretation to all originally treated patients requires additional justification.
Worked Example: Calculating Both Matched Outcome Estimates
Assume the outcome is blood-pressure reduction, measured in mmHg, with larger reductions being better. For the six pairs retained by the 0.10 caliper, suppose the treated-minus-comparison differences are 2, 1, 3, 0, 2, and 1 mmHg. The estimated average difference is
The estimate is an average difference in blood-pressure reduction for the six retained treated patients, relative to their matched comparisons. It is not, solely from this calculation, evidence that treatment caused a 1.50 mmHg greater reduction: that causal interpretation depends on the design assumptions and the quality of measured-covariate adjustment.
Suppose the two additional pairs available under the 0.20 caliper have differences of 8 and \(-4\) mmHg. The wider-caliper estimate is
The estimate changes from 1.50 to 1.625 mmHg, but this difference is not a reason to choose the wider caliper. The estimate includes different treated patients and different matches. The design choice should be made from prespecified balance, overlap, and retention criteria; then outcome estimates can be reported for each design as a sensitivity analysis.
A Transparent Caliper-Selection Rule
There is no single caliper that is optimal for every cohort. A useful choice is an explicit rule that acknowledges tradeoffs before outcomes are examined. For this example, suppose investigators prespecify that the preferred candidate should (1) retain at least 75% of treated patients and (2) have smaller absolute SMDs than the competing candidate on each of the two key baseline covariates. Under that rule, both candidates satisfy the retention condition, but the 0.10 caliper has smaller absolute age and risk-score SMDs. It is therefore the preferred candidate under this stated rule.
That choice does not mean balance is perfect. Both absolute SMDs under the narrower caliper exceed 0.10, so investigators using 0.10 as their required balance threshold would conclude that neither candidate is satisfactory on these covariates. The appropriate response may be to revisit the design—such as the propensity-score model or the feasible comparison pool—rather than to declare a caliper acceptable because it is the better of two inadequate options. Any revised design should be assessed without using outcome results to select it.
The standardized mean difference is useful because it summarizes the size of a mean difference relative to covariate variability, and does not depend on the measurement units. Its sign indicates direction; its magnitude is used to assess imbalance. It is not a p-value: a small SMD does not prove the covariate distributions are identical, and a larger SMD is not evidence from a hypothesis test that the groups differ. Also inspect distributions when important imbalances may be hidden by similar means, for example nonlinearities or extreme values.
Diagnostics, Alternatives, and Common Mistakes
Before interpreting matched outcome estimates, check that the reported matches satisfy the caliper, that treated and comparison units are not reused when matching without replacement, and that the retained sample sizes agree with the cohort flow. Review absolute SMDs for all prespecified baseline covariates and inspect whether discarded treated units occupy a region of the propensity-score distribution with few eligible comparisons. Poor balance, low retention, or a small set of unusually distant matches may signal that the proposed matched design is not credible for the intended target population.
A very narrow caliper can improve match proximity but discard many treated units, reducing precision and narrowing the target population. A wider caliper can increase retention but permit poorer matches. Matching without a caliper avoids discarding units solely for exceeding a distance threshold, but may create very poor pairs when overlap is weak. Other approaches, such as propensity-score weighting or subclassification, change how comparisons are constructed and may target different populations; they are alternatives to consider, not automatic remedies for poor overlap. The appropriate method depends on the estimand and available covariate overlap.
- Choosing the caliper with the most favorable outcome estimate. This is outcome-guided design selection and can make reported results selectively favorable. Specify design criteria first; report outcome results after the design is settled.
- Reporting retention without naming its denominator. State the number of treated units retained out of the original number, and identify the population represented by the matched estimate.
- Treating a caliper as a balance guarantee. The distance restriction limits pairwise propensity-score separation; it does not ensure balance on every covariate. Calculate balance diagnostics after matching.
- Calling a retained-sample estimate the full-cohort ATT without qualification. If treated units are discarded, the direct estimate applies to retained treated units. Generalizing to all treated units requires assumptions beyond the matching calculation.
A defensible report gives the propensity-score distance scale, candidate calipers, matching algorithm, number and percentage of treated units retained, balance summaries, and the prespecified reason for choosing the design. It should also make clear when no candidate achieves the desired balance. This separates design evidence from outcome evidence and makes the consequences of caliper choice visible.
Check Your Understanding
Answer each question using the definitions and calculations above.
- A study starts with 20 treated units, and a caliper retains 15. What is the retention proportion, and what should be specified alongside it?
- For a covariate, the treated mean is 10, the comparison mean is 12, and both sample variances are 4. Calculate the SMD and describe its sign.
- Why should the caliper not be selected because it gives the larger estimated treatment effect?
- In the worked study, what population does the 0.10-caliper outcome estimate directly describe?
- What conclusion is justified if a prespecified balance threshold is not met by either candidate caliper?
Solutions. (1) Retention is \(15/20=0.75\), or 75%; also report that 15 of the original 20 treated units were retained and describe the retained versus excluded treated units. (2) The pooled variance is \((4+4)/2=4\), so the denominator is \(\sqrt{4}=2\). The SMD is \((10-12)/2=-1\); the negative sign means the treated mean is lower. (3) Choosing by the outcome estimate uses outcome information to select the design and risks selective reporting; use prespecified balance, overlap, and retention criteria. (4) It directly describes the six treated patients retained by that caliper, subject to the causal assumptions, not automatically all eight treated patients. (5) Neither candidate meets the stated design requirement; report that limitation and consider revising the design rather than claiming the better-performing candidate is adequately balanced.