Sensitivity of the Target to Caliper Width

Why Caliper Width Can Change the Target

A caliper limits how far apart matched units may be on a chosen matching scale, often the propensity score or its logit. Tightening the caliper can prevent poor matches, but it may also exclude units that have few close counterparts. The resulting estimate can therefore concern a different group of units, not merely estimate the same effect with less precision. Loosening the caliper may retain more units, yet those additional units can have different treatment effects or covariate profiles.

The key is to track two distinct consequences of changing the width: which units and weights define the target, and how much sampling uncertainty is attached to the estimate. A standard error describes uncertainty for an estimator under its design and assumptions; it does not tell us whether two calipers target the same population effect.

Definition: Let $c$ denote a caliper width, and let $R(c)$ be the units retained for the target after matching with that width. Let $a_i(c)\geq 0$ be the target weight assigned to retained unit $i$, with positive total weight. The caliper-specific target effect is $$ \tau(c)=\frac{\sum_{i\in R(c)}a_i(c)\{Y_i(1)-Y_i(0)\}}{\sum_{i\in R(c)}a_i(c)}. $$ Here $Y_i(1)$ and $Y_i(0)$ are unit $i$'s potential outcomes under treatment and control. The weights and retained units specify the population summary being targeted; they are not, by themselves, a guarantee that the effect is identified from observed data.

For example, in a design targeting retained treated units, $R(c)$ may consist of the treated units that can be matched under width $c$, with weights reflecting the matched-set design. The resulting target is an effect for those retained treated units, not automatically the ATT for all treated units. Other matching designs can induce other retained populations and weights.

Causal interpretation requires the usual potential-outcome conditions appropriate to the design. These include consistency (the observed outcome equals the potential outcome under the treatment actually received), no interference between units as assumed by the analysis, and conditional exchangeability of treatment assignment given the measured covariates used to control confounding. The relevant treatment and comparison units must also have support for the comparisons being made. A caliper can restrict comparisons to closer matches, but it cannot ensure that unmeasured confounding is absent or repair a misspecified propensity-score model. These assumptions support identification of a target; they do not make targets at different widths equal.

When a Wider Caliper Adds Units

Suppose increasing the width leaves all previously retained units in the design and simply adds a group of newly retained units. If the old units' target weights do not change, the wider target is a weighted average of the old target and the average effect for the additions. This identity shows exactly why the target can shift.

\tau_{\mathrm{wide}} = \frac{W_{\mathrm{old}}\tau_{\mathrm{old}}+ W_{\mathrm{new}}\tau_{\mathrm{new}}} {W_{\mathrm{old}}+W_{\mathrm{new}}}

Here $W_{\mathrm{old}}$ and $W_{\mathrm{new}}$ are the sums of target weights for the old and newly added units, and each group average uses those same weights. Subtracting $\tau_{\mathrm{old}}$ gives

\tau_{\mathrm{wide}}-\tau_{\mathrm{old}} = \frac{W_{\mathrm{new}}}{W_{\mathrm{old}}+W_{\mathrm{new}}} (\tau_{\mathrm{new}}-\tau_{\mathrm{old}}).

Thus, with unchanged old weights, the shift depends on both the weight share of the additions and the difference between their average effect and the old target. Adding many units does not necessarily move the target much if their average effect is similar; even a small added group can matter if its effects differ substantially.

This decomposition needs care. If widening the caliper changes the weights of previously retained units, the old-group average in the decomposition must be recalculated using the weights they have at the wider width. Simply carrying forward the narrow-width target and replacing counts by wider-width weight totals is generally incorrect. If the matching algorithm changes pairings or drops previously retained units, the retained sets may not be nested at all, so the simple “old plus additions” identity does not apply.

Worked Example: A Nested Set With Fixed Weights

Consider a hypothetical matching design targeting treated units. At a narrow width, treated units A and B are retained, each with weight 1. Their hypothetical individual treatment effects are 2 and 4 outcome points. At a wider width, unit C is also retained, with weight 1 and effect 10 points. Assume the design leaves A and B's weights unchanged.

At the narrow width, the target is $$ \tau_{\mathrm{narrow}}=\frac{1(2)+1(4)}{1+1}=\frac{6}{2}=3 $$ outcome points.

At the wider width, the target becomes $$ \tau_{\mathrm{wide}}=\frac{1(2)+1(4)+1(10)}{1+1+1} =\frac{16}{3}=5.333 $$ outcome points.

The shift is $5.333-3=2.333$ points. The decomposition gives the same result: the added unit has weight share $1/3$, and its effect differs from the old target by $10-3=7$ points, so the shift is $(1/3)(7)=2.333$ points. These hypothetical effects are supplied to illustrate the target arithmetic; in an actual study, both potential outcomes for each unit are not observed.

When Widening Also Reweights Old Units

Some matching procedures change the weights of units that were already retained when the caliper changes. For instance, a newly permitted match may alter matched-set sizes or allocation of control weights. In that case, calculate the old group's average using its wider-caliper weights before separating the effect of the new units.

Worked Example: Correcting for Reweighting

Suppose two old units have hypothetical effects 0 and 10 points. At the narrow width, they have equal weights, so the narrow target is $$ \tau_{\mathrm{narrow}}=\frac{1(0)+1(10)}{1+1}=5 $$ points.

At the wider width, their weights change to 9 and 1, and a new unit with effect 0 and weight 1 is added. The old-group average under the wider weights is $$ \tau_{\mathrm{old,wide}}=\frac{9(0)+1(10)}{9+1}=1 $$ point. The full wider target is therefore $$ \tau_{\mathrm{wide}}=\frac{9(0)+1(10)+1(0)}{9+1+1} =\frac{10}{11}=0.9091 $$ points.

Using the valid decomposition, the reweighted old group has total weight 10 and the new group has weight 1, so $$ \tau_{\mathrm{wide}}=\frac{10(1)+1(0)}{11}=0.9091 $$ points. It would be wrong to use the narrow target of 5 as the old-group average: that would give $\{10(5)+1(0)\}/11=4.545$, which is not the wider target. The discrepancy arises because the old units' relative weights changed, even before accounting for the added unit.

More generally, if the old and new groups are well-defined but old weights change, let $\tau_{\mathrm{old,wide}}$ denote the old units' average under their wider-caliper weights. Then the total shift can be written as a reweighting component plus an additions component:

\tau_{\mathrm{wide}}-\tau_{\mathrm{narrow}} = \frac{W_{\mathrm{old}}}{W_{\mathrm{old}}+W_{\mathrm{new}}} (\tau_{\mathrm{old,wide}}-\tau_{\mathrm{narrow}}) + \frac{W_{\mathrm{new}}}{W_{\mathrm{old}}+W_{\mathrm{new}}} (\tau_{\mathrm{new}}-\tau_{\mathrm{narrow}}).

This follows by writing the wider target as the weighted average of the reweighted old-group mean and the new-group mean, then subtracting the narrow target. The first term captures the effect of reweighting old units and the second captures the contribution of new units. If matching changes the identities of retained units, use the direct definition of each width-specific target rather than forcing this particular partition.

Target Shifts Are Not Precision Changes

An estimate at width $c$, written $\hat{\tau}(c)$, is calculated from observed outcomes and the matching design. Its standard error $SE\{\hat{\tau}(c)\}$ describes estimated sampling variability under the analysis. Tightening a caliper often excludes units, which can reduce the amount of information and increase the standard error; however, the standard error need not move monotonically because matching structure and weights can also change.

A change in $\hat{\tau}(c)$ across widths can reflect a change in the target, sampling variation, or both. Conversely, similar estimates do not prove identical targets: different unit compositions can have similar average effects. A useful sensitivity analysis therefore reports both the effect estimate and how the retained population and weights change. It should not interpret a narrower confidence interval as evidence that a caliper is more causally credible.

Worked Example: Comparing Estimates and Standard Errors

Suppose a hypothetical analysis gives a narrow-caliper estimate of 2.0 outcome points with standard error 0.9, and a wider-caliper estimate of 3.1 points with standard error 0.6. For illustration, normal-approximation 95% intervals are computed as estimate $\pm 1.96SE$.

For the narrow width: $$ 2.0\pm1.96(0.9)=2.0\pm1.764, $$ so the interval is 0.236 to 3.764 points. For the wider width: $$ 3.1\pm1.96(0.6)=3.1\pm1.176, $$ so the interval is 1.924 to 4.276 points.

The wider analysis has the smaller reported standard error, but its estimate is 1.1 points higher. If the wider width retains a different composition of units, these intervals concern potentially different target effects. The intervals do not establish that the target shifted by exactly 1.1 points, nor do their overlap or non-overlap alone provide a valid test of equality of two targets. The comparison must account for the design, dependence between analyses, and target definition.

Practical Sensitivity Analysis and Diagnostics

A defensible comparison starts by specifying the matching scale and a set of caliper widths before interpreting results. For each width, record the number and identity of retained treated and control units, the distribution of the matching covariates or propensity score among retained units, the matching weights, and the effect estimate with its uncertainty. These summaries help distinguish a target-population change from a precision change.

Check whether the retained sets are actually nested. This may be true for some matching rules, but matching without replacement can reassign matches as the allowed distance changes. A unit matched at one width can lose its match at another, depending on the algorithm and its tie-breaking rules. If set membership changes in both directions, describe the additions and losses explicitly and compare the width-specific targets directly.

Also inspect balance and match distances at every width. A wider threshold permits more distant matches by construction; it does not guarantee acceptable covariate balance. A tight threshold can yield good balance among retained units while sharply narrowing the target population. A small retained sample, discarded treated units, extreme weights, or substantial changes in covariate distributions are warning signs that the reported effect may be highly specific to a restricted group.

The main alternatives answer different questions. Choosing one prespecified caliper and reporting its target is straightforward, but can conceal sensitivity to that choice. Reporting a range of widths makes the dependence visible, provided each result is labeled by its retained population and weights. Alternative matching designs or weighting approaches may target different populations; they are not interchangeable precision improvements. Standardizing effects to a chosen population can support a common-population comparison when the required information and assumptions are available, but it does not turn the raw caliper-specific estimates into the same estimand.

Common pitfall: “The wider caliper has more units and a smaller standard error, so it gives a more precise estimate of the same effect.” This reasoning assumes, without checking, that the target units and their relative weights have not changed. The correction is to compare retained-unit identities and weights first, define the target at each width, and then discuss precision for each target separately.

Check Your Understanding

Answer each question using the target and weight definitions above. Full solutions are given below.

A narrow width retains units with average effect 4 points and total weight 8. A wider width adds units with average effect 1 point and total weight 2, while old weights are unchanged. Find the wider target and its shift from the narrow target.
Why is it invalid to use the narrow-width old-group average when widening the caliper changes the relative weights of old units?
A wider caliper has a smaller standard error but also changes the retained units. What does the smaller standard error establish, and what does it not establish?
What should be checked before using an “old units plus newly added units” decomposition?

Solutions. (1) The wider target is $\{8(4)+2(1)\}/(8+2)=34/10=3.4$ points. Its shift is $3.4-4=-0.6$ points. Equivalently, the added units have weight share $2/10$ and differ from the old target by $1-4=-3$, giving $(2/10)(-3)=-0.6$. (2) The narrow average uses the old weights, whereas the wider target must average the old units using their weights at the wider width. A change in relative weights can change their group average even if no old unit is removed. (3) It establishes a smaller estimated standard error for that estimator under the stated variance calculation. It does not establish that both analyses target the same population effect or that one target is more causally credible. (4) Check that previously retained units remain retained, identify any units added or lost, and verify whether old-unit weights stay fixed. If they change, recalculate the old-group mean under the new weights; if membership is not nested, compare the width-specific targets directly.