Uncertainty in Data-Adaptive Matched Targets

Why a Matched Target Can Be Uncertain

Matching is often described as if it simply estimates an effect for a specified group. But the matching procedure typically uses observed treatment and covariate data to decide which units are retained, paired, or weighted. In another sample, or under another realization of the data, it may select a different set of units. When treatment effects vary across people, a change in selected units can change the population effect being summarized—not just the estimate of that effect.

This creates two kinds of uncertainty that should not be conflated. Estimation uncertainty concerns how far an estimate may be from the target defined for the realized matched data. Target uncertainty concerns how that target itself changes when the data-adaptive selection is repeated. A standard error that addresses the first does not necessarily describe the second.

Definition: Let $D$ denote the observed data used by a specified matching procedure, and let $S_i(D)$ equal 1 if unit $i$ is retained and 0 otherwise. Write the unit-level treatment effect as $\tau_i=Y_i(1)-Y_i(0)$. For a finite set of $N$ units, the retained-unit average treatment effect is $$ \tau_S(D)=\frac{\sum_{i=1}^{N} S_i(D)\tau_i}{\sum_{i=1}^{N} S_i(D)}, $$ provided at least one unit is retained. Because $S_i(D)$ can change across repeated datasets, $\tau_S(D)$ can also change. With nonnegative matching weights $W_i(D)$, the analogous weighted target is $\tau_W(D)=\frac{\sum_i W_i(D)\tau_i}{\sum_i W_i(D)}$, provided the total weight is positive.

For a realized dataset, the matched units and their target are fixed once the algorithm has run. The uncertainty arises when considering the procedure across possible datasets: the data change, the matching decisions may change, and the target may consequently change. This is not the same as saying that a target is unknowable in the realized sample.

The notation also separates a target from its estimate. The target uses both potential outcomes, $Y_i(1)$ and $Y_i(0)$, for each relevant unit. In observed data, only one is seen for each person. Matching and outcome analysis use the observed data to estimate a causal contrast for the selected target; they do not make the missing potential outcomes observable.

What Is Random: The Selection Rule and the Target

Suppose a matching algorithm is fully specified in advance, including its covariates, distance measure, caliper, and retention rule. Even a fixed algorithm can produce different retained sets across repeated samples. The algorithm is fixed; its input is not. If the sample, treatment assignments, or observed covariates vary, the resulting selection indicators and weights can vary.

A useful conceptual distinction is between the realized matched target, $\tau_S(D)$, and the distribution of that target over repeated datasets. For example, $E_D[\tau_S(D)]$ is the average target over the specified data-generating process and matching procedure. It is generally not interchangeable with a ratio formed from average selection probabilities: $$ E_D\left[\frac{\sum_i S_i(D)\tau_i}{\sum_i S_i(D)}\right] \ne \frac{\sum_i E_D[S_i(D)]\tau_i}{\sum_i E_D[S_i(D)]} $$ in general. The left side averages the ratio from each dataset; the right side forms a ratio after averaging the selection indicators. They need not agree because the denominator is itself random.

This distinction connects to finite-sample and superpopulation estimands. A finite-sample target concerns the particular units under consideration; a superpopulation target concerns a population-generating process. If the matched target is also data-adaptive, its repeated-sampling interpretation requires specifying how the data and matching decisions are generated. Merely giving the matching algorithm does not make the target a single fixed population effect.

Worked Example: Two Possible Matched Targets

Consider four hypothetical units with fixed treatment effects of 1, 3, 7, and 9 outcome units. A matching procedure, across repeated datasets, retains either units 1 and 2 or units 3 and 4, with probability $1/2$ for each outcome of the selection process. Assume exactly two units are retained each time.

If units 1 and 2 are retained, the target is $$ \tau_A=\frac{1+3}{2}=2\text{ outcome units}. $$ If units 3 and 4 are retained, it is $$ \tau_B=\frac{7+9}{2}=8\text{ outcome units}. $$

Thus the target across repetitions equals 2 or 8, each with probability $1/2$. Its mean is $E_D[\tau_S(D)]=(2+8)/2=5$ outcome units, and its variance is $$ \operatorname{Var}_D\{\tau_S(D)\} =\frac{1}{2}(2-5)^2+\frac{1}{2}(8-5)^2=9\text{ outcome units}^2. $$ The standard deviation of the target is therefore 3 outcome units. These calculations describe variation in which effect is being targeted, even if the effects themselves are treated as fixed. A result for the realized matched set targets 2 or 8, not automatically the average value of 5.

Effect Modification Turns Composition Changes into Target Changes

Selection uncertainty matters especially when treatment effects differ across covariate-defined groups. If every unit has the same treatment effect, changing the retained set does not change the average effect (though it can still change the estimate’s precision). If effects vary, the covariate composition of the matched target determines how subgroup effects are combined.

For example, write $G_i=1$ for a subgroup with average effect $\tau_1$, and $G_i=0$ for a subgroup with average effect $\tau_0$. If a matched target contains a fraction $Q(D)$ of subgroup 1, its average effect is $$ \tau_S(D)=Q(D)\tau_1+\{1-Q(D)\}\tau_0. $$ A fluctuation in $Q(D)$ shifts the target by an amount proportional to the difference $\tau_1-\tau_0$. When effects are similar across groups, the shift is small; when effects differ substantially, the same composition change can matter greatly. This is why covariate-distribution diagnostics also inform interpretation of the target, not only assessment of balance.

Worked Example: Target Variation from Subgroup Composition

Suppose the hypothetical average effect is 2 outcome units in subgroup 0 and 8 outcome units in subgroup 1. A matching procedure returns four retained units. Across repeated datasets, suppose the number $K$ from subgroup 1 follows a binomial distribution with $n=4$ and $p=0.5$. Then the subgroup 1 fraction is $Q=K/4$, and the target is $$ \tau_S=2(1-Q)+8Q=2+6Q=2+\frac{6K}{4}. $$

The expected target is $$ E_D[\tau_S]=2+\frac{6}{4}E[K] =2+\frac{6}{4}(4\times 0.5)=5\text{ outcome units}. $$ Since $\operatorname{Var}(K)=np(1-p)=4(0.5)(0.5)=1$, the target variance is $$ \operatorname{Var}_D(\tau_S) =\left(\frac{6}{4}\right)^2\operatorname{Var}(K) =2.25\text{ outcome units}^2, $$ so the target standard deviation is 1.5 outcome units. If a particular dataset retains three subgroup 1 units, then $K=3$ and the realized target is $2+6(3/4)=6.5$ outcome units. The average target of 5 is not the effect for that realized composition.

The calculation assumes the specified subgroup effects and selection distribution. In an actual analysis, neither the distribution of $K$ nor the subgroup effects is usually known exactly; the example isolates the mechanism by which composition changes move the target.

Separate Target Variation from Estimation Error

Suppose $\hat{\tau}(D)$ estimates $\tau_S(D)$. Conditional on the realized target, the estimation error is $\hat{\tau}(D)-\tau_S(D)$. Across repeated datasets, the squared deviation from a single reference value can reflect both this error and variation in the target. The law of total variance makes the distinction explicit: $$ \operatorname{Var}\{\hat{\tau}(D)\} =E_D\!\left[\operatorname{Var}\{\hat{\tau}(D)\mid D_{\text{target}}\}\right] +\operatorname{Var}_D\!\left(E[\hat{\tau}(D)\mid D_{\text{target}}]\right), $$ where $D_{\text{target}}$ denotes the selection information that determines the target. If the estimator is conditionally unbiased for that target, then $E[\hat{\tau}(D)\mid D_{\text{target}}]=\tau_S(D)$; the second term is target variation. This decomposition is conceptual as well as inferential: it identifies which quantity a reported uncertainty measure is intended to describe.

A conventional standard error may quantify sampling variability for a specified realized target under its inferential assumptions. It does not automatically include the variation of $\tau_S(D)$ across datasets. Conversely, adding target variation to a standard error is not always appropriate: doing so may answer a different question, one about the repeated matching procedure and its changing targets rather than uncertainty about the target selected in the observed analysis.

Worked Example: Two Components of Repeated-Procedure Variability

Suppose a hypothetical procedure yields a target of 2 or 8 outcome units with equal probability, as in the first example. Conditional on whichever target is realized, assume an estimator is unbiased and has variance 1 outcome units squared. Thus $$ E[\hat{\tau}\mid \tau_S]=\tau_S \quad\text{and}\quad \operatorname{Var}(\hat{\tau}\mid \tau_S)=1. $$

By the law of total variance, $$ \operatorname{Var}(\hat{\tau}) =E\{\operatorname{Var}(\hat{\tau}\mid\tau_S)\} +\operatorname{Var}\{E(\hat{\tau}\mid\tau_S)\} =1+9=10\text{ outcome units}^2. $$ The overall standard deviation across repetitions is $\sqrt{10}=3.162$ outcome units. The conditional standard error is 1 outcome unit; it describes estimation variability around the realized target. The additional variance of 9 comes from the target changing between 2 and 8. Reporting 1 as though it summarized all repeated-procedure variation would omit target variation. Reporting $\sqrt{10}$ as uncertainty about one fixed realized target would instead mix two distinct questions.

Assumptions, Diagnostics, and Practical Choices

Target uncertainty is a property of the selection-and-analysis procedure; it is not itself evidence of bias or a failure of matching. But causal interpretation still requires appropriate assumptions. In potential-outcomes terms, consistency links the observed outcome to the potential outcome under the treatment received. Conditional exchangeability, $Y(a)\perp A\mid X$ for $a\in\{0,1\}$, and positivity, $0<P(A=1\mid X=x)<1$ in the relevant covariate region, are common identification conditions. They must be credible for the population targeted by the analysis. Good balance after matching does not establish these assumptions or eliminate unmeasured confounding.

The target calculation also presumes a defined procedure and a nonzero retained set or positive total weight. To describe repeated target variation, one must additionally specify what is repeated: for example, sampling new units from a population, generating new treatment assignments, or both. Different repetition schemes can produce different distributions of selected targets. If the matching decisions depend on outcomes, selection can introduce further problems; matching is ordinarily designed using treatment and covariate information, not outcome values.

Useful diagnostics include documenting the number and characteristics of retained units, examining covariate distributions and subgroup proportions, and assessing how these summaries change under reasonable perturbations of the data or matching settings. Earlier material on caliper width and matching ratio concerns how design choices can shift a target; here the additional question is whether sampling variation in the inputs makes the realized target unstable. A large change in subgroup composition is especially consequential when effect modification is plausible.

A bootstrap that repeats the entire matching procedure can be informative about the stability of selection and the estimate, but it is not a universal solution. Matching decisions can change discontinuously when small data perturbations alter nearest neighbors or cross a caliper threshold. Standard bootstrap approximations may therefore be unreliable for some matching estimators. The resampling scheme and inferential justification must fit the specific estimator and design; a bootstrap distribution should not be described as target uncertainty unless the target is recalculated and that is indeed the quantity being summarized.

The main alternatives answer different questions. A prespecified target such as an ATE or ATT is not automatically identical to a target induced by realized matching. Restricting eligibility in advance can define a fixed population more directly, though it changes the scientific question and still requires causal assumptions. Describing the realized matched target and reporting its composition makes clear what the analysis actually summarizes. There is no universally preferred choice: the target should follow the scientific question, and conclusions should not be generalized to a different population without a separate justification.

Common pitfall: “The matched effect has a small standard error, so the target population is known precisely.” This reasoning confuses uncertainty in an estimate conditional on the realized matched set with variation in which units the procedure would select across repetitions. A small conditional standard error can coexist with substantial target variation. State which target the estimate describes, report its realized composition, and avoid interpreting it as an effect for a fixed broader population by default.

Define the target-producing rule.
Specify which data determine retention or weights and how the matching procedure is applied.

Describe the realized target.
Report retained-unit counts or weight distribution and relevant covariate or subgroup composition.

Assess target stability.
Consider whether plausible repeated samples or small data changes could produce materially different compositions and effects.

Match the uncertainty statement to the question.
Distinguish uncertainty about the realized target’s effect from variation in targets across repetitions of the full procedure.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If every unit has the same treatment effect, can changing the retained set change the retained-unit average treatment effect? What uncertainty can remain?
Two subgroup effects are 1 and 7 outcome units. A matched set has subgroup 1 fraction $Q=0.25$. Calculate its target effect.
In the previous question, suppose the subgroup 1 fraction varies across repetitions with variance $0.04$. Find the variance of the target due to this composition variation.
Explain why a conditional standard error for an observed matched set does not, by itself, quantify variation in the target across repeated datasets.

Solutions. (1) No: the average remains the common treatment effect. Estimation uncertainty can remain because outcomes and samples vary. (2) The target is $1(1-0.25)+7(0.25)=0.75+1.75=2.5$ outcome units. (3) The target is $1+6Q$, so its variance is $6^2(0.04)=1.44$ outcome units squared. (4) The conditional standard error concerns estimation around the target determined by the observed selection; repeated datasets can select different units and therefore define different targets.