Bootstrap Pitfalls in Matching and Weighting

Why Bootstrap Choices Matter

Resampling is often used to estimate the sampling variability of an estimator: repeatedly create data sets by sampling observations with replacement, recompute the estimator, and examine the resulting distribution. This procedure is useful only when the estimator responds to resampling in a way the bootstrap can reproduce. That distinction matters in causal analyses because a fixed-neighbor matching estimator is not a smooth function of the observed data, whereas many inverse-probability-weighted (IPW) estimators are smoother under suitable regularity conditions.

The issue is not that bootstrap methods are always inappropriate for causal effects. Rather, the ordinary pairs bootstrap—resampling observational units as complete records—does not generally estimate the sampling distribution correctly for fixed-number nearest-neighbor matching. In contrast, it is often a reasonable option for a regular weighted estimator, provided the full estimation procedure is repeated and the data satisfy the relevant conditions.

Why Fixed-Neighbor Matching Is a Problem for the Ordinary Bootstrap

In fixed-neighbor matching, the estimator depends not only on observed outcomes but also on which observations are nearest in covariate space. A small change in the data can change a selected neighbor. In a bootstrap sample, some original units are absent and others appear more than once. That changes the set of candidate matches, the distances between candidates, and how often a control observation is used. The estimator can therefore jump rather than change gradually.

This behavior is called nonsmoothness. The ordinary bootstrap works well for many estimators whose sampling behavior can be approximated by a stable, first-order linear expansion. Fixed-neighbor matching generally does not have the regular smooth behavior needed for the ordinary pairs bootstrap to reproduce its sampling distribution. In particular, bootstrap duplicates and omissions alter the local neighbor structure in a way that does not mimic the original estimator's large-sample variation.

The problem is not repaired merely by keeping the number of treated and control observations fixed in every resample. Stratified resampling can prevent a group from disappearing, but it still omits some units, duplicates others, and changes the nearest-neighbor relationships. Nor does one unstable bootstrap replicate prove that all bootstrap results are wrong: the concern is the method's general lack of validity for this estimator, not the appearance of any particular replicate.

Variance Methods Designed for Matching

Matching inference should account for the estimator actually being used: the match structure, reuse of comparison observations, and, where relevant, bias from imperfect matches. For fixed-neighbor matching, established approaches include matching-specific analytic variance estimators, such as the Abadie–Imbens variance estimator, and bias correction paired with an appropriate variance calculation. These methods are not interchangeable with the ordinary pairs bootstrap.

One way to see why reuse matters is to condition temporarily on the observed covariates and the selected matches. For a treated-group effect with \(N_1\) treated units and \(M\) matched controls per treated unit, let \(K_j\) be the number of times control \(j\) is used. Under conditionally independent outcome errors, the control outcome \(Y_j\) has coefficient \(-K_j/(MN_1)\) in the estimate. If its conditional outcome variance is \(\sigma_0^2(X_j)\), its contribution to the conditional outcome-noise variance is the squared coefficient times that variance. The treated outcomes contribute their own conditional variance terms. This calculation explains why a control used repeatedly cannot be treated as if it were a new independent observation each time.

That conditional calculation is not, by itself, a complete sampling variance for matching. It holds the observed covariates and selected matches fixed and therefore does not capture all variation from sampling the covariates and forming matches. Matching-specific inference must handle the relevant sources of variation under its assumptions. In addition, matching bias may be material when covariates are continuous and matches are not exact; bias correction can be needed for valid large-sample inference. A variance estimate does not remove bias.

When a Pairs Bootstrap Can Be Useful for Weighted Effects

A weighted estimator is often a smoother function of the empirical data than a fixed-neighbor matching estimator. For example, with known propensity scores \(e(X_i)=P(D_i=1\mid X_i)\), the inverse-probability-weighted estimator of the average treatment effect can be written

Here \(D_i\) indicates treatment, and \(Y_i\) is the observed outcome. Under identification assumptions, including conditional independence of treatment and potential outcomes given observed covariates and positivity, the estimator targets the population average treatment effect. For bootstrap inference, the relevant statistical requirements also include regularity conditions: probabilities must not be too close to zero or one, the estimator must have finite enough moments, and its sampling behavior must be sufficiently stable.

When the propensity score is estimated, each bootstrap replicate should ordinarily repeat the whole procedure: resample complete observational units, refit the propensity-score model, recalculate the weights, and then recompute the effect estimate. Holding estimated propensity scores fixed ignores their estimation variability. A pairs bootstrap is not a universal guarantee: extreme weights, weak overlap, highly influential observations, or unstable estimation can make ordinary bootstrap approximations poor. The diagnostic material on overlap and positivity is therefore relevant to deciding whether a weighted estimator is suitable for this approach.

Practical Choice and Common Pitfalls

The resampling method should follow the estimator's statistical behavior, not simply be selected because it is familiar or easy to run. For fixed-neighbor matching, use a matching-specific variance method with assumptions appropriate to the design and estimator, and address bias where needed. For a regular IPW estimator, a pairs bootstrap may be appropriate under suitable conditions; check overlap and weight stability, and repeat all estimated steps. If observations are dependent within clusters, resampling individual rows does not preserve that dependence; the resampling unit must reflect the sampling design.

Subsampling or an \(m\)-out-of-\(n\) bootstrap is sometimes considered for nonregular estimators, but neither is an automatic substitute for the ordinary bootstrap. Its validity and implementation depend on the estimator and on choices such as the subsample size. Use such methods only with a justification specific to the matching procedure and inferential target.

• Mistake: “The bootstrap always works because each replicate uses the same sample size.” Correction: Equal sample size does not ensure validity. Fixed-neighbor matching can have an unstable match structure under resampling.
• Mistake: “A matched control used three times is equivalent to three independent controls.” Correction: Reuse creates a repeated contribution from the same observed outcome; matching variance methods must account for that dependence.
• Mistake: “A good variance estimate makes matching bias disappear.” Correction: Variance describes sampling variability; bias correction addresses systematic discrepancy between matched units.
• Mistake: “For IPW, I can bootstrap the final weighted formula while keeping fitted scores unchanged.” Correction: When scores are estimated, refit the propensity model within each replicate to include its estimation variability.