Why Bootstrap the Entire Weighting Procedure?
A weighted treatment-effect estimate depends on more than the observed outcomes. It also depends on estimated propensity scores, which determine the weights. If those scores were estimated from the same sample, treating them as fixed can omit a source of sampling variability. The nonparametric bootstrap addresses this by resampling observational units and repeating the complete estimation procedure in each resample.
The key instruction is refit, then reweight. Resampling only the final weighted outcomes, or carrying the original propensity scores into every resample, does not reproduce the estimator whose uncertainty is being assessed. The procedure below focuses on the average treatment effect (ATE) with inverse probability weights; the same full-pipeline principle applies when a different target and its corresponding weights are specified.
Let \(A_i\) be the binary treatment indicator, \(X_i\) the measured pretreatment covariates, and \(Y_i\) the observed outcome for unit \(i\). Write \(e(X_i)=P(A_i=1\mid X_i)\) for the propensity score and \(\hat e_i\) for its estimate. For the ATE, a common normalized inverse probability weighted estimator is the treated weighted mean minus the control weighted mean:
This is a Hájek-style normalized estimator. In bootstrap sample \(b\), let \(m_{ib}\) be the number of times unit \(i\) was drawn. Fit the chosen propensity-score model to that resampled data to obtain \(\hat e_{ib}^{*}\). Then calculate:
The multiplicities \(m_{ib}\) are a convenient way to represent resampling: a unit drawn twice contributes twice, and a unit not drawn contributes nothing. Equivalently, one can make a resampled data set with \(n\) rows, including duplicate rows, and fit and calculate the estimator on those rows.
Conditions and the Bootstrap Calculation
The bootstrap describes sampling variability under a sampling scheme; it does not make a causal estimate valid by itself. A causal interpretation of the ATE still requires consistency, no interference between units, conditional exchangeability given the measured covariates, and positivity for the target population. The propensity-score model must also be suitable for estimating the scores used by the estimator. These assumptions are distinct from the additional conditions needed for a bootstrap approximation to work well.
For the ordinary unit-level bootstrap, the observational units should be independent and identically sampled from the relevant population, or the sampling design must be represented appropriately. The estimator should behave regularly under resampling: in particular, its weights and fitted model should not be dominated by a few extreme or unstable values. If units are dependent within groups, resampling individual rows does not preserve that dependence; a unit-level bootstrap is then not automatically appropriate.
A practical implementation follows the same steps in every replicate. Keep the outcome, treatment, and covariates together as a unit; draw \(n\) such units with replacement; refit the same propensity-score model specification; calculate the chosen weights; recompute the treatment effect; and save the estimate and any diagnostic information. “Same specification” means the same prespecified modeling procedure. If the estimator includes data-driven variable selection or tuning, that selection step is part of the procedure and generally must be repeated in each resample.
Draw \(n\) observational units with replacement, retaining each selected unit’s treatment, covariates, and outcome together.
Apply the specified model to the resampled treatment and covariate data to obtain new fitted scores.
Use the replicate’s scores, not the original scores, to calculate its weights and weighted treatment-effect estimate.
Use the empirical spread of the \(B\) estimates for a bootstrap standard error and their empirical quantiles for a percentile interval.
If the bootstrap replicates are \(\hat\tau_1^{*},\ldots,\hat\tau_B^{*}\), their estimated standard error is their sample standard deviation:
A two-sided percentile confidence interval with nominal level 95% uses the 2.5th and 97.5th percentiles of the bootstrap estimates. It is an interval procedure whose repeated-sampling coverage is intended to be about 95% when its assumptions and approximation are adequate; it does not mean there is a 95% probability that the fixed population effect lies in this particular interval. A normal-approximation interval, \(\hat\tau_{\mathrm{ATE}}\pm1.96\widehat{SE}_{\mathrm{boot}}\), is another option, but can be less suitable when the bootstrap distribution is skewed or irregular.
Worked Calculations
Worked Example: Refit Scores in a Resampled Dataset
Suppose a hypothetical data set has two covariate strata, \(X=0\) and \(X=1\), and a saturated propensity model that estimates the treatment probability separately in each stratum. In one bootstrap resample of eight rows, the counts and outcomes are as follows. Duplicate rows are possible because sampling is with replacement.
| Stratum | Treated outcomes | Control outcomes | Fitted score |
|---|---|---|---|
| \(X=0\) | 6, 8 | 4, 5 | \(2/4=0.50\) |
| \(X=1\) | 10 | 7, 7, 9 | \(1/4=0.25\) |
The refitted score in each stratum is its treated proportion. Thus \(\hat e_0^{*}=2/(2+2)=0.50\) and \(\hat e_1^{*}=1/(1+3)=0.25\). The control probabilities are \(0.50\) and \(0.75\), respectively. These are recalculated from the resample, rather than copied from the original data.
For the treated weighted mean, the \(X=0\) contribution to the numerator is \((6+8)/0.50=28\), and its denominator contribution is \(2/0.50=4\). The \(X=1\) contributions are \(10/0.25=40\) and \(1/0.25=4\). Therefore:
For controls, the corresponding numerator contributions are \((4+5)/0.50=18\) and \((7+7+9)/0.75=30.667\). The denominator contributions are \(2/0.50=4\) and \(3/0.75=4\). Hence:
This is one bootstrap estimate, not a confidence interval or a final treatment-effect estimate. It illustrates that the resampled treatment proportions change the fitted scores and therefore change both sets of weights.
Worked Example: Summarizing a Bootstrap Distribution
Consider a separate hypothetical analysis with an original ATE estimate of \(2.75\) outcome units. The analyst performs \(B=1{,}000\) full-pipeline resamples. Suppose the bootstrap estimates have mean \(2.78\), sample standard deviation \(0.62\) outcome units, and empirical 2.5th and 97.5th percentiles of \(1.48\) and \(3.92\).
The bootstrap standard error is the standard deviation of the replicate estimates, so the estimated sampling uncertainty is \(0.62\) outcome units. The percentile interval is \([1.48,3.92]\) outcome units. Because the observed estimate is a weighted difference in mean outcomes, the interval is also expressed in outcome units.
The interval is not calculated by adding and subtracting \(1.96\) standard errors. That alternative would give \(2.75\pm1.96(0.62)\), or approximately \([1.535,3.965]\). The two intervals differ because one uses empirical quantiles and the other imposes a symmetric normal approximation. The percentile interval is the one specified by the reported quantiles; neither interval is automatically reliable if resampling reveals severe instability.
The small difference between the original estimate, \(2.75\), and the bootstrap mean, \(2.78\), is not itself evidence of bias or a reason to replace the original estimate. Bootstrap replicates approximate the estimator’s sampling distribution; their mean may differ from the original estimate in a finite sample.
Worked Example: Why Fixed Scores Give a Different Procedure
Suppose that, on the same hypothetical data, a procedure that holds the original propensity scores fixed produces a bootstrap standard error of \(0.48\) outcome units, while the full-pipeline procedure produces \(0.62\). The difference is \(0.62-0.48=0.14\) outcome units. The fixed-score calculation varies the resampled observations but omits variation from refitting the propensity model.
The value \(0.48\) is not necessarily an underestimate in every data set: the effect of holding scores fixed depends on how score estimation and the weighted outcome calculation vary together. But it is the uncertainty for a different, conditional calculation, not the full procedure that estimates scores from the sample. If the target is the sampling variability of the full estimator, refitting is required in every replicate.
This comparison does not correct misspecification or unmeasured confounding. Both procedures can produce narrow intervals around a biased or noncausal estimate. The bootstrap quantifies sampling variation under the procedure and assumptions used; it does not validate those assumptions.
Diagnostics, Alternatives, and Limitations
Inspect more than the final standard error. Record whether every replicate converged, whether both treatment groups appear, the range of fitted scores, the largest weights, and the distribution of replicate effects. A long tail, marked skewness, or a few extreme estimates can make a symmetric interval misleading and can signal practical positivity problems. Also inspect covariate balance and weight behavior in the original analysis; resampling does not repair poor overlap.
A replicate may have no treated or no control units in a relevant covariate region, produce fitted probabilities of zero or one, or fail to converge. Such a replicate may make the weighted estimator undefined or numerically unstable. Do not silently discard failures and report the remaining bootstrap distribution as if it came from the planned procedure. Report their frequency, investigate the cause, and reconsider whether the model, target population, or estimator is supportable. Increasing \(B\) reduces Monte Carlo error in summaries but does not fix a fundamentally unstable estimator.
The full-pipeline bootstrap is an alternative to analytic variance calculations that account for estimated propensity scores, and to a bootstrap that holds the scores fixed. Analytic methods can be more computationally efficient and may be well justified for a specified estimator, while the bootstrap can be convenient for a complicated fitting-and-weighting workflow. Their validity depends on their own assumptions; a bootstrap is not automatically superior simply because the estimator is complicated. For dependent observations, the resampling unit must respect the sampling structure rather than defaulting to independent rows.
Finally, a standard nonparametric bootstrap cannot reveal confounding from unmeasured variables, guarantee adequate overlap, or establish that a propensity model is correctly specified. It estimates uncertainty conditional on the observed data-generating setup and the estimator being repeated. The earlier material on inference with estimated propensity scores provides related context for why score estimation matters; here the operational point is to repeat the entire estimator, not only its last weighted-mean calculation.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In a bootstrap replicate, should the original estimated propensity scores be retained or should the propensity model be refit? Why?
- A bootstrap analysis uses \(B=800\) replicate ATE estimates with a sample standard deviation of \(1.3\) outcome units. What is the bootstrap standard error, and what does it describe?
- What does a 95% percentile interval from bootstrap estimates mean, and what does it not mean?
- Several resamples produce fitted scores of zero for some units and undefined weighted effects. Is it appropriate to silently drop those replicates? Explain.
- Name one important source of uncertainty or bias that a bootstrap does not automatically address.
Solutions. (1) Refit the propensity model in each replicate so the bootstrap reflects estimation of the scores as well as the weighted outcome calculation. (2) The bootstrap standard error is \(1.3\) outcome units; it estimates the sampling spread of the estimator under the resampling procedure. (3) The interval procedure is intended to cover the fixed population effect in about 95% of repeated samples when its assumptions and approximation are adequate. It is not a 95% probability statement about the parameter being in this particular interval. (4) No. Report and investigate the failures; silently discarding them changes the resampling procedure and can conceal instability. (5) Examples include unmeasured confounding, a misspecified propensity model, and lack of positivity. The bootstrap does not automatically correct any of these.