Bootstrap Diagnostics For Estimated Bounds

Why Diagnose Bootstrap Resamples?

An estimated identified interval has two endpoints, \(\hat L\) and \(\hat U\). Both can vary from sample to sample, even when the method and its identifying assumptions are unchanged. Bootstrap resampling offers a way to examine that sampling variability by repeatedly drawing new samples from the observed data and recomputing both endpoints. For estimated bounds, however, the resamples can also expose a practical problem: an endpoint may depend heavily on a few observations or on treatment and control observations appearing together in a sparse covariate stratum.

The bootstrap is a diagnostic for the behavior of an estimator under resampling; it does not measure the width of the identified interval itself. A wide interval \([\hat L,\hat U]\) concerns what the assumptions and observed data identify. Variation in \(\hat L\) or \(\hat U\) across bootstrap samples concerns the sampling uncertainty of the estimated endpoints. Nor can a stable bootstrap distribution establish that the assumptions used to construct the bounds are credible.

Definition: For each bootstrap replicate \(b\), resample observational units according to the sampling design, apply the full bound-estimation procedure to that resample, and record the resulting endpoints \(\hat L^{*(b)}\) and \(\hat U^{*(b)}\). The empirical distributions of these replicate endpoints approximate how the estimators may vary under repeated sampling, subject to the adequacy of the resampling scheme and estimator.

Procedure and Conditions

For independent and identically distributed observational units, the ordinary nonparametric bootstrap draws \(n\) rows with replacement from the \(n\) observed rows. A row should contain the variables that belong to that unit, including treatment, outcome, and covariates. Keeping each row intact preserves the observed joint relationships among those variables. In each replicate, redo every data-dependent step that is part of the estimator: construct covariate strata if they are estimated from data, calculate the relevant cell quantities, impose the chosen restrictions, and solve any optimization problem used to obtain the endpoints.

The bootstrap is most defensible when the sample represents independent draws from a population and the estimator has sampling behavior that the empirical distribution can approximate. If observations are clustered, resampling individual rows as if independent generally gives the wrong variability; resample independent clusters instead. If treatment counts are fixed by design or the desired procedure conditions on treatment-group sizes, a treatment-stratified bootstrap may be appropriate. The resampling scheme must reflect the actual design and the target sampling variation, not merely produce convenient replicates.

Specify the target estimator.
Write down how the data produce \(\hat L\) and \(\hat U\), including fixed sensitivity limits, estimated weights, strata, or optimization constraints.

Resample the appropriate units.
Draw \(n\) units with replacement for an iid sample, or use a design-respecting scheme such as cluster resampling when units are dependent within clusters.

Recompute both endpoints.
Apply the complete estimation procedure to replicate \(b\). Record endpoints, convergence status, and any empty-cell or constraint failures.

Inspect the replicate distributions.
Summarize endpoint spread, tail behavior, endpoint ordering, unusual piles of values, and the proportion of failed or problematic replicates.

With \(B\) successful replicates, the bootstrap standard deviation for the lower endpoint is the sample standard deviation of its replicate estimates:

\widehat{SE}_{boot}(\hat L)= \sqrt{\frac{1}{B-1}\sum_{b=1}^{B} \left(\hat L^{*(b)}-\bar L^*\right)^2}, \qquad \bar L^*=\frac{1}{B}\sum_{b=1}^{B}\hat L^{*(b)}.

The analogous calculation applies to \(\hat U\). These summaries should be accompanied by inspection of the distributions, because a standard deviation can conceal skewness, boundary piles, or a small number of extreme replicates. When failures occur, the fraction of failed resamples is itself a diagnostic; silently discarding them can make the remaining distribution look deceptively stable.

Worked Examples

Worked Example: A Rare Stratum Drives Endpoint Variation

Suppose a hypothetical estimator has a lower endpoint \(\hat L=-0.10-0.70\hat q\), where \(\hat q\) is the sample proportion in a rare covariate stratum. In a sample of \(n=20\), one unit belongs to that stratum, so \(\hat q=1/20=0.05\). The estimated endpoint is

\hat L=-0.10-0.70(0.05)=-0.135.

In an ordinary pairs bootstrap, the number \(K^*\) of rare-stratum observations in a resample has a binomial distribution conditional on the observed data: \(K^*\sim\operatorname{Binomial}(20,0.05)\). Thus \(q^*=K^*/20\), and the replicate endpoint is \(\hat L^*=-0.10-0.70q^*\). The binomial variance gives

\operatorname{SD}(q^*)= \sqrt{\frac{0.05(0.95)}{20}} =\sqrt{0.002375} \approx 0.04873.

The standard deviation in the endpoint attributable to this stratum-count variation is therefore

0.70\operatorname{SD}(q^*)= 0.70\sqrt{\frac{0.05(0.95)}{20}} \approx 0.03411.

This is not \(0.1079\): the square root is about \(0.04873\), and multiplying by \(0.70\) gives about \(0.03411\). The source of instability is also visible from the chance of drawing no rare-stratum unit at all: \(P(K^*=0)=0.95^{20}\approx0.3585\). In roughly 35.9% of bootstrap resamples, this stratum disappears. The calculation shows why a small number of observations can matter even when the original endpoint is numerically well defined. It describes only this source of variation; other components of the full estimator can add further variation.

Worked Example: Empty Treatment Cells in Resamples

Consider a hypothetical sample of 40 independent patients. A particular covariate stratum contains only one treated patient, and the bound estimator requires a treated outcome mean for that stratum. Under an ordinary pairs bootstrap, each draw selects one of the 40 observed rows. The sole treated patient is omitted from all 40 draws with probability

\left(\frac{39}{40}\right)^{40}\approx 0.3632.

So about 36.3% of resamples contain no treated observation from that stratum. If the endpoint formula requires that cell mean, those replicates cannot be computed as specified. This is not a minor numerical inconvenience: it signals that the estimator depends on a cell supported by a single treated observation.

The appropriate response is to report the failure rate and investigate the source. Do not replace an empty-cell quantity with zero, drop the affected stratum, or discard failures without explaining how that changes the estimator. If the bounds are instead defined using logical restrictions that remain meaningful in an empty cell, apply those restrictions consistently and inspect whether the resulting endpoint jumps sharply. Either way, a high failure rate or abrupt endpoint changes reveal a limitation of the data for the chosen procedure.

Worked Example: Reading a Bootstrap Diagnostic Summary

Suppose a hypothetical analysis runs \(B=1{,}000\) resamples and obtains 982 successful endpoint pairs. Eighteen replicates have an empty required cell. The observed failure proportion is

\frac{18}{1{,}000}=0.018,

or 1.8%. Among successful replicates, suppose \(\hat L^*\) has standard deviation 0.074 and its 2.5th and 97.5th percentiles are \(-0.43\) and \(-0.14\). The corresponding percentile span is \(-0.14-(-0.43)=0.29\). Suppose \(\hat U^*\) has standard deviation 0.061 and percentiles \(0.31\) and \(0.55\), a span of \(0.55-0.31=0.24\). These summaries suggest appreciable variability in both estimated endpoints, and a nonzero failure rate that should be investigated.

The percentile ranges here are descriptive diagnostics of the successful bootstrap replicates; they are not automatically a valid confidence set for the partially identified effect. Inferential coverage requires an appropriate method for the endpoint and its boundary behavior. Also check that each replicate preserves \(\hat L^{*(b)}\leq\hat U^{*(b)}\). If ordering is violated, inspect the code, constraints, and optimization tolerances rather than treating the violation as ordinary sampling fluctuation.

What the Diagnostics Can and Cannot Tell You

Several patterns merit attention. A large bootstrap standard deviation, a long tail, or a small number of replicate endpoints far from the rest can indicate sensitivity to influential units. A pile-up at a constraint boundary can arise because many resamples activate the same restriction; the resulting distribution may be non-smooth. Frequent empty cells point to sparse support. A high rate of solver failures may instead indicate numerical or implementation problems, though it can also reflect resamples in which the optimization problem is not well posed. Check the cause rather than interpreting every failed computation as a statistical result.

For estimated covariate strata, report their sample sizes and treatment counts alongside resampling diagnostics. If strata were formed using estimated cut points or a data-driven rule, repeat that construction within each replicate when it is part of the estimator; otherwise the bootstrap omits that source of uncertainty. Conversely, a sensitivity limit treated as fixed by the analysis should remain fixed in the resamples. If it is estimated from data, whether to re-estimate it depends on the target estimator and sampling procedure.

Bootstrap diagnostics complement, rather than replace, other approaches. A delta-method calculation can be efficient for a smooth estimator with a reliable variance approximation, but can be misleading for endpoint estimators involving minima, maxima, active constraints, or sparse cells. Stratified resampling can preserve treatment-group sizes when that conditioning matches the design, but it does not create observations in a rare treatment-by-covariate cell. A cluster bootstrap respects within-cluster dependence when clusters are the independent sampling units. Alternative procedures such as subsampling may be useful for some non-smooth estimators, but require their own justification and tuning choices.

The bootstrap is conditional on the observed data and the resampling model. It does not account for uncertainty in an unmeasured-confounding restriction merely because endpoints were bootstrapped, and it does not verify exchangeability, measurement quality, or any other causal assumption. It also cannot repair a lack of overlap or sparse support. When endpoints are near a boundary, use inference appropriate to that setting; bootstrap spread alone is not a warrant for a routine normal approximation.

Common Pitfall

Common mistake: “The bootstrap standard deviation is small, so the estimated bounds are reliable.”

Correction: A small spread can coexist with an invalid resampling scheme, omitted estimation steps, failed replicates that were discarded, or a boundary distribution that ordinary summaries describe poorly. Check the sampling units, failure rate, endpoint distributions, cell support, and assumptions. A bootstrap diagnoses sampling behavior under its resampling model; it does not certify the estimator or the identifying restrictions.

Key takeaway: Recompute both endpoints in every design-appropriate resample, and treat sparsity, boundary piles, endpoint-order violations, and failed replicates as substantive diagnostics—not details to hide.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In an iid sample, one of 25 observations belongs to a rare stratum. What is the bootstrap probability that the stratum is absent from a resample of size 25?
Why should treatment, outcome, and covariates be resampled together for an ordinary pairs bootstrap?
A bootstrap produces 40 failed endpoint calculations out of 800 attempted resamples. Calculate the failure proportion and name one conclusion it supports and one conclusion it does not support.
Explain why a small bootstrap standard deviation does not establish that the identifying assumptions for the bounds are correct.

Solutions. (1) The rare-stratum proportion in the empirical distribution is \(1/25\), so the probability of omitting it in all 25 draws is \((24/25)^{25}\approx0.3604\), or about 36.0%. (2) Resampling complete rows preserves the observed joint values and relationships; resampling columns separately would create combinations not present in the data and would not represent the pairs-sampling scheme. (3) The failure proportion is \(40/800=0.05\), or 5%. This supports investigating sparse cells, numerical issues, or a poorly defined estimator in some resamples; by itself, it does not establish which cause is responsible or that the successful-replicate distribution has valid inferential coverage. (4) Bootstrap variation concerns repeated-sampling behavior under the resampling scheme. The identifying assumptions are substantive restrictions and are not tested or validated by a small estimated standard deviation.