Why Estimated Support Can Shift
A propensity score is a conditional treatment probability, \(e(X)=P(A=1\mid X)\), where \(A=1\) denotes treatment and \(X\) denotes measured covariates. In practice, support is assessed using estimated scores, not the unknown \(e(X)\). Because the data used to estimate the score would differ across repeated samples, the fitted scores and the apparent boundaries of overlap can differ too. A boundary that looks precise in one data set may therefore be a fragile consequence of which observations happened to be sampled and how the score model was fitted.
This tutorial treats estimated support as a sample-dependent diagnostic. The aim is not to replace a substantive support assessment with a single bootstrap number. Rather, bootstrap score distributions and retention rates show whether a chosen support summary is stable under resampling and model refitting. That distinction matters: a variable boundary can signal that conclusions depend on a small number of observations, but a stable boundary does not establish that every relevant covariate pattern is represented.
These definitions describe score ranges in the observed sample; they are not a proof of population positivity or of overlap in the full covariate space. They also depend on the chosen score model. The earlier material on positivity, extreme weights, and information explains why weak support can matter for an analysis. Here the focus is how much the estimated evidence about support changes across samples.
What the Bootstrap Measures
A nonparametric bootstrap approximates repeated-sample variation by repeatedly resampling observed units, with replacement, and recalculating the statistic of interest. For support diagnostics based on estimated scores, the score model must generally be refitted in each replicate. Otherwise, the resampling captures variation in which observed rows are included but treats the original fitted scores as fixed, omitting the variation from estimating the score model.
One practical procedure is to resample rows separately within the treated and control groups, preserving each group's sample size, then refit the same prespecified propensity-score model in each replicate. The choice to resample within groups should match the sampling design and inferential target; it is not a universal requirement. In replicate \(b\), compute each resampled unit's fitted score, the selected support summary (such as \(\hat L_b\) and \(\hat U_b\)), and the retention count or rate \(R_b\). Keep a record of model failures and any replicate for which the estimated interval is empty.
Specify whether the diagnostic uses empirical score-range endpoints, fixed score cutoffs, or another clearly defined rule. Also specify whether endpoints are inclusive.
Use a resampling scheme consistent with the design. For a stratified diagnostic, resample with replacement separately within treatment groups.
Refit the propensity-score model in each replicate, then recompute the support summary and retention rate using that replicate's fitted scores.
Inspect the distributions of the boundaries and retention rates, alongside convergence failures and the number of replicates with inadequate or empty estimated support.
The resampling scheme, model specification, and support rule must be held consistent across replicates for the resulting distributions to be interpretable. If a replicate has no observations in a treatment group, or its fitted model does not converge, its support summary may be undefined. Such failures should be counted and reported, not silently discarded: frequent failures are themselves evidence that the procedure is unstable in the available data.
Worked Examples
Worked Example: Retention Under an Inclusive Score Rule
Hypothetical data. Suppose a fitted model produces the following ten scores in each group. A prespecified inclusive rule retains scores from 0.16 through 0.86.
Treated scores: 0.18, 0.24, 0.31, 0.42, 0.50, 0.58, 0.66, 0.72, 0.79, 0.88.
Control scores: 0.08, 0.16, 0.22, 0.35, 0.47, 0.55, 0.63, 0.74, 0.80, 0.86.
Among treated units, the nine scores from 0.18 through 0.79 satisfy the rule; 0.88 is above the upper cutoff. Thus, 9 of 10 treated units are retained. Among controls, the nine scores from 0.16 through 0.86 satisfy the rule; 0.08 is below the lower cutoff. Thus, 9 of 10 controls are retained. The total retention rate is
So the rule retains 90.0% of these 20 hypothetical units. The calculation checks both groups against the same stated cutoffs; it does not imply that these cutoffs are appropriate in every analysis. The diagnostic question is whether comparable retention and support boundaries would appear in resamples after refitting the score model.
Worked Example: Reading Bootstrap Boundary Variation
Hypothetical illustration. Suppose the original sample has 100 units, with 50 treated and 50 controls. In each of eight bootstrap replicates, the model is refitted, and the common empirical interval and number retained are recorded. The table gives the resulting summaries; the eight replicates are for illustrating the arithmetic, not for estimating precise tail quantiles.
| Replicate | Lower boundary \(\hat L_b\) | Upper boundary \(\hat U_b\) | Retained | Retention rate |
|---|---|---|---|---|
| 1 | 0.12 | 0.84 | 82 | 0.82 |
| 2 | 0.18 | 0.79 | 76 | 0.76 |
| 3 | 0.09 | 0.87 | 86 | 0.86 |
| 4 | 0.15 | 0.82 | 80 | 0.80 |
| 5 | 0.21 | 0.76 | 72 | 0.72 |
| 6 | 0.11 | 0.85 | 84 | 0.84 |
| 7 | 0.17 | 0.80 | 78 | 0.78 |
| 8 | 0.13 | 0.83 | 82 | 0.82 |
The mean retention rate in these replicates is
The smallest and largest replicate retention rates are 0.72 and 0.86, a span of 0.14, or 14 percentage points. The lower boundaries range from 0.09 to 0.21, while the upper boundaries range from 0.76 to 0.87. These ranges suggest that both the apparent support boundaries and the retained fraction respond to resampling. They are descriptive summaries of these eight hypothetical replicates, not a 95% confidence interval and not evidence that the population retention rate lies between 0.72 and 0.86.
Worked Example: Comparing Fixed Scores With Model Refitting
Hypothetical illustration. Imagine an original sample of 40 units assessed with a fixed score rule. A bootstrap that resamples the already estimated scores, without refitting the model, gives 32, 34, 30, and 36 retained units in four replicates. Their retention rates are 0.80, 0.85, 0.75, and 0.90, with mean
Now suppose the same four resamples are used to illustrate a procedure that refits the propensity-score model and then reapplies the rule. The retained counts are 30, 33, 28, and 35, giving rates 0.75, 0.825, 0.70, and 0.875. Their mean is
In this illustration, refitting changes both individual replicate rates and their average. The difference between these four-replicate averages is not a general bias correction: four replicates are far too few for a stable assessment, and the particular values are hypothetical. The point is methodological. Fixed-score resampling measures variation conditional on the original fitted scores; refitting also allows the estimated score model to vary. A full analysis would use many more replicates and examine the entire distribution, not only its mean.
Justification and Interpretation
The reason a bootstrap can reveal instability is that a support summary is a function of the observed data and the estimated model. Write that summary as \(T(\mathcal D)\), where \(\mathcal D\) is the sample. In each bootstrap replicate, the resampled data \(\mathcal D_b^*\) are used to estimate a new model and calculate \(T(\mathcal D_b^*)\). The resulting values show how the summary changes under repeated resampling from the observed empirical distribution.
If instead the original scores are held fixed, the calculated quantity is a different function: it varies with which scores are resampled but does not include variation from estimating the score model. This explains why a bootstrap intended to assess fitted-support uncertainty should usually repeat the fitting step. It does not guarantee that the bootstrap distribution is a valid approximation to a population sampling distribution for every support statistic. Empirical minima and maxima are extreme order statistics, and their behavior can be irregular, especially when support is sparse or boundaries are near a model constraint.
Bootstrap percentiles of \(\hat L_b\), \(\hat U_b\), or \(R_b\) can be reported as descriptive uncertainty summaries. A conventional confidence-interval interpretation requires additional justification: the bootstrap must adequately approximate the sampling distribution of the target statistic under the sampling design and model. For min/max boundaries or retention rules with abrupt cutoffs, that approximation may be poor. Avoid presenting percentile ranges as guaranteed confidence intervals without checking the relevant regularity conditions and inferential target.
Diagnostics, Alternatives, and Limitations
A useful report pairs the bootstrap summaries with the original fitted support and explains the rule being assessed. Important warning signs include wide or asymmetric boundary distributions, large variation in retention, an appreciable fraction of failed model fits, and many replicates with an empty common interval. It is also informative to identify whether instability is driven by one treatment group or by a small number of extreme observations. These checks describe uncertainty in the diagnostic; they do not establish that the score model is correctly specified.
The ordinary nonparametric bootstrap resamples observed units, so it cannot create covariate patterns beyond those observed. If the bootstrap holds the fitted scores fixed, it cannot generate scores beyond the original observed score extremes either. With model refitting, however, predicted scores for resampled rows can move beyond the original fitted-score range because the fitted parameters change. Even then, the bootstrap may underrepresent uncertainty beyond observed covariate extremes: no resampled data set supplies genuinely new covariate patterns that were absent from the original sample.
Several approaches answer related but different questions:
| Approach | What it shows | Important limitation |
|---|---|---|
| Single-sample score or tail diagnostics | Where the current fitted sample appears sparse or extreme | Does not show how sensitive that picture is to sampling variation |
| Bootstrap with scores held fixed | Variation from resampling the observed score values | Omits variation from refitting the propensity-score model |
| Bootstrap with model refitting | Resampling variation in both model estimates and the chosen support summary | Cannot reveal unobserved covariate patterns; may be unreliable for non-smooth endpoints |
| Analytic uncertainty for model parameters | Uncertainty in specified model coefficients under their assumptions | Does not directly translate into uncertainty for a thresholded retention rate or empirical support boundary |
Bootstrap instability should not be “fixed” by choosing a rule that happens to retain a desired number of units in the original sample. The rule should have a defensible design rationale, and its consequences should be assessed transparently. Nor should a narrow bootstrap distribution be interpreted as proof of adequate support: every replicate is built from the same finite set of observed covariate patterns, and a systematic model error can persist across all of them.
Common Mistakes
- Holding the score model fixed while claiming to assess model-estimation variability. This only resamples the original scores. Refit the model in each replicate when the target diagnostic is uncertainty in estimated support.
- Claiming resampling cannot produce more extreme refitted scores. Resampling fixed scores cannot exceed their original extremes, but refitting can change predicted scores. The more defensible limitation is that resampling cannot supply covariate patterns beyond those observed, and may therefore underrepresent uncertainty beyond observed extremes.
- Treating bootstrap percentiles as automatically valid confidence intervals. They are descriptive unless the bootstrap approximation is justified for the target statistic and design.
- Ignoring failures or empty intervals. Report their frequency and investigate them; excluding them without explanation can make instability look smaller than it is.
- Equating stable score boundaries with positivity in the population. Stability across resamples does not prove support for unobserved or sparsely represented covariate combinations.
Check Your Understanding
Answer each question using the stated information. Solutions are included below.
- A rule retains 42 of 60 treated units and 45 of 60 controls. What is the total retention rate?
- Why should the propensity-score model be refitted in each bootstrap replicate when assessing uncertainty in estimated support?
- In bootstrap replicates, the lower boundary varies substantially but the upper boundary is nearly constant. What does this suggest, and what does it not establish?
- Does resampling observed rows and refitting guarantee that no replicate score can exceed the original fitted-score maximum? Explain.
- Why should bootstrap percentile ranges for empirical minimum or maximum support boundaries be interpreted cautiously?
Solutions. (1) The total retained count is \(42+45=87\) out of \(60+60=120\), so the rate is \(87/120=0.725\), or 72.5%. (2) Refitting allows the estimated model parameters and resulting scores to vary across resamples; holding scores fixed omits this source of variation. (3) It suggests the estimated lower edge is more sensitive to resampling than the upper edge. It does not prove a population boundary or establish positivity beyond the observed covariate patterns. (4) No. Refitting can change predicted scores and can produce values beyond the original fitted-score maximum for resampled rows, although the resampling still cannot create covariate patterns absent from the observed data. (5) Empirical extrema are non-smooth statistics whose bootstrap approximation may be poor, particularly with sparse support or boundaries; percentile ranges should not automatically be called valid confidence intervals.