Bootstrap Variability in Weight Diagnostics

Why Bootstrap Weight Diagnostics?

A weight diagnostic is calculated from one observed sample, but the sample is only one realization of the study population. A high upper quantile, low effective sample size, or residual imbalance may partly reflect which participants happened to be sampled. Bootstrap resampling makes this sampling variability visible: it asks how the diagnostic changes when the observed participants are resampled and the analysis is repeated.

For longitudinal marginal structural models, the resampling unit is ordinarily the participant. A participant contributes a treatment and covariate history across follow-up, so those records must stay together. In each replicate, resample participants with replacement, refit the treatment models, reconstruct the weights, and recalculate the diagnostics. Resampling individual person-time rows separately would break the observed histories and generally does not represent the sampling process.

Definition: A participant-level nonparametric bootstrap replicate is a sample of $n$ participants drawn with replacement from the $n$ observed participants. The complete observed record of each selected participant is included. A bootstrap weight diagnostic is the diagnostic statistic recalculated from that replicate, ordinarily after refitting the treatment models and recomputing the weights.

Let $W_i$ denote a participant's weight under the fitted analysis, and let $W_i^{*(b)}$ denote the weight in bootstrap replicate $b$. The star marks a replicate quantity. Common summaries include empirical weight quantiles, effective sample size, and balance measures. For example, for nonnegative weights $W_1,\ldots,W_n$, the effective sample size is

\operatorname{ESS}(W)=\frac{\left(\sum_{i=1}^n W_i\right)^2}{\sum_{i=1}^n W_i^2}, \qquad \sum_i W_i^2>0.

The ESS is a weight-concentration summary, not a literal count of independent observations. Multiplying every weight by the same positive constant leaves it unchanged. In a longitudinal analysis, the exact weight being summarized—such as a cumulative weight at a specified follow-up time—must be stated.

What the Bootstrap Estimates

For a diagnostic statistic $T$, the empirical bootstrap approximates the sampling variation of $T$ by the variation of $T^{*(b)}$ across replicates. A practical analysis uses $B$ replicates and reports, for example, the median and spread of the replicate values, or selected quantiles. These summaries describe variability under resampling from the observed empirical distribution. They do not establish that the treatment models are correctly specified or that population positivity holds.

The bootstrap procedure should follow the analysis pipeline. If the original weights use fitted numerator and denominator treatment probabilities, each replicate must refit the same models using the resampled participants and construct the replicate weights in the same way. Any prespecified truncation rule must also be applied consistently, while results before and after truncation should not be conflated. If a model fails to fit or a diagnostic is undefined, record that outcome rather than silently discarding the replicate.

Resample participants.
Draw $n$ participant identifiers with replacement and carry each selected participant's complete longitudinal record into the replicate.

Repeat weight construction.
Refit the treatment models, calculate the replicate probabilities, and recompute the corresponding weights using the original analysis rules.

Recalculate diagnostics.
Record the chosen weight quantiles, ESS, balance summaries, and any prespecified positivity-warning indicators.

Summarize variation and failures.
Describe the replicate distribution and report how many replicates had fitting failures, undefined summaries, or warnings.

Two Properties That Clarify Interpretation

The simplest bootstrap calculation helps distinguish what resampling does from what it does not do. The result below is exact for the bootstrap distribution of the sample mean, conditional on the observed data.

Result: Let $x_1,\ldots,x_n$ be observed real values, and draw $X_1^*,\ldots,X_n^*$ independently from their empirical distribution, which assigns probability $1/n$ to each observed value. For $\bar X^*=n^{-1}\sum_{j=1}^n X_j^*$, $$ E^*[\bar X^*]=\bar x,\qquad \operatorname{Var}^*(\bar X^*)=\frac{1}{n}\left\{\frac{1}{n}\sum_{i=1}^n(x_i-\bar x)^2\right\}. $$ Here $E^*$ and $\operatorname{Var}^*$ refer to the conditional bootstrap distribution given the observed data.

To prove the result, each bootstrap draw has conditional expectation $E^*[X_j^*]=n^{-1}\sum_i x_i=\bar x$. Linearity of expectation gives $E^*[\bar X^*]=\bar x$. The conditional variance of one draw is $n^{-1}\sum_i(x_i-\bar x)^2$. Since the $n$ draws are conditionally independent,

\operatorname{Var}^*(\bar X^*) =\operatorname{Var}^*\left(\frac{1}{n}\sum_{j=1}^n X_j^*\right) =\frac{1}{n^2}\sum_{j=1}^n\operatorname{Var}^*(X_j^*) =\frac{1}{n}\left\{\frac{1}{n}\sum_{i=1}^n(x_i-\bar x)^2\right\}.

This is a property of the resampling distribution, not a guarantee that the bootstrap variance estimates the true sampling variance well for every statistic. In particular, a rare treatment history may be poorly represented in the empirical sample to begin with.

That concern can be made precise. Suppose a particular history occurs for $m$ of the $n$ observed participants. The probability that a size-$n$ participant bootstrap omits every one of them is

\left(1-\frac{m}{n}\right)^n.

Each draw misses the $m$ participants with probability $1-m/n$; independence of the $n$ draws gives the displayed probability. If $m$ stays fixed while $n$ increases, then $(1-m/n)^n\to e^{-m}$, which is positive. Thus, when only a few observed participants represent a history, bootstrap replicates can omit that history frequently even in large samples. The calculation warns about instability; it does not diagnose the population probability of that history.

Worked Examples

Worked Example: Summarizing Bootstrap Weight Quantiles

Suppose eight bootstrap replicates produce the following values for a specified upper weight quantile:

0.02,\ 0.05,\ 0.07,\ 0.09,\ 0.10,\ 0.13,\ 0.17,\ 0.24.

The sum is $0.02+0.05+0.07+0.09+0.10+0.13+0.17+0.24=0.87$, so the replicate mean is $0.87/8=0.10875$. The smallest value is $0.02$ and the largest is $0.24$, giving a range of $0.24-0.02=0.22$. With eight ordered values, the conventional sample median is the average of the fourth and fifth values: $(0.09+0.10)/2=0.095$.

These calculations show that the upper-quantile diagnostic varies appreciably across these replicates. Eight replicates are far too few for a dependable estimate of extreme bootstrap percentiles. The range is descriptive, not a confidence interval, and the mean is not a replacement for inspecting the replicate distribution.

Worked Example: Effective Sample Size in Resampled Samples

For a small illustration, hold four observed weights fixed at $1,1,2,4$ and consider three bootstrap resamples of those weights. This isolates the arithmetic of resampling weights; it is not the recommended full procedure when the weights were estimated from treatment models.

The original ESS is $(1+1+2+4)^2/(1^2+1^2+2^2+4^2)=64/22$, or about $2.91$. The first resample has weights $1,1,1,4$. Its sum is $7$, its sum of squares is $1+1+1+16=19$, and its ESS is $49/19\approx2.58$.

The second resample has weights $1,2,2,4$. Its sum is $9$, its sum of squares is $1+4+4+16=25$, and its ESS is $81/25=3.24$. The third resample has weights $1,1,2,2$. Its sum is $6$, its sum of squares is $1+1+4+4=10$, and its ESS is $36/10=3.60$.

The same observed weight list can therefore yield different ESS values under resampling. In a real analysis, resample participants and refit the treatment models in every replicate: the resulting weights can change both because participants are repeated or omitted and because model estimates change.

Worked Example: Bootstrap Variation in a Balance Summary

Consider a one-time illustration with a binary treatment $A$, covariate $X$, and fixed nonnegative weights. Use the weighted difference in means, treated minus untreated, as the balance summary. The six records are: treated participants 1, 2, 3 have $(X,W)=(0,2),(2,1),(1,1)$; untreated participants 4, 5, 6 have $(X,W)=(0,1),(1,2),(3,1)$. The original treated weighted mean is $(0\cdot2+2\cdot1+1\cdot1)/(2+1+1)=3/4$. The untreated weighted mean is $(0\cdot1+1\cdot2+3\cdot1)/(1+2+1)=5/4$. The difference is $3/4-5/4=-1/2$.

For arithmetic clarity, use a stratified bootstrap here: sample three treated records with replacement from the treated group and three untreated records with replacement from the untreated group. In replicate 1, select treated indices $(1,2,2)$ and untreated indices $(4,5,6)$. The treated weighted mean is $(0\cdot2+2\cdot1+2\cdot1)/4=1$; the untreated mean is $5/4$. The difference is $1-5/4=-1/4$.

In replicate 2, select treated indices $(1,1,3)$ and untreated indices $(4,4,5)$. The treated mean is $(0\cdot2+0\cdot2+1\cdot1)/5=1/5$. The untreated mean is $(0\cdot1+0\cdot1+1\cdot2)/4=1/2$. The difference is $1/5-1/2=-3/10$.

In replicate 3, select treated indices $(2,3,3)$ and untreated indices $(5,6,6)$. The treated mean is $(2+1+1)/3=4/3$; the untreated mean is $(1\cdot2+3\cdot1+3\cdot1)/4=2$. The difference is $4/3-2=-2/3$. These three replicates range from $-2/3$ to $-1/4$, showing that a single balance summary can vary under resampling. In a longitudinal marginal structural analysis, the full procedure would resample participants and refit the models rather than hold weights fixed; this example isolates the weighted-mean calculation.

Assessing Warning Stability and Diagnosing Failures

A positivity warning needs an operational definition before its bootstrap stability can be summarized. For example, an analyst might define a warning as a fitted treatment probability below a prespecified threshold, or as a weight quantile exceeding a prespecified threshold. The choice depends on the diagnostic and the analysis context; it should not be selected after seeing which definition produces the most reassuring result.

If $I_b$ equals 1 when replicate $b$ meets the chosen warning rule and 0 otherwise, then $B^{-1}\sum_{b=1}^B I_b$ is the fraction of replicates with that warning. A large fraction means the warning recurs under empirical resampling. A small fraction means it is less stable under this resampling scheme, not that positivity has been established. Report the rule, the denominator $B$, and the number of failed or undefined replicates so the fraction can be interpreted.

Several patterns deserve attention. A wide replicate distribution of an upper weight quantile or ESS indicates that the corresponding summary depends strongly on the observed sample. Frequent model-fitting failures may indicate sparse treatment histories, separation, or a fragile specification. A balance summary that is stable near zero is reassuring only for that summary and those replicates; it does not rule out imbalance in other aspects of the covariate distribution. The earlier material on distributional balance and higher-moment balance explains why a small set of mean differences cannot represent every kind of imbalance.

Check replicate completion. State the number of successful fits and how failures were handled. A summary based only on successful replicates can be misleading if failure is systematic.
Inspect the full distribution. Report selected quantiles or a plot in addition to a standard deviation. A few extreme replicates can make the distribution asymmetric.
Separate sources of variation. A bootstrap that refits models includes both participant resampling and treatment-model fitting variability. A bootstrap with fixed weights omits the latter.
Preserve the sampling structure. If participants were sampled in clusters, resample the independent clusters. If a stratified resampling scheme is used, say so; it targets variation conditional on the strata counts rather than the ordinary participant-level bootstrap.

Alternatives, Limits, and a Common Mistake

Bootstrap variability complements, rather than replaces, the diagnostics and sensitivity analyses used elsewhere in the course. Analytic standard errors can be efficient for smooth statistics under a well-specified sampling model, but deriving valid uncertainty for a complex fitted-weight diagnostic may be difficult. Bootstrap resampling is flexible, but computationally intensive and not automatically reliable for rare histories or nonsmooth statistics. Perturbing treatment-model specifications, as in the earlier material on treatment-model instability, asks a different question: whether conclusions change under alternative model choices. Bootstrap resampling holds the analysis rule fixed and asks how results vary across samples drawn from the observed data.

Bootstrap quantiles and extrema can be nonsmooth, and a statistic near a boundary or based on very sparse histories may have a bootstrap distribution that poorly approximates its sampling distribution. The usual bootstrap rationale is most credible when participants are independent sampling units, the statistic is sufficiently regular, the relevant histories are represented adequately, and model fitting behaves stably in nearby samples. It cannot manufacture information about histories absent from the observed data.

Common mistake: “Only one bootstrap replicate showed a positivity warning, so positivity is not a concern.” This reasoning treats the bootstrap warning fraction as a test of population positivity. The correction is to interpret it as a stability summary conditional on the observed data and the chosen resampling and warning rules. Sparse histories can be absent from the data or frequently omitted in bootstrap replicates, so substantive positivity assessment requires more than the warning fraction.

A second frequent error is to resample person-time rows independently. In longitudinal settings, this breaks the within-person sequence and changes the dependence structure. Resample the independent participant (or cluster) and preserve each selected unit's complete record. The bootstrap's value is disciplined repetition of the intended analysis, not repetition of a different data-generating structure.

Check Your Understanding

Answer each question, then compare with the brief solutions.

In a longitudinal study, what should ordinarily be resampled, and why?
Why should treatment models generally be refitted within each bootstrap replicate?
A rare history is observed in $m$ of $n$ participants. What is the probability that a size-$n$ bootstrap omits all participants with that history?
What does the fraction of replicates meeting a prespecified positivity-warning rule describe, and what does it not establish?
How does bootstrap resampling differ from perturbing the treatment-model specification?

Solutions. (1) Resample complete participants, or independent clusters when clustering defines the sampling units, to preserve each unit's longitudinal dependence structure. (2) Refitting includes variability from estimating treatment probabilities; holding weights fixed omits that source of variation. (3) The probability is $(1-m/n)^n$, because each of the $n$ draws misses the history with probability $1-m/n$. (4) It describes how often the warning recurs under the empirical resampling scheme and chosen rule; it does not prove or disprove population positivity. (5) The bootstrap varies the observed sample while holding the analysis rule fixed; specification perturbation changes the treatment model to examine sensitivity to modeling choices.