Why the Bootstrap Must Reproduce the Analysis
In a longitudinal marginal structural model (MSM), the final estimate depends on more than the outcome model. It also depends on estimated treatment and censoring probabilities, which are combined into weights and then used to fit the MSM. Consequently, uncertainty in those probability estimates can contribute to uncertainty in the causal effect estimate.
A bootstrap intended to quantify uncertainty in the full procedure must repeat the full procedure. In each replicate, resample independent study units, re-estimate the treatment and censoring models, reconstruct the weights, and refit the MSM. Resampling the data while keeping the original weights fixed answers a narrower question: how much does the MSM estimate vary when the observed records change but the estimated weighting procedure is treated as known?
What Must Be Repeated in Each Replicate
Let \(i\) index independent study units and \(t=0,\ldots,K\) index treatment times. Write \(A_{it}\) for treatment, \(H_{it}\) for the observed history used in the treatment model, and \(C_{it}=1\) if the person is censored during interval \(t\). A stabilized longitudinal weight combines treatment and censoring probability ratios. Schematically, for a person observed through the relevant times, it has the form
Here \(\widehat g_t\) and \(\widehat r_t\) are fitted denominator probabilities, while \(\widehat q_t\) and \(\widehat s_t\) are fitted numerator probabilities; \(V_{it}\) denotes the history included in the corresponding numerator model. The exact factors depend on the specified weight construction and the person's observed follow-up. The important point for bootstrapping is that these are estimated quantities. They must be estimated anew from each resampled data set.
The independent sampling unit is usually the person, not the person-visit record. Draw \(n\) people with replacement. For each selected person, carry along the complete longitudinal record required by the analysis, including treatment, covariate, censoring, and outcome information. If a person is selected more than once, their copies must be treated as separate bootstrap copies when fitting the models. Resampling visit rows separately would break within-person histories and would not reproduce the original sampling design.
Within each replicate, fit the same treatment and censoring model specifications used in the original analysis. Then calculate the replicate-specific weights, apply the same prespecified truncation rule if truncation is part of the analysis, and fit the same weighted MSM. The resulting coefficient or contrast is one bootstrap estimate, say \(\widehat\psi^{*(b)}\). The full-pipeline sequence is summarized below.
Draw \(n\) independent units with replacement and retain each selected unit’s longitudinal record.
Fit the specified treatment and censoring numerator and denominator models in the bootstrap sample.
Calculate the replicate-specific longitudinal weights and apply the original analysis’s prespecified weight processing.
Fit the weighted marginal structural model and save the target coefficient or contrast.
Use the distribution of the saved estimates to estimate uncertainty and, where appropriate, form a confidence interval.
Worked Example: Reconstructing a Longitudinal Weight
Suppose a hypothetical two-time treatment analysis has fitted probabilities for one person's observed treatment and uncensored status as follows. At the first time, the treatment numerator and denominator probabilities are \(0.40\) and \(0.50\); at the second time they are \(0.30\) and \(0.60\). The corresponding censoring numerator and denominator probabilities are \(0.90\) and \(0.95\) at the first time, and \(0.80\) and \(0.90\) at the second time. Assume these are the probabilities for the observed events and that the person contributes at both times.
The person's combined stabilized weight is the product of the four ratios:
The calculation uses the original ratios through the multiplication; the displayed decimal is rounded only at the end. In a bootstrap replicate, the same calculation is made with probabilities from models refitted to that replicate, rather than with the probabilities from the original sample. Thus the replicate weight can differ even for a resampled copy of the same observed person.
Bootstrap Standard Errors and Intervals
For \(B\) bootstrap replicates, let \(\widehat\psi^{*(1)},\ldots,\widehat\psi^{*(B)}\) be the estimates from the complete procedure. Their sample standard deviation estimates the bootstrap standard error:
A percentile bootstrap confidence interval uses the empirical quantiles of the replicate estimates. For example, the 2.5th and 97.5th percentiles give a nominal 95% percentile interval. Under suitable sampling and regularity conditions, the bootstrap distribution approximates the sampling distribution of the estimator. This does not mean there is a 95% probability that a fixed causal parameter lies in the interval. The interval procedure is designed to cover the parameter in about 95% of repeated samples when its assumptions and approximation are adequate.
Worked Example: Calculating a Bootstrap Standard Error
In a small hypothetical demonstration, five full-pipeline bootstrap replicates produce effect estimates of \(1.7\), \(2.1\), \(2.5\), \(2.9\), and \(2.3\) units. These five replicates are chosen to show the arithmetic; a real analysis would use many more.
Their mean is
The squared deviations from \(2.3\) are \(0.36\), \(0.04\), \(0.04\), \(0.36\), and \(0\), with sum \(0.80\) squared units. Therefore,
This is a bootstrap estimate of the standard error, not a standard error of \(0.4472\) units known without uncertainty. With only five replicates it is highly imprecise. In practice, use enough replicates for the standard error and the interval quantiles to be stable, and report the number of replicates and the interval construction.
Why Fixed-Weight Resampling Can Miss Variation
A fixed-weight bootstrap resamples units and refits the MSM but carries the original estimated weights into every replicate. This can be useful for a limited conditional calculation, but it is not the bootstrap analogue of the full analysis. It omits changes in the fitted treatment and censoring probabilities across resamples and the resulting changes in the weights.
One way to understand the distinction is through a first-order expansion. Let \(\widehat\eta\) collect the nuisance-model parameters used to estimate the weights. In a regular setting, the full estimator's error can be represented schematically as
where \(\phi_i\) represents the contribution from sampling variation in the estimating procedure when the nuisance parameters are held at their true values, and \(D(\widehat\eta-\eta)\) represents the first-order effect of estimating those nuisance parameters. This is a justification, not an exact decomposition for every estimator. In a full bootstrap, both the empirical data and the fitted nuisance models vary, so the replicate distribution can reflect both sources of variation and their dependence. A fixed-weight bootstrap holds \(\widehat\eta\) constant and does not reproduce its estimation contribution. The two standard errors therefore need not be equal; there is no general guarantee that the full-pipeline result is larger in every finite sample.
Worked Example: Comparing Full and Fixed-Weight Bootstrap Results
Suppose five hypothetical resamples are used to illustrate two procedures. Refitting all weights and the MSM gives estimates \(1.7, 2.1, 2.5, 2.9, 2.3\) units. Holding the original weights fixed and refitting only the MSM gives \(1.9, 2.1, 2.3, 2.5, 2.2\) units. These values are illustrative replicate outputs, not a claim about which method must be more variable in every data set.
For the full-pipeline estimates, the mean is \(2.3\) units and the squared deviations sum to \(0.80\) squared units. The estimated standard error is therefore
For the fixed-weight estimates, the mean is
Their squared deviations from \(2.2\) sum to \(0.20\) squared units, so
In this illustration, the fixed-weight bootstrap reports less variation because it does not refit the probability models. This numerical comparison alone does not prove the fixed-weight estimate is always too small; rather, it shows why it answers a different uncertainty question and may omit a relevant source of variation.
Practical Checks and Common Pitfalls
A bootstrap cannot repair a poorly specified analysis or guarantee reliable inference when the estimator behaves irregularly. Before interpreting the results, check that the independent units were resampled correctly, that each replicate followed the original model-fitting and weight-processing rules, and that the fitting procedure succeeded. The usual bootstrap justification relies on independent sampling units and a sufficiently regular estimator. Sparse treatment histories, probabilities near zero, separation in a fitted model, or severe weight instability can make replicate estimates erratic or cause model fitting to fail.
If a prespecified truncation rule is part of the target analysis, apply that same rule inside every replicate after recalculating the replicate weights. Do not choose a new truncation rule in each replicate based on which choice gives the most appealing result. If a model fails to converge or a bootstrap sample contains no information for a required model component, record the problem and investigate it; silently discarding failed replicates can distort the bootstrap distribution.
- Wrong: “The weights are already estimated, so I can resample rows and reuse them.” Correction: For full-pipeline uncertainty, resample independent units and refit the treatment and censoring models to recalculate weights.
- Wrong: “Each visit is an independent observation for resampling.” Correction: Resample the independent unit, ordinarily the person, and retain that unit’s longitudinal record.
- Wrong: “The full-pipeline bootstrap must always have a larger standard error.” Correction: It includes nuisance-model re-estimation, but finite-sample variability can move in either direction.
- Wrong: “A large number of replicates fixes unstable weights or model failure.” Correction: More replicates reduce Monte Carlo error; they do not resolve identification, positivity, or fitting problems in the data and models.
Check Your Understanding
Try these questions without looking back.
- Why is resampling person-visit rows separately inappropriate when people contribute repeated observations?
- Name the main model-fitting and weight-calculation steps that must be repeated in a full-pipeline bootstrap replicate.
- In what sense does a fixed-weight bootstrap answer a narrower uncertainty question than a full-pipeline bootstrap?
- If weight truncation is part of the original analysis, when and how should the truncation rule be applied in each replicate?
- Why does increasing the number of bootstrap replicates not resolve separation or extreme-weight problems?