Why Refit the Treatment Models?
Treatment weights are functions of estimated treatment probabilities. Consequently, an estimated weight can vary because the observed treatment histories vary and because the probability model fitted to those histories varies. A bootstrap intended to represent the sampling uncertainty of a weighted analysis must reproduce both sources: resample observational units, fit the treatment model again, and then recompute the weights and the target statistic.
Holding the original fitted probabilities fixed while resampling weights or outcomes is a different procedure. It treats the estimated treatment mechanism as known and generally omits the variation from estimating it. This distinction is especially important when weights are products of estimated probabilities across decision times: variation in each fitted factor can affect the product.
For example, with binary treatment \(A_t\) at decision time \(t\), a weight for an observed treatment history can contain factors of the form \(1/P(A_t=a_t\mid H_t)\), where \(H_t\) is the history used by the treatment model. In each replicate, the conditional probabilities are newly estimated from the resampled histories. The bootstrap procedure does not change the target estimand; it approximates how the chosen estimator would vary across repeated samples.
Procedure and Scope
Draw \(n\) observational units with replacement from the original sample. Keep each selected unit’s complete treatment and covariate history together.
Repeat the original probability-estimation procedure in the replicate, including the specified predictors, fitting rules, and any prespecified probability-handling steps.
Use the replicate-specific fitted probabilities to calculate weights, then rerun the weighted analysis to obtain one replicate estimate.
Repeat the process \(B\) times. Use the replicate estimates to estimate uncertainty or form an interval, while recording failed fits and extreme weights.
The resampling scheme must match the independent sampling units. If individuals are independent and each contributes several records, resample individuals, not person-time rows. Row-wise resampling can break within-person histories and does not reproduce the sampling structure assumed by the analysis. Clustered assignments require a different scheme; that setting is beyond the independent-unit setup here.
The basic procedure also assumes that the original treatment-model fitting rule is part of the estimator. If a model is selected by a data-dependent rule, the selection step must be repeated in each replicate if inference is meant to include that selection uncertainty. Repeating a complicated or unstable algorithm does not, by itself, guarantee valid inference.
Worked Example: A Bootstrap Replicate Can Produce an Empty Treatment Group
Suppose four independent units have binary treatment values \(1,1,0,0\), and an intercept-only model estimates the treatment probability by the observed proportion. The original fitted probability is \(\hat p=2/4=0.5\). The inverse probability weight is \(1/\hat p\) for a treated unit and \(1/(1-\hat p)\) for an untreated unit, so both weights equal \(2\).
In an ordinary bootstrap sample of size four, the number \(K\) of treated draws has a \(\operatorname{Binomial}(4,0.5)\) distribution. Thus the replicate estimate is \(\hat p^*=K/4\). The probabilities are \(P(K=k)={4\choose k}/16\): for \(k=0,1,2,3,4\), they are \(1/16,4/16,6/16,4/16,1/16\).
When \(K=1\), the fitted probability is \(1/4\), giving treated weight \(4\) and untreated weight \(4/3\). When \(K=3\), it is \(3/4\), giving treated weight \(4/3\) and untreated weight \(4\). When \(K=0\), the fit gives \(\hat p^*=0\); applying the fitted model’s probability to an observed treated unit would require division by zero. When \(K=4\), the analogous problem occurs for an untreated unit. Each boundary event has probability \(1/16\), so the total probability of a boundary replicate is \(1/8\).
This is not a numerical nuisance that can be silently discarded. It diagnoses that the empirical bootstrap frequently produces data with no example of one treatment value, making the intended inverse-probability calculation undefined for some units. A different model, a justified prespecified handling rule, or a different inferential strategy may be needed; simply reporting an interval after dropping such replicates changes the procedure.
Why the Full Bootstrap Captures Model-Fitting Variation
A useful derivation applies to smooth statistics based on a finite-dimensional treatment model. Let \(O_i\) denote an independent and identically distributed unit, \(P\) its population distribution, and \(P_n\) the empirical distribution. Suppose the fitted parameter \(\hat\beta\) solves an estimating equation \(P_n s(O,\hat\beta)=0\), where \(s\) is a vector-valued score or estimating function. Write \(E[s(O,\beta_0)]=0\), \(A=E[\partial s(O,\beta_0)/\partial\beta]\), and assume \(A\) is nonsingular.
Consider a smooth statistic \(\hat\theta=P_n h(O,\hat\beta)\), with \(\theta=E[h(O,\beta_0)]\) and \(G=E[\partial h(O,\beta_0)/\partial\beta]\). Here \(h\) can represent a weight or a downstream weighted contribution. The assumptions below include finite second moments for the displayed influence terms, differentiability in a neighborhood of \(\beta_0\), and Taylor remainders that are negligible at the \(n^{-1/2}\) scale.
To derive the expansion, first linearize the estimating equation around \(\beta_0\). Since \(P_n s(O,\hat\beta)=0\) and \(E[s(O,\beta_0)]=0\), $$ 0=(P_n-P)s(O,\beta_0)+A(\hat\beta-\beta_0)+o_p(n^{-1/2}), $$ so \(\hat\beta-\beta_0=-A^{-1}(P_n-P)s(O,\beta_0)+o_p(n^{-1/2})\). Next, Taylor-expand \(P_nh(O,\hat\beta)\). The empirical derivative converges in probability to \(G\), giving $$ \hat\theta-\theta=(P_n-P)h(O,\beta_0)+G^\mathsf{T}(\hat\beta-\beta_0)+o_p(n^{-1/2}). $$ Substitution yields the stated sum. In particular, the term involving \(s\) represents variation from estimating the treatment model; a procedure that fixes \(\hat\beta\) does not reproduce that term through model refitting. \(\square\)
For the bootstrap argument, let \(P_n^*\) denote the empirical distribution of a resample. Conditional on the observed data, the same Taylor expansions give $$ \sqrt n(\hat\theta^*-\hat\theta) = \frac{1}{\sqrt n}\sum_{j=1}^n \left\{\hat\psi(O_j^*)-\bar{\hat\psi}\right\}+o_p^*(1), $$ where \(\hat\psi\) estimates the influence function in braces in the theorem and \(\bar{\hat\psi}\) is its original-sample average. The conditional central limit theorem for independent draws from the empirical distribution, together with convergence of the empirical second moments of \(\hat\psi\), gives the same limiting normal distribution as the original-sample expansion. This justifies bootstrap inference under the stated regularity conditions; it does not establish validity for every treatment model or statistic.
Worked Example: Refit a Saturated Treatment Model in Each Resample
Let \(X\) be binary, and fit a saturated logistic treatment model with an intercept and \(X\). In this model the fitted probability within each \(X\) stratum is that stratum’s treated proportion. Suppose the original sample has four units in each stratum: for \(X=0\), one treated and three untreated; for \(X=1\), three treated and one untreated. The original fitted probabilities are \(1/4\) and \(3/4\). A treated unit in stratum \(X=0\) therefore has weight \(1/(1/4)=4\).
Consider a possible bootstrap resample with the following multiplicities. Among the \(X=0\) units, draw the treated unit twice, one untreated unit once, and the other two untreated units zero and once. This gives four \(X=0\) draws, of which two are treated, so \(\hat p_0^*=2/4=1/2\). Among the \(X=1\) units, draw two treated units once each, the third treated unit zero times, and the untreated unit twice. This gives four \(X=1\) draws, again with two treated, so \(\hat p_1^*=2/4=1/2\).
The bootstrap logistic model is therefore refitted with probability \(1/2\) in each stratum. A resampled treated unit with \(X=0\) now receives weight \(1/(1/2)=2\), rather than the original-fit value \(4\). If the original fitted probabilities were held fixed, that change would be missed. This replicate is one possible resample, not an estimate of the bootstrap distribution by itself.
Worked Example: Refit Every Factor in a Two-Time Weight
For a deliberately simple two-time illustration, suppose treatment probabilities are estimated by separate intercept-only proportions at the two times. Four observed histories are \((1,0),(1,1),(0,0),(0,0)\). The fitted treatment proportions are \(\hat p_1=2/4=1/2\) at time 1 and \(\hat p_2=1/4\) at time 2. The unstabilized weight for the first unit’s history \((1,0)\) is $$ \frac{1}{\hat p_1}\frac{1}{1-\hat p_2} =\frac{1}{1/2}\frac{1}{3/4} =\frac{8}{3}. $$
Now take a bootstrap sample with multiplicities \(2,1,1,0\) for the four listed histories. It contains four draws. Three have treatment 1 at time 1, so \(\hat p_1^*=3/4\). One has treatment 1 at time 2, so \(\hat p_2^*=1/4\). For a copy of the first history, the recomputed weight is $$ \frac{1}{\hat p_1^*}\frac{1}{1-\hat p_2^*} =\frac{1}{3/4}\frac{1}{3/4} =\frac{16}{9}. $$ This differs from \(8/3\) because both time-specific fits are part of the procedure. The example uses simple marginal models for arithmetic clarity; the essential bootstrap operation is to refit every probability factor specified by the analysis.
Intervals, Diagnostics, and Alternatives
Given \(B\) valid replicate estimates \(\hat\theta^{*(1)},\ldots,\hat\theta^{*(B)}\), their sample standard deviation estimates bootstrap standard error. Percentile intervals use empirical quantiles of the replicate estimates. Such summaries require enough replicates to estimate the tails reliably; Monte Carlo error can be reduced by increasing \(B\), but a larger number of replicates cannot repair a misspecified treatment model or a bootstrap distribution that fails to represent the sampling law.
Inspect the replicates, not only the final standard error. Record convergence failures, treatment cells with no observations, very small fitted probabilities, maximum weights, and the distribution of the final statistic. A small number of extreme estimates can dominate the standard deviation or make percentile endpoints unstable. Report how many replicates failed and what prespecified rule was used. Do not silently remove failed replicates and describe the remainder as an ordinary bootstrap distribution.
The delta method, including the earlier material on products of estimated probabilities, uses a local approximation to propagate parameter uncertainty. It can be efficient for smooth finite-dimensional models but may be cumbersome for complicated pipelines. A sandwich variance estimator can also account for estimating equations under its assumptions. The full bootstrap is attractive when refitting and recomputing the statistic are straightforward, but it can be computationally expensive and unstable in sparse samples. These methods are not interchangeable guarantees: their validity depends on the estimator’s regularity and the sampling design.
A common mistake is to resample units, retain the original fitted probabilities, and call the resulting spread full-bootstrap uncertainty. That procedure conditions on the original model fit. The correction is to repeat the entire probability-estimation algorithm, then recompute all weights and the weighted statistic. Another mistake is to resample repeated records independently when the unit of independent sampling is a person. In that case, keep each person’s full history intact.
Finally, bootstrap inference may be unreliable when the estimator is nonsmooth, treatment probabilities approach zero, model selection changes discontinuously, or many replicates have separation or empty cells. Truncating extreme weights can define a more stable statistic, but if truncation is part of the estimator it must be applied consistently in the original sample and every replicate. It also changes the estimator and does not automatically remove bias.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Why must a longitudinal bootstrap resample whole individuals rather than individual decision-time records?
- In an intercept-only binary treatment model, a bootstrap replicate contains \(K=3\) treated draws out of \(n=6\). What is its fitted treatment probability and its treated inverse probability weight?
- What component of uncertainty is omitted when treatment probabilities are held fixed across bootstrap resamples?
- Name two diagnostics that should be checked across bootstrap replicates, and explain what each can reveal.
Solutions. (1) Repeated records from one person are dependent and form a single history; resampling rows independently breaks that history and changes the sampling structure. (2) The fitted probability is \(3/6=1/2\), so the treated weight is \(1/(1/2)=2\). (3) The variation caused by estimating the treatment-model parameters and hence the fitted probabilities is omitted. (4) For example, count failed fits to detect separation or empty cells, and examine maximum weights to detect near-zero fitted probabilities and unstable weighted estimates.