Choosing Bootstrap Replicate Counts

Why the Replicate Count Is a Precision Choice

A bootstrap analysis uses repeated resamples and model refits to approximate the sampling distribution of a statistic, such as clearance (\(CL\)) or an exposure measure. Those repeated calculations are themselves a Monte Carlo computation: a finite set of successful bootstrap replicates only approximates the distribution the bootstrap procedure defines. Too few replicates can make reported standard errors and confidence limits noticeably dependent on the particular random resamples generated.

The replicate count should therefore be chosen in relation to the quantity being reported and the precision needed for the analysis. Standard errors and confidence limits do not have identical Monte Carlo behavior. A standard-error estimate summarizes spread across the distribution; a confidence limit is a tail quantile and can be much more sensitive to the number of observations near that tail. There is no single replicate count that guarantees adequate precision for every parameter, model, and reporting purpose.

A failed refit does not contribute a valid \(T_b^*\), so the number of attempted refits may exceed \(B\). The earlier material on bootstrap convergence and failed replicates is relevant here: increasing the number of attempts can provide more successful replicates, but it does not make a systematic or informative failure mechanism harmless. Precision calculations below presume that the successful replicates are independent draws from the intended bootstrap distribution, or close enough for the approximations to be useful, and that the statistic is defined consistently across replicates.

Monte Carlo Error in a Bootstrap Standard Error

The bootstrap standard error is the standard deviation of the replicate statistics. Its estimate from \(B\) successful replicates is written \(\widehat{SE}_{boot}\). The question here is not how to interpret that standard error as sampling uncertainty; it is how much Monte Carlo variation to expect in its finite-\(B\) estimate.

A useful approximation follows from the sampling variability of a sample variance. Let the bootstrap distribution have standard deviation \(\sigma_*\) and standardized fourth moment \(\kappa=E[(T^*-\mu_*)^4]/\sigma_*^4\). For independent replicates, the large-\(B\) variance of the sample variance is approximately \(\sigma_*^4(\kappa-1)/B\). Applying the square-root transformation gives the following approximate relative Monte Carlo standard error for the estimated standard deviation:

If the replicate distribution is approximately normal, then \(\kappa\approx3\), giving the familiar planning approximation:

The \(B-1\) version is a convenient finite-sample approximation; it is not an exact guarantee for a nonnormal bootstrap distribution. Rearranging gives a rough planning count for target relative Monte Carlo error \(r\):

This is a planning rule, not a universal minimum. It relies on independent replicate draws and a distribution whose fourth moment is not so large that the normal approximation is misleading. A heavy-tailed or irregular replicate distribution can require more replicates.

Why Confidence Limits Often Need More Replicates

A confidence limit computed from bootstrap replicates is commonly an empirical quantile. For example, a two-sided percentile interval with nominal 95% coverage uses the 2.5th and 97.5th percentiles. The Monte Carlo precision of an empirical \(p\) quantile depends both on \(B\) and on how densely the bootstrap distribution is populated near the true quantile.

If the bootstrap distribution has a smooth density \(f\) that is positive at its \(p\) quantile \(q_p\), the large-sample approximation for the Monte Carlo standard error of the empirical quantile is:

The numerator describes uncertainty in the rank of the quantile; the denominator converts rank uncertainty into the statistic’s units. A small density at the endpoint means even a modest change in rank can move the endpoint substantially. For a \(p=0.025\) endpoint, the expected number of replicates below that endpoint is only \(0.025B\). Thus \(B=1{,}000\) provides about 25 replicates in that tail, while \(B=10{,}000\) provides about 250. A count that is adequate for a standard-error estimate may therefore be inadequate for stable tail limits.

The density \(f(q_p)\) is unknown and must itself be estimated if this formula is used to estimate endpoint Monte Carlo error. In practice, local density estimates can be unstable, especially with few tail observations, tied values, skewness, or a boundary. The formula is best treated as a guide to what controls endpoint precision, not as a precise promise about a particular dataset.

Practical Precision Checks

A useful workflow starts by specifying precision in the units and context of the reported quantity. For a standard error, a relative target may be sensible; for a confidence limit, an absolute tolerance in units such as \(\mathrm{L/h}\) or \(\mathrm{mg\cdot h/L}\) is often easier to interpret. The target should be consequentially small compared with the precision at which the result will be reported, not merely small in a mathematical sense.

Batch checks are diagnostics, not proof that a limit has converged. Cumulative estimates at successive counts are dependent because earlier replicates remain in the later calculation. Small changes can occur by chance, and a later batch can move a previously stable endpoint. Where feasible, compare independent batches or continue adding batches until estimates remain within the stated tolerance across more than one update. For independent estimates with Monte Carlo standard errors \(m_1\) and \(m_2\), the standard error of their difference is approximately \(\sqrt{m_1^2+m_2^2}\); this can help distinguish a meaningful batch discrepancy from ordinary simulation variation.

Diagnostics, Alternatives, and Limitations

Several warning signs indicate that a simple replicate-count calculation may be unreliable:

• Few tail replicates. With \(B=1{,}000\), a 2.5th percentile is determined by observations near the 25th ordered replicate. A handful of unusual values or a small change in rank can shift the limit.
• Unstable endpoint estimates. Large changes across successive batches, particularly relative to the chosen tolerance, call for more replicates and an examination of the replicate distribution.
• High skewness or heavy tails. The normal-based standard-error approximation assumes a well-behaved fourth moment. Extreme replicate values can increase variability and make a small-\(B\) approximation optimistic.
• Boundaries, ties, or multimodality. A smooth positive density at the quantile may not be a reasonable approximation. Empirical quantiles can jump between distinct values or modes.
• Selective failures. If valid refits are disproportionately obtained from particular resamples or parameter regions, adding more successful replicates does not by itself repair the resulting selection problem.

One alternative is to use an analytic or model-based approximation to uncertainty instead of relying on a large bootstrap computation. Such an approach may be faster, but it answers the intended question only if its assumptions and model approximation are appropriate. Another is to run independent bootstrap batches and compare their summaries; this is a useful Monte Carlo diagnostic but does not remove bias or misspecification in the bootstrap procedure. Increasing \(B\) reduces finite-simulation noise at the usual approximate rate of \(1/\sqrt{B}\), but does not make the underlying resampling design, refit strategy, or model valid.

A replicate count is consequently a computational precision choice, not a substitute for study design or model assessment. Precision targets should be reported or documented alongside the achieved successful count, attempted count, failure pattern, and any endpoint checks. A visually smooth histogram alone is not evidence that tail quantiles are precise, and printing more decimal places does not improve their Monte Carlo accuracy.